Batch Processing

Mainframes are built to move huge amounts of data around.  IBM did a fantastic job of building giant batch processing engines.  Granted there are online options, but, to be honest, their bread and butter is huge amounts of data in and out.  Because of that mentality, the fact that the course designers at Universities and older people in the IT field "growing up" with this technology and even the integration of the term "batch processing" into the general lexicon of life, people design systems to include batch processing.

*WRONG*

OK, that may be too strong.  Let's say that the majority of the "reasons" for batch processing are actually replaceable with the phrase "because we've always done it that way".  Batch processing, the processing of huge volumes of data in a scheduled, off prime time manner, is a relic of the mainframe and needs to be replaced with the idea of a batch of one.  This idea of a batch of one is not new, as I`ve even used it on the mainframe.  The user needed certain processing done immediately as opposed to that night and we dynamically created a job, complete with the JCL and necessary parameters, in a low priority job queue and the job was processed when the machine had spare cycles.  It was batch, but a batch of one.

The idea is applicable in today's world.  When I ask people why they are creating a batch job they say something like "the printer is loaded with a certain paper at night so that is the only time we can do this".  *WRONG*  The processing can be done at anytime, the printing can only occur at night.  Instead of doing a lot of processing at night, do the processing in an asynchronous manner throughout the day and do the printing at night.  Split things into two pieces:  processing and printing.  They are two separate functions and do not need to be combined.

Let me rephrase in a more "global consulting" manner:  by separating processes into their unique steps, you create more opportunities to provide value to the business client.  Through a selective process of decoupling the steps in a business process you provide opportunity to inject new business functionality into a more flexible framework with less rework to the existing systems. Isn't that really what the business user wants?  More flexibility?  More value?

Short tempered

Some have said that my comments in the Daily Migration Note are brusque or just plain old rude.  Others have said that I am condescending in my notes.  A number of years ago a project manager actually said he found me "abusive and your attitude repugnant".  (I always keep those good emails so that I can refer to them later.)  Others call me up to see if I am still sane or whether I'm going to go postal.

I am, by my own admission, a strange duck.  Pressure doesn't affect me the same as other people.  I've been in situations where an the vast majority of an organization was sitting on their hands, waiting for me to come up with a solution. I've been the key person involved in resolving a payroll issue that affected tens of thousands of people.  While they are stressful situations, the pressure of the problem, the adrenaline that is produced, all blend together.

And yet, some things make me react quite harshly.  Yes, even quite rudely.  Why?

At a recent event there was an opportunity to evaluate myself based on a number of different characteristics.  Based on my answers to the questions, I was (drum roll please) not a people person.  This may not be a surprise to a bunch of you, maybe all of you.  What the quiz showed me was something that I knew all along:  process and logic are the key factors when I am dealing with work.

So, what does this mean?  Well, if things don't make sense to me I have a tendency of digging my heels in and saying "No".  If there is no logical sequence of events I have a tendency of saying "stop".  If people try to pressure me to do something that I think is illogical, I fight back.

Do you know what actually makes sense to me?  Something that is a logic progression when resolving a problem?  A partnership.  Not a partnership where one or more sides pay lip service to the idea of a partnership so that they can check it off their list of marketing responsibilities (there are many such partnerships amongst major corporations that fall into this category), but a true partnership where each side is working towards a common goal and where all parties are free to bring forth ideas to the table.  No finger pointing.  No "blame game" tactics. 

I have been involved in a number of different groups that were true partnerships between two or more groups and the results are amazing.  With the freedom to make any suggestion about any topic without fear of reprisal the ideas and solutions that come out of such a group are amazing.  There is tremendous difficulty in this as you need to leave your ego at the door and be willing to admit you were wrong.  You need to understand that there is more than one way to skin a cat and that yours may not be as effective.  And, in a true partnership, you don't try to pressure people to follow your lead or your suggestion, you true to influence them and persuade them that your way is appropriate to try.  As soon as you try pressuring people the partnership is gone.  And, in my case, I start resorting to my stock answers of "no" and "you must be kidding".

Partnerships.  Give them a try and you'll be amazed at what you can do.

Debugging

Don't blame the Deployment Team if your application fails and you can't figure out why.

Increasingly we have seen project teams with problems in Production (or UAT for that matter) and they are at a loss to explain what the problem is.  Sometimes there is a cryptic error message in one of the event logs along the lines of "Object reference not set to an instance of an object".  <sarcasm> Now that is a descriptive error message. </sarcasm>

There are a number of things you can do to make your life easier:

• The various .NET languages come with a feature called a Try...Catch block.  Where you try something and if things don't work you catch the exception.  Amazing how that works.  If you haven't put Try...Catch blocks in your code, particularly around any code that is calling a service outside of your control (SHR, EPS, DMS, PRS, etc.), then you score -5 points on the attention to detail rating.
• Did you know that if you catch an exception you can add your own information?  Yes, if you've caught something you can add information such as what you were doing when you got the exception or copies of the data used in the method so that people don't have to guess at what went wrong.  If you just rethrow the exception without adding any information, deduct 5 points from your "value added" score.
• Did you know that there is a Trace class in .NET?  It can do all sorts of things for you, such as allowing you to dump out detailed trace information (that is if you've written detailed trace information in your application) by changing a config file or even dynamically through code.  This is a very handy feature when used in combination with breadcrumbing so that you can see the complete route that an application has taken and, if you've added it, the data used throughout the process.  If you've never used Trace then how do you ever expect to trace anything in Production?

There are lots of things you can do to make your life easier when it comes to resolving problems in Production.  Simple things that can be useful in any environment.  Simple things that you should know how to use and how to use effectively.

Introduction to an old friend

I'd like to introduce to you, an old friend of mine.  The PRINT statement.  Granted, he's not as fancy as a GUI Debugger, nor as sexy as ANT or AVICODE, but he's still really useful.  Have anyone thought of adding PRINT statements to their T-SQL that they are executing?

For instance you could have something like this:

(14 row(s) affected)

(14 row(s) affected)

Or something like this:

Merge Learner Script
(14 row(s) affected)
Learner Merge - From SIN : 918728130 To SIN : 665171625
Start date - 2008-08-22 09:30:03.120
Create an Activity
Close the Activity
Learner Merge - From SIN : 999014679 To SIN : 654457258
Start date - 2008-08-22 09:30:03.420
Create an Activity
Close the Activity
Learner Merge - From SIN : 999011572 To SIN : 647141506
Start date - 2008-08-22 09:30:03.480
Create an Activity
Close the Activity
Learner Merge - From SIN : 917032336 To SIN : 661696161
Start date - 2008-08-22 09:30:03.527
Create an Activity
Close the Activity
(14 row(s) affected)
Done at 2008-08-22 09:30:03.590

The second version looks much better as it is clearly describing what is occurring inside the script.  If you are trying to find out what is going on the second one is so much better.  If you have an auditor on your butt, the second one is so much better. 

Why don't more people do this?  Well, it takes effort.  Typing the characters can consume multiple seconds per statement.  <sarcasm> I mean, over the course of a year this could add up to an hour or more of your valuable time. </sarcasm>

I'd like to thank the SFS team and Janice Pyle for the second example above that clearly shows the advantages of the lowly PRINT statement.

Using a hammer

Have you ever seen Canada's Worst Handyman?  I must admit that I am all thumbs, but, even so, I'm better than these guys.  There are lots of things to look at in this show because so many of the items can be translated into almost any occupation, including the IT field.  One of my favorite things is watching the people, when confronted with a screw, pick up their hammer.  Or using any one of a dozen other tools in the wrong way.

IT people have the same tendency.  We have a problem.  We have a tool.  We try to solve the problem with the tool, regardless of whether or not it is the right tool for the job.  Sometimes you have no tool, just a problem, and in that case almost any help is worth the effort.  Well, maybe not.  Sometimes the effort that the tool saves is totally outweighed by the costs associated with using the tool.  Much like a carpenter that has a plethora of tools to help him do his job, IT people have a variety of tools to help them as well.  Our problem is that there is no simple guide saying "for this problem use this tool and for this problem use this tool".

That's not to say that if we had such a guide people would actually use it.  I mean, look at Canada's Worst Handyman, they were taught certain things and still ignored what was taught.  I have no fear that even if we told people to do X they would still end up doing Y.

I'm not restricting myself to just tools, but methodologies as well.  Some people grab on to a methodology and use it like a life preserver.  Not necessarily a wise thing.  An all or nothing approach is inevitably a bad thing.  Even the Agile Methodology that people either hate or love (there we go with extremes again) understands this.  One of it's principles is:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Essentially, you evaluate what you are doing and adjust if necessary.  I guess my biggest fear is the life preserver mentality for methodologies.  A methodology is a tool and proper use of the tool depends upon proper understanding of the problem.  We sometimes adopt methodologies because they are "cool" or have the "agile" word somewhere in the title.  We don't always understand the problems that we are trying to solve with the methodology and are just so thrilled with using a new tool that we overlook some of the challenges that we have with the tool.

Don't get me wrong, I love new tools, but how many times have you ever used a new tool incorrectly the first time.  If you are going to make a big leap to a new methodology/tool, make sure that you've tested it first and that it actually solves the problems that you need solved.

Quality vs. Speed -- The Agile Way

There has always been a debate over producing a quality application versus an application that is delivered on time.  Indeed, even within the quality camp there are divisions over what exactly constitutes quality.  There have been efforts in the past to merge quality and speed of delivery so that you can get a high quality application in as short a period of time as possible.  The Agile methodology tries to merge these two, to mixed results.

Why mixed results?  Like everything else in life, more than just how you do something determines the end result.  In the case of the Agile methodology (or should I say methodologies) there is more than one way to skin a cat.  The Agile methodology is so different from what people were taught in school that there are three large stumbling blocks that need to be overcome in order for Agile to be truly effective.

• Organization.  There are forces at work in an organization to keep things moving forward.  Plans need to be made, followed and tracked.  The agile methodologies have the idea that "responding to change over following a plan" is a better way to develop software.  And, while I personally believe this, organizations tend to want plans.
• Team. While Agile has come up with some basic philosophies behind how to develop software, the larger the team the more unlikely it is that these philosophies will be effective due to outside pressures: when is going to be done? It is difficult for project managers as they are measured on progress.  Agile, because of its fundamental desire to respond to change, is havoc on plans, at least traditional "we will have this functionality delivered by that date" type of plan.
• Individuals. Everyone comes to a project with baggage:  their old projects and the "we did things this way" ideas.  Getting everyone on the same page, following the same ideas and concepts is very difficult. 

This is not to say that Agile will not work and that you will produce a quality application in less time.  The problem lies in being able to show somebody that this will happen.  Without a proven track record it is hard to show people that it will work, but you need that acceptance so that you can build that track record.  And without the right people in the right places, Agile is never going to fully accepted, fully understood or fully implemented.

Results vs. Methodology

Sometimes it is a story that strikes a chord and makes me write one of these notes.  Sometimes it is an ad on tv.  In this case it is a newsletter from Bruce Schneier, the Chief Security Technology Officer of BT.  He said, that governments should:

...legislate results and not methodologies.

That immediately showed up on my internal radar screen.  While his examples were with regard to things like legislating securing personal data, but not legislating which technologies to use.  Using my own quirky logic I equated this to:

...contract for the end result of the project, not how the project is going to get there ...

When a company puts out an RFP for a piece of work, they routinely say that the winner must "... follow all standards and processes currently in place or that may be put in place during the course of the project...".  We've all seen these as they are not specific to government.  But is it the right thing to do?  Let me go out on a limb and say "no".

In the long run, does it matter if the project team has detailed requirements tracing from the initial conceptual design stage straight through to production?  If you had two projects that delivered the same end result, does it really matter which route they took to get there?  If there are certain deliverables that you want then specify those deliverables in the contract.  Don't specify how you want them created, just that you want them created.

Coming from a programmer background and having been introduced to one of the most document oriented methodologies in existence - Method/1 - I've seen both sides of the coin.  Method/1 was created, to be honest, to minimize liability should the client ever decide to sue.  Dozens, hundreds, thousands of documents are routinely created so that an audit trail could be created in the event that a law suit was launched.  Method/1 was created by auditors, for software development, to be used in the event of a law suit.  Not exactly a glowing endorsement of a methodology.

Other methodologies are created for other purposes, but what it really comes down to is results.  You have certain expectation of results.  And whether the application was written people in blue suits with a document laden process or by a long haired geek who wrote it between episodes of Death Note, it shouldn't really matter, as long as the results required are achieved.

Your Environment is Wrong

"But, it works when I do it, so your environment must be wrong."

When I first heard that comment I took it seriously.  What could be wrong with our environment?  Why couldn't we install the application? 

Not different, because our UAT and Production environments are different than the development environments for a lot of reasons, but wrong.  I searched for something wrong.  Occasionally, I did find something wrong, but as we put more and more applications on to our UAT and Production boxes the number of things that were wrong obviously diminished to the point where their is not a whole heck of a lot that can be wrong.  The differences, however, are still there.  Differences that may prevent the application from installing or working correctly.

• clustered environment - we operate in a clustered environment, so never, ever think that this is the only application that is being installed on this server and never, ever think that it is the only instance of your application which is running
• not a clean environment - the odds are that we've installed your application before - sometimes once or twice before and in some cases a hundred times before - never assume that the environment is "clean" because the odds are that there are remnants of other installs of yours left around
• multiple installs - never assume that just because there is some manual configuration at the end that needs to be done that it is "no big deal" because in our UAT and Production environments we need to do the installs multiple times so that any manual configuration that needs to be done will be done multiple times, giving us plenty of opportunity to make a mistake

"Mistakes"?  Yes, we sometimes make mistakes when manually configuring an application.  Considering that in the UAT and Production environments we are going to be installing and manually configuring your application seven times or more, once we add more servers, this gives us seven chances to make a mistake.  Combine this with multiple releases to UAT and you've got a wonderful opportunity to introduce an unexpected bug, with the most likely problem being that the manual configuration was not saved on a specific machine.  Have the application self-configure itself where possible: prompt for data and complete the installation itself.

Oh, I forgot, you're only going to UAT and Production once, right?

More examples

Going on vacation seems to give me plenty of ammunition for these notes.  The following example of what not to do is based on a conversation with the Day Manager at the Sandman Inn in Calgary West.

My wife and I needed to go to Calgary to drop our kids off with her sister.  They were going to horse riding camp for the week and we thought we'd start it off early by going to a hotel when we got there instead of sleeping at sister's place.  On the Friday before we went we reserved a room at the Calgary West Sandman Inn through the Sandman web site.  We got a room with a King sized bed and, most importantly, a confirmation number.

When we got to the inn on Saturday, however, we were surprised because they did not have the room that we reserved.  They had a room in the same "class", but they did not have any rooms with King sized beds.  I was a little surprised because I thought a confirmation from the hotel chain guaranteed the room was available.  I was told by the clerk at the front desk that the Sandman web site and the computer system at the hotel do not talk to each other.  And that they had been trying all day to get hold of someone to have the web site stop booking rooms.

I was, quite frankly, amazed at this revelation.  Imagine building a reservation system for a hotel chain, but not hooking it into the hotels reservation system!  So what did I do?  Well, there was nothing to do about the room as there were supposedly no other rooms in the hotel.  (They did give me a $3.00 discount for my inconvenience.  The pop in the pop machine was $2.50)  When I got to the room I immediately hooked up to their Internet connection and went to the Sandman web site and filled out a form with a complaint.

After waiting 5 days for a response I called the 1-800 number for reservations.  I got the Manager of Reservations and she was going to call the Calgary West Sandman to see what the scoop was.  Three days later I got a call from the Day Manager.  She confirmed that the systems do not talk to each other and that in order to get the web site to stop reserving rooms the general manager has to fill out a form and send it in.  (He only works 9 - 5 on weekdays, so if something needs to be done outside of this timeframe, oh well.)  By the way, the 1-800 line uses the same system the web site does, so that does not guarantee you the room either.  You need to talk to the hotel itself.  Same thing is to be said with regard to Expedia or Travelocity.

So what do we have?  A hotel chain with thirty five properties that reserves rooms without actually knowing whether or not the room is available and a form based system that is required to be filled out to stop selling rooms.  Does anyone see a problem here?  As I escalate this problem up the Sandman management tree I'll let you know if this is going to be changed.  If not, unless you call the hotel directly, don't bother reserving a room with Sandman.

Planning, ad nauseum

OK, I've talked about planning in the past and I'm going to mention it again, but from a different perspective.  There are many different types of planning for many different types of people:

Programmers. These people need to know what they are doing now and for the rest of the week.  To be honest, while they might like to know what they are doing next week, as long as they are confident that their boss knows, that is probably good enough.

Team Leads.  With programmers reporting to them they need to have a much better grasp of what is currently going on and what is coming up.  A week isn't good enough.  A month may not be good enough depending on the project, so let's say that these people need 6 weeks of planning under their belt.

Project Managers.  Ah, the brains behind the project.  For any particular project that  they are working on, they need to have a good idea of staffing requirements from beginning to end.  Yes, they need to know staffing for the entire project, because if they are short, it is their responsibility to get staff.  If they are going to be over staffed then it is their responsibility to see what other projects need help.

Program Managers.  These people are responsible for the smooth execution of multiple projects within an overall program.  For example, you may have a Program that is composed of a lot of smaller initiatives, all of which are leading to a single vision, much like PASI.  Program Managers need to understand the interrelationship between the different projects and how they impact other projects and the program as a whole.  This sort of planning stretches into the years.

Organization.  Different organizations use different benchmarks, but it is considered a good idea if an organization can tell you what they are going to be doing for the next three calendar years.  This will not be to a great deal of accuracy three years down the road, but the overall plan of the organization, the inter-relationships between ongoing programs and the intended implementation date of various projects should be at least plotted out so that both clients and staff have an idea as to what is currently happening and what the short and longer term goals are.

As you can see, the level of detail goes from the microscopic, tactical information needed for day to day activities to the macroscopic, strategic information needed for long range plans.  Virtually everyone in an organization needs a different level of planning with a different focus, depending upon the roles that they are currently occupying.  The key to all of this, though, is that there needs to be someplace where all of this information can be stored.  Some place where the microscopic and macroscopic views can both find a home.  A place to go when information is needed and a place to go when you have new information.

Peer Review

Why are some people against having their work reviewed by their peers?

Commonly called a "peer review" in academia, the concept revolves around letting other people knowledge in the same field comment on your paper (design, theory or observation) prior to "publishing" it.  Specifically to the IT world a peer review is when a design is reviewed by knowledgeable people in an effort to determine any flaws in the design.  I will grant you that these reviews can be brutal on the ego, but aren't they the right thing to do anyway?

One of our primary objectives in our job should be to produce quality work.  Now, the exact measurements behind defining quality are highly contentious, but in general, the idea is to produce something that accomplishes the objective and does it as efficiently as possible.  One of the keys to creating a quality product is having a good design, one that is as simple as possible yet still satisfies all of the objectives and goals.

A good design.  How do you know if a design is good?  Well, one of the ways that the rest of the world  does it is through design reviews or peer reviews.  When a design has been completed it goes to someone else knowledgeable about the requirements so that they cam see if the design does indeed meet those requirements and does it in an efficient manner.  Seems pretty simple doesn't it?  So why don't we do it more often?

The Enterprise Architecture Working Group (EAWG) did this, to a very limited extent, when they reviewed the application architecture on various projects, but the problem was that at 50,000 feet everything looks blurry.  As a friend of mine used to say "Good from far, but far from good".  Understanding how someone is going to piece things together is important, but the details, the microscopic manipulations that occur on an individual program level are completely oblivious to everyone except for the designer and programmer.

Every critical piece should be peer reviewed to ensure that it is designed properly.  This means that if you are writing something that manipulates funds it needs to be peer reviewed.  If you are writing a mission critical import/export/reporting piece, it needs to be peer reviewed.  If you are writing a security component, it needs to be peer reviewed.

If it's never bee peer reviewed, never had an objective review of it's design, how are you every going to know if you designed something properly?  Is it fear that is stopping you?  Fear of people pointing out mistakes?  Or is it narcissistic in nature?

GUIDS and Clustered Indexes

GUIDS should never ever be used as a clustered index.

Wrong.  GUIDS can be used as a clustered index if they are not an MBUN: meaningless but unique number  I've talked about this before, but it seems to be difficult for people to understand so I'm going to take this slowly and give some examples.  For these examples, let's assume that you are running an online bookstore.

Naturally you want to keep track of orders and you have a table in SQL Server that does just this.  For each order you assign a GUID.  Should you cluster on this GUID?  Probably not and the reason is that the GUID, in this table, is meaningless.  It has no context with any other data and as a result it is truly an MBUN.  Indeed, you would probably want to sort on the customer number instead as you are more likely to retrieve all of a customers orders than you are sequential GUIDs.

Hmmmm.  We're keeping track of orders, but not the items on the orders.  Let's have an OrderItem table.  In order to connect the rows in the OrderItem table with the Order table we would probably link them via the GUID that you assigned to the Order itself.  To show the individual items within the order we would need to create another column, let's call it an ItemNumber.  This could be a GUID, but it makes more sense just to make it an integer.  For this table the clustered index should be the Order GUID and the ItemNumber.

Why?  Well,. when we create a clustered index we are telling the database that we want records that appear next to each other in the index to actually be stored next to each on the disk.  In the first example, using the GUID made no sense as there was no reason for the records to appear next to each other.  In the second example it makes perfect sense to use the GUID because we want all of the items for an order to appear next to each other on the disk.  This speeds up the retrieval of data and makes more effective use of memory.  In the second case the GUID is no longer an MBUN as it actually has meaning: it is the order number that we are tracking.

So you see, if the GUID has meaning it can be part of the clustered index.  If the GUID has no meaning, don't put it in the clustered index.  Any questions?

Database Design

How much time do you spend designing your database?  Twenty percent of your overall design time?  Ten percent?  Five percent?  Two percent?  Design?

Well, the magic number should be 15%, or thereabouts, and can be determined by the juxtaposition of the solar days as they are related to the cardinality of the days of the week, the overage as attributed to project management style and the variability imposed by Pantone colours as they are blended on a surreal palette of olives and daisies.  Just kidding, there is no magic number.

Good database developers require less time to develop a well structured and functional database.  A smaller application with a more focused purpose requires less time as do re-writes of existing systems and applications for which there is a wealth of knowledge available.  Conversely, applications that are new to the organization, new to the designers or are complex in nature require considerably more time in the design phase.

Database design, contrary to what many people believe (and contrary to what I've seen implemented) is a complex task.  Understanding the data access paths that the application takes, the appropriate indices to create and how to optimize that access is difficult for anything more complex than a simple inventory system.  Why?  Well, if you have a column that contains two values "YES" and "NO", with approximately 100 "NO" for every "YES",  and you create an index on that column, will SQL Server use it?  That depends on how many rows are in the table.  SQL Server will make the determination, at some point, that the index in its current form is useless.  How about if the column contains four values, will that help?  It will delay the inevitable, but regardless, SQL Server will, at some point, say "To heck with it" and not use the index.  The cardinality of the columns in an index is as important as the index itself, yet very few people understand this rule and as a result, indexes are created that may work well today or tomorrow or even for months.  One day, however, the house of cards will collapse.

Take some extra time and work on the design up front and I'm sure you'll be pleasantly surprised by how little pain you feel when it goes in.

Interrupted ... Again

OK, now that you've taken a look at your own life, realized that you are totally unproductive due to interruptions, what can you do to become productive again?

Work at home, or at least, away from the office..  Seriously.  Talk to your boss and see if there is a way for you to spend every second Monday/Tuesday/etc. at home to do your work.  For some of you this isn't going to work due to the work that you do.  For others, those that producing tomes, need some time to plan the next great project or even, goodness gracious, catch up on that documentation you forgot to do - six months ago - spending some solid time working on these items can be a tremendous benefit.  Now, you don't necessarily have to be at home.  Some people find that working at home is not as rewarding as it could be due to hundreds of other distractions.  An alternative location is the key.  Whether this means you book a meeting room for all day, find an abandoned office that no one is using or even use the desk/office of someone who is on vacation, getting away from your desk, the spot where people find you, is key.

Another step?  Flush your Crackberry down the toilet.  Figuratively.  Unclip it from your belt and put it in your briefcase/deskl/backpack and bring it out every hour and see what you missed.  The constant buzzing of new messages, the incessant weight just totally p***** me off sometimes and I need to just get away from it.  Why check every hour?  Well, research has shown that people work most effectively if they take a five minute break every hour.  Go to the bathroom, get something to drink, check your voicemail, etc.  Just remember that this is a five minute break and not the start of five minutes of different work.

Turn off email notification (thanks Garth).  You know that little window that comes up and says "You've got new mail and here is the subject line and who it is from..."  Turn that off or even shutdown Outlook when you don't need it.  Just remember the five minute rule once an away.

Take a look at sites like Lifehacker (Tech tricks, tips and downloads for getting things done).  (On your spare time, of course)  This site contains some useful and some useless tips on how to keep organized and get your life back together.

Unfortunately, for some of the people reading this note none of the solutions are going to work.  You know who you are:  support personnel.  Those people who support other people and who, quite frankly, need to be in constant touch because their clients need help.  An email about the best way to re-engineer a COM+ application can be done later.  An email about a crashed system needs to be handled NOW.  So, while many of you can take advantage of these and other tips, others ... oops, have to go, later.

Environment Building on ten cents a day ....

I have been giving project team members a hard time about advance notice.  You know what I mean, giving the Deployment Team more than 15 minutes notice that they need a deployment done.  Well, I'm not going to bug developers in this note, I'm heading higher up the food chain for the project managers.

I've talked about how we are short on space and short on power.  Apparently many project managers (and I am talking about a good dozen or more at the moment) think that these facts don't apply to them because they are special.  You know what?  Everyone is special.  When there are so many special projects, the appellation is meaningless.  So, let's talk turkey.

I want a performance test environment, and I need it next Monday.  OK, to be honest I haven't heard this .. too often.  Most projects give us a lot more warning, although sometimes there is not a lot I can do for them.  (Did you miss my notes on not having space and being electrically challenged?)  Let's take a look at a typical application and see what a performance test environment entails.  A smallish application that access Stakeholder Registry.

So, what do we need for this performance test environment?  Well, we could go with clustered web servers, but let's give ourselves some leeway and just pick one.  We then need a SQL Server box for SHR and a SQL Server box for the application database.  Wait, in Production these are SAN attached, should we duplicate that?  Well, that would mean buying another SAN and the storage and configuring it, etc., so for the purposes of our example let's not go there.  Oh, wait, in order for this to be even remotely representative we need to emulate the existing servers, not buy new ones.  So now we need Dell 6850s, but they're not made anymore.  Have no fear, eBay is here!!  Oh, darn, that reminds me, we need to get an Dell 1850 for the web server.  I hope we haven't gotten rid of all of them yet. 

OK, we done yet?  No?  What else do we need?  Well, if we are using SHR we need to hook in ISM, which requires another license for SSA-NAME3 as well as a Dell 2650.  (Remember eBay?  It's your friend.)  Here is an interesting fact for you:  If you restore the SHR database, you need to rebuild the database that SSA uses for searching.  This probably isn't more than a six hour job these days, but it might be longer.  I guess this means that if you run a test at 9:00 AM in the morning we might be able to rebuild everything for a 3:30 PM test.

I'm pretty sure that you're not going to be using the stopwatch facility on your Timex watch to record everything, so that means we need some tools for testing and recording.  Let's assume that we have these, but the tools we might need are Intercept Studio licenses (aka Avicode) to pinpoint some performance issues for us.  These are not necessary, but they have proven themselves to be quite valuable in the past.

There are some other infrastructure things like backup tapes, disk space for intermediate backups

So what do we have?  Four computers that need to be purchased, refurbished and installed in a computer room where we are already short on space and power.  Additional licenses for software we don't have.  A lot of effort on part of a lot of different people just to get things in place.  Just think what would happen if we needed PRS (Oracle server and license) or perhaps CGES (two servers and SharePoint) or, heaven forbid, all of those, plus more!!!!

Or, *shudder*, you have multiple projects coming to you at the same time with the identical requirements.  Welcome to my world.

Setting up a performance testing environment is very expensive in terms of hardware, software and sheer manpower needed to set it up and keep it running.  The question is, what are your really trying to test?

Don't get me wrong, I like the fact that you want to performance test.  I guess what we really need to understand is what you are trying to test.

Loss of Power

Electricity.  It lights our houses, cooks our food and allows us to watch and listen to events that occur half way around the globe.  It powers our watches and allows us to pick up a phone and call someone a thousand miles away.  It powers our lives.

It powers our computers.

The idea was that computers would become smaller, cheaper, more efficient at doing work for us.  Guess what?  In order to do this they consume more power.  The average Dell "high end" server is smaller, faster, more capable of doing fantastic things, but it also consumes about 20% more energy than its predecessor.  Unlike everything else that becomes smaller, faster, better, the power consumption of computers has gone up. People have always expected a higher density for server rooms (more computers in the same space) and server rooms that are constantly growing in size, but for the same number of computers to chew up more power?  In many respects that slipped by peoples radar.

Server rooms have always been built with some excess capacity and ours is no problem.  Indeed, if our server racks were filled to the brim with hardware from three years ago, I think we wouldn't be having this conversation.  But, power requirements for an individual server have gone up and as a result our ability to add more hardware is limited by the capacity of our server room.  We're short on capacity.  Indeed, I don't think we can add all of the new hardware we have without seriously overloading every circuit we have. 

Let's look at some examples and cry:

Example 1:  a Dell 2650 consumes 357 watts while a Dell 2850 consumes 524 watts.  That is almost a 50% increase, even though the amount of space consumed is the same. 

Example 2: a Dell 2950 consumes 307 watts and takes up 2 U of space, approximately 154 watts per U.  A Dell 1850 consumes 444 watts in a single U.  So while we consume half the space we almost triple the power consumption per U.

Newer machines are greener, consuming less power per U of space, but we don't have a lot of the latest generation of machines, those that are efficient with their use of electricity.  In the past we've upgraded machines to give clients the power they need and we've accidentally hampered out ability to expand.  The Dell '8' series (1850/2850) are huge millstones around our neck as we're trying to stay afloat in the electricity ocean.  So what are we doing?  Replacing as many 1850s/2850s as we can.  (See yesterdays note on virtualizing, evergreening and replacing).

So, we don't have a lot of room and the servers that we do have are consuming huge amounts of power.  Can't get any worse, can it?

No space at the inn

OK, for a number of months now I have been using the explanation "Sorry, but we have no room at fulfill that request."  Many of you have been the recipient of this sort of email and you may be thinking that I have just been using this as an excuse.  While there are many things that impact the "room" that is available, let me just talk about physical room today.

Within our server room we have a number of racks.  These racks are industry standard racks for holding rack mounted servers.  (Hence the name similarity).  These racks contain 42 U of space. and we have 17 of these racks in place. That's a lot of space.  We are also about 80% full. Theoretically we have enough room for about 70 servers or so, based our own current 2 U standard server.

That was theoretically. In practice we do not have this much usable space. Due to the fact that the older servers had a wide range of sizes, the fact that we put more than just servers in the racks (storage enclosures, load balancers, etc), the amount of usable space where we can install our standard server is about 72 U in size; or about 36 servers. This is pretty close to the number of servers that we need to install, but this gives us absolutely no room for growth. (Based upon meetings I've had recently, we are going to be growing.)

So, how do we free up space? Virtual machines.

We have converted about 40 machines to the virtual variety in the past few months. We want/need, to do more and we are going to that. My personal goal is to free up another 50 U of space in the next couple of months. so, that we can have that room for growth. We need some help from Microsoft at this point as we are having problems with one of their "P2V" (physical to virtual) tools that is making this transition a lot easier.  They have managed to replicate our problem and are working on a solution.

We are currently in the process of mapping out precisely what is in our server room, which rack it sits in and exactly where in the rack it is taking up space.  Based upon this very detailed map we will be able to plan out where new machines should go and what we need to do in order to consolidate some of our fragmented free space.  (Much like a disk defragmenter, but on a much larger scale.)

This is not the final solution, however, as a larger battle still looms and we will discuss that tomorrow.

Budgeting

I'm not really a big fan of budgets.  My wife and I try to make them and then we blow them out of the water within a month.  For projects, however, budgets are important as they lay down some limitations on what can and cannot be done.  Sometimes, however, we don't set the budget properly.  Let me give you an example:

You walk into your bosses office and they tell you that they have a new project for you.  They tell you what the end result of the project is going to be and what your budget for the project is.  The problem is that there has not been any detailed analysis done so the size of the project is totally unknown, but not the budget.

I'm sure that most of you have been there before, even in a somewhat different perspective.  Imagine if you will, going up to your team lead and have them tell you about a new feature you are going to implement -- in one weeks time.

it is important to understand that budget does not necessarily limit itself to dollars as budget can also be equated as people time.  If you have X dollars that equates to Y days of effort. Whether you use the top down approach to budgeting, the bottom up approach or the throw a dart at the dart board approach, you need to understand what you are going to build and how you are going to build it.  Once you figure out that you can come up with an estimate.  Based on that estimate you also include a certain percentage of time for documentation, testing, UAT support, project management, internal technimcal support and deployment time and even post implementation support. 

I know, you normally don't do this.  Maybe that explains why you feel so stressed about being late?

Process and DJ

I am a great believer in following "the law", not because it's the "right" thing to do, but because without a standard basis upon which to rest our morals and beliefs we descend into anarchy.  But you know what?  Going too far the other way is just as bad.

This is also true for IT projects.  For instance, the Project Management Institute offers all sorts of guidance on how to manage projects.  The have produced a Book of Knowledge that contains much useful information.  BUT, it is not the understanding of this information that makes a project manager effective, nor is it merely have the PMP certification that makes a project manager good at his job, it is understanding when to use the tools at their disposal and the reasons for doing so.

Over-managing a project is almost as bad as under-managing.  (Depending upon your perspective it may actually be worse, but I digress.)  Over-managing puts too much rigor into a process that does not require that rigor and that is where the skill of a good project manager comes into play.  Like a poker play you need to know "when to hold 'em, know when to fold 'em, kown when to walk away".  The effective application of the tool is the difference between a good project manager and someone who just follows the rules.

Am I being too harsh on project managers?  Maybe, but considering the wide range of project styles that I have seen and been victim to, probably not.  The project manager has more influence over the end result of the project than anyone else.  Don't influence in the wrong direction.  Give the team the freedom to innovate, the freedom to get their job done while at the same time putting in place the structure necessary to do that.

Picking on Managers ... again

It's been said that I've been picking on programmers too much recently, so I thought I would skip the low hanging fruit and go after something higher up on the tree.  Let's start with one of my favourite slogans:

Failure to plan on your part does not constitute an emergency on my part.

A project manager has a specific role on a project team:  planning and coordinating the activities of the team members.  On larger projects this may actually be the delegation of this responsibility to other members of the team.  On a large 75+ person project I worked on there was one Project Manager who ran the entire project, but four other managers who actually did the planning.  Please note that this 75+ person project had a total of 5 managers.  There were a number of team leads on the project as well bringing down the ratio to 1 to 5 for supervisors to staff.  This is by many standards a fairly low ratio, but it was typical in the organization I was with, with many of the supervisors spending only a portion of their time supervising and the rest of the time working on different tasks that needed to be done.

Each team lead knew what was expected of their team in the short term and what the long term objectives were.  Every week there were meetings amongst team leads and their respective manager and every week the managers got together to discuss the status of the project.

This same type of organization is applicable in smaller projects.  Planning is one of the key factors:  know what needs to be done now and in the future.  Too often have I seen and read about projects that do not have a clear understanding of what is required and the usual impact of this is larger, more complex projects than is required and the resulting cost and timetable impacts this has.

Worst of all, from my perspective right now, this has a tremendous impact on the Deployment Team.  Too often have we had requests like "...but the user needs it now, do we have to give you five minutes advance notice? ..." or "... it's just a small change, but we need to recompile everything and send you a complete package, if we get it done by midnight can you have it in place by 12:01? ...".  T

he reason we ask for advance notice on migrations is so that we have the opportunity to plan our own work.  If you don't give us advance notice we can't plan and that annoys me.  And everyone knows what happens when I get annoyed.

It's not always a technical solution ...

One of the interesting things I learned at NAIT was that technology is not always the best solution.  Sometimes simple, every day solutions, non-technological solutions, are actually better.  For instance:

I take the bus to and from work.  While on the bus if I get a brilliant idea, or even a not-so-brilliant idea for a daily migration note I need to capture the idea so that I can later cogitate on it and then spew it forth.  I carry with me, a Blackberry.  Seems like the appropriate thing would be to type it into the Blackberry using my thumbs, but I've found that, for me, it is the wrong solution.  A pen and a small notepad are what I need.  Faster.  More intuitive.  And I'm less prone to correct my spelling.  Later I can look at it and determine if I want to save it electronically.  Initially though, paper is king.

While I consider myself a geek, I also consider myself to be open to alternatives and, quite frankly, paper is a valid alternative for me.  While I like electronic toys and I like building little applications that do something, I also understand that sometimes pen and paper, a new process, or even revised expectations, are all valid solutions to a problem.  What we, as IT professionals, need to understand is that when a client comes to us with a problem it is in our best interest to come up with a proper solution, regardless of what technology we are or are not using.

If you talk to a user and persuade them to change their process to resolve a problem, rather than create an entirely new system, then you will have done everyone a service.  Over fifteen years ago I was asked to build a system for a client that did some inventory work for them.  Our recommendation was that they change one of their existing processes to handle the requirement, but the client was adamant that he wanted a new computer system built.  We built it in VB4 in about 20 hours.  It didn't have any error handling and the interface was kind of clunky, but it did everything he wanted.  Within two weeks of rolling it out to his staff they changed their business process and dumped the application.  Sometimes you need to experience the pain to understand the truth.

Casual clothes = casual work?

I was watching a report on CNN that talked about "Casual Fridays" in England - London to be specific - and how people did not want to "dress down" for work.  The reason that was given was that having a dress code made it simpler to know what was required and actually made things easier for the worker.

There is another underlying fear, as made quite evident from my days with Andersen Consulting, that if employees dress casually, they will take their work casually and not produce the same quality of work they would otherwise.  While one of the unwritten rules was that you were to follow your clients standards with regard to what you wore, another unwritten rule was that you would wear a tie at all times.  So, while you could dress casually (i.e. take your jacket off), too casually was a no-no.

Being part of the generation that grew up with suits and has subsequently ushered in the "Casual-era", I've seen both sides of the coin.  While I believe that going casual has reduced some of the discipline that is required, I believe it has been more than made up for by the atmosphere in which we work.  A suit and tie impose a certain level of discipline when you're working and losing those pieces of clothing has, apparently, reduced the level of discipline that people seem to have. (Please note that there is a limit to being casual, a limit that some people just don't understand.)

However, I also think that the work atmosphere now is more conducive to change, more willing to accept new challenges and new ideas and, well, more fun than it used to be.  The fact that I don't wear a suit and tie makes me more comfortable at work and, as a result, makes me more attentive and able to focus on a problem.  (Who can fix something when you're convinced that the next time you cough your tie is going to spontaneously tighten itself around your throat and stop you from breathing?)

Back in the suit and tie world the Internet would never have been a success: it breeds contempt for boring; offers success for being different; allows people to do things in amazingly different ways; and was built by people without ties.

The kids nowadays ...

I was talking with someone the other day and we were both commenting on how we both had high expectations of developers.  Perhaps too high.

When I was just starting out in this business the mainframe was still the big thing.  I remember going to the launch of the IBM PC that IBM had in town and thinking that if they did this right IBM had a gold mine on their hands.  Well, they blew it, but the industry did not let the personal computer drop and the world was changed.  I learned lots of stuff on the mainframe and then applied it to the PC world.  I learned how to debug properly when you could only do one compile per day.  I learned how to design things properly when you had 4K of memory to use for your application before you had to start worrying about addressing problems.  I learned how to adapt the mainframe mindset and use it in an intelligent fashion on the PC.

Kids these days don't do that.  Schools don't teach it and the kids have no opportunity to learn what I learned.  While I may have owned my first computer before some of you were born (March 15, 1981 was when I bought my Apple ][+) I've had so much more varied opportunities.  I've worked on large mainframe applications, large PC applications, large web applications,  I've worked on machines were space was at a premium, both disk space and memory and you were challenged to be as efficient as possible.

The kids these days are challenged to be efficient.  If it works on their dekstop then it must work on the server for hundreds of people and, if not, it's the deployment team that needs to make it work.

I think it's a shame that mainframes are being relegated to the back room because I think the lessons that they taught are still applicable today, but there is an entire generation that will not learn those lessons.

Integrity

One thing that continually amazes me about Hollywood, I guess about their writers, is that they sometimes choose the most idiotic, but convenient plot devices because it saves them the effort of actually thinking about something.  In this case what I am talking about is the infamous "back door". 

You know what I'm talking about:  a hacker mysteriously manages to access a top secret application through a simple userid and password that he hard coded into the application when he built five years ago.  You can imagine the sort of code that he put into the application:

if userid = "Fred" and password = "flintstone" then
access = "SuperAdmin"
endif

The trouble is most access systems work this way.  Any access system for a top secret application is not going to use something that the programmer wrote because they want it to be more secure than what they're paying this guy who is later going to hack into their system.

But the thing that annoys me most is the idea that this is commonplace in the IT industry.  I have never written such code for a production application, neither do I know of anyone who has written such code.  I've been involved with systems that dealt with hundreds of millions of dollars in payments, yet there were no secret back doors into the system.  I am annoyed by the fact that Hollywood, and the image they present of the IT industry, is that of a bunch of cowboys with no integrity.  Why don't they show social engineering in action?

Hacker: Excuse me, ma'am, but I'm from the IT department and we've noticed that there is a problem with your ID.  Could you please tell me your password and when we've resolved the problem we'll reset your password and you can go back to your job.

Social engineering is by far the easiest way to gain access to a system because people fall for the lines quite easily.  If you don't believe me just look around and you'll see stories about hundreds, thousands of people who have fallen for the Nigerian 419 scam.  So instead of showing how innocent people can be easily duped they decide to show that the bad guy was evil long before the movie started.  Yeah, like I'm evil.

Quality

My wife is constantly reminding our kids that they need to follow all of the directions when doing their projects and make sure that all of the objectives of the project are met.  But it that quality? I guess that really depends on whether you want to measure quality during the build process or whether you want to measure quality as at the end of the process.  I'm going to argue that in the IT field we need to measure it both ways.

Quality at the end of the process - the delivery of the application to the business client - is very important as this is how IT is predominantly measured by the client:  were my needs met and if they were met how well were they met?  If you don't meet the needs of the client it doesn't matter whether you built it ahead of schedule and under budget because the client is not going to be satisfied.  We don't do JD Power & Associates surveys of all of our work, but maybe we should.

Quality during the build isn't something that the client concerns themselves with, that much, but it has a direct impact on time and budget.  Fewer mistakes during the build process means a faster time to finish and a lower cost.  So, while it may not seem important it has a vital role in the cost of the product at the end of the process.  To be honest, this is where most projects fail to perform any measuring and this is where the most important information about the build process can be found.  Measuring the end result is fine for final reports and is something that you can parade around on your business card, but it doesn't tell you how well the job was done on a daily basis, just on an overall basis.

In my opinion, understanding how well the project is doing on a daily basis, the quality of the work that each person creates each day, is just as important as the end result.  But, hey, that's just my perspective.

Death March, part two

Let's talk about a death march again.  My note from last week seems to have stirred up a little bit of angst amongst the crowd.  Let's talk about some of the other characteristics of a true death march.

The project that I worked on had a requirement for a lot of people to be working on the project simultaneously.  To that end we converted a large room on the floor below us to a large "pod".  Affectionately called "The Pit", it housed twenty people in an area that should not have housed that many people.  To put it in a context that everyone could appreciate, there was approximately 22 square feet of space per person.  Go into the bathroom where you work and go into one of the bathroom stalls.  Imagine it about a foot wider and that is the amount of space that each person had.  You couldn't push your chair away from the desk if the person behind you had already done that as there wasn't enough room.

Accenture (Andersen Consulting at that point in time) had a strict dress code policy.  Shirt and tie with dress pants and dress shoes.  Always.  The Pit was not designed for twenty heat generating computer, twenty heat generating monitors and twenty heat generating people for extended periods of time.  With a southern exposure the room got warm.  Unbearably warm.  The thermometer didn't register above forty Celsius, so after the temperature went above that for a number of days in a row the dress code was relaxed, only for those that spent the entire day in the room.  (By the way, the highest temperature recorded in the room was fifty Celsius by a thermometer at my desk.)

My boss at that time once gave me supreme heck for leaving the building to get something to eat.  He expected me to stay in the building from 7:00 AM to 7:00 PM, even though he worked fewer hours and had air conditioning.

Lack of respect for the individual is a key component of a death march.  If you care about the people around you, if your supervisor cares about how things are going, if the organization as a whole understands what is going on and tries to alleviate some of the stress (paint ball parties, laser tag events, movie nights, pizza lunches) then what you've got is an unfortunate set of circumstances, not a death march.  Some projects have periods of high stress and high workload and that is, unfortunately, a common occurrence in our line of work.  How we and our associates handle the situation is what makes it a death march or a learning experience.

Is it just me or ...

Is it just me or do kids these days have a false sense of size and a false sense of what they need?  Maybe it's just me, but let me give you some examples.

• This developer came up to me and said that he was working on a huge project, it had over 20,000 lines of code.  When I was working for the WCB I had to recompile every single COBOL program that we had because we changed a copybook that was used in every single program.  I had to recompile 1850 programs with an average length of 20,000 lines of code and I had to do this in four separate environments, including production.  I recompiled approximately 150,000,000 lines of code.  Twenty thousand lines of code seems small by comparison.
• "I've got a huge database and I'm not sure that the database can handle something this size."  The database in question could fit on a single DVD - less than 4GB of data and contained no table with more than a million rows.  Compare this with a financial disbursement table that grew more than 10 million rows per year and had a history table that had grew by 20 million rows per year.
• "You guys need a bigger server.  My app requires four quad core processors."  A properly written application does not need four quad core processors.  I've been part of a system that had 1200 users, over 400 of which were considered "power users", serviced by two, dual 700MHz processor boxes.   Less processing power than a single web server we have today.  Considerably less and yet our applications push the current boxes to the limit for far fewer users!!!
• "My application has a huge user base:  200,000 people per year will access it."  So?  Let's say that people will access your application only on business days, and then, only between 8:00 AM and 5:00 PM.  That leaves 100 people accessing your system every hour.  That's not a big application.  It my handle work for a lot of users, but it's not a big application, particularly on the hardware we're running.

OK, That being said, it may not be the kids fault.  He could just be repeating what he is being told.  He might never have experienced anything Enterprise-scale.  He might just be a rookie.  Let's give the kid the benefit of the doubt, educate him on what an enterprise scale actually consists of and then watch him go back to his desk, thankful that he's not working on one.

Road Kill

Road Kill.  You know, the dead animals you see on the highway that had the unfortunate luck to step out in front of a speeding car/truck and get hit.  It's not a pretty sight.  Over the course of the past 20+ years in IT I have felt like road kill a number of times.

Most projects have timelines that they need to live by.  Sometimes these timelines are arbitrary dates, sometimes they are based on business processes and cycles that need to be adhered to and occasionally, every so often, they are legislated dates that have a tremendous social and political implication if they are not met.  In those rare cases when you are working on a project like this it might seem like a death march, but it doesn't have to be.  This is where the role of Project Manager is particularly difficult:  how do you convince the staff to continue working on this project while at the same time telling the business client that the dates seem a little unrealistic?

I've been involved in a number of projects where the target date was just totally unrealistic.  It was way too aggressive for the amount of work that needed to be done with the resources that were available.  Putting more resources wasn't going to work due to the start up effort associated with new staff and the decrease in productivity of existing staff as they helped out the newbie.  Working longer hours wasn't going to help because the team was already working 50%+ overtime.  This is where project managers bring out the Successories posters and try to motivate the staff with short little quotes.  I've even had project managers blatantly lie to me about the project:  "You'll only have to work overtime for the next four weeks and then we're back on schedule"  (It took six months) 

At the end of the six months I felt like road kill.  Most of the staff felt like road kill.  This didn't have to happen.  This didn't have to turn out the way it did.  The timeline was set in place by the client but, you know what?  It was flexible.  They wanted a specific implementation date so that they could start showing a return on investment for the application.  They were doing it so that it would have a bigger impact on the current years year end report.  Our management, so intent on pleasing the client, burnt out a considerable number of staff and presented an image to the local IT community of a bad place to work.

Sometimes you need to step back and take a look at what you're doing and see if it makes sense.  In my case it didn't.  Your case may be different.  If there isn't a hard and fast deadline that you need to meet, err on the side of the people and adjust the schedule.  Losing staff may be worse for your project than implementing late.

Diminishing Returns

What is the optimal size of a project team?

What is the optimal size of a meeting?

These questions have plagued people for years.  The egotists/dictators amongst the crowd will think that the best number is one.  If there is no one else to discuss things with then you will always get your way.  Unfortunately, getting your way is not always the best solution for the client, so we need to have at least two people.  Indeed, for most projects there will be multiple people involved ranging from project managers to developers to business analysts to the actual clients themselves.  So what's the answer?

I was reading an email this morning which mentioned getting business analysts, project managers, clients, Directors, project teams and the Deployment Team involved in a "little" project.  I almost collapsed when I read that part of the note, fearing a room that contained twenty five people, all of whom wanted their voices heard and none of whom were willing to budge.  I was quite relieved when I read the next couple of sentences and realized that the writer understood the "Law of Diminishing Returns" (The larger the group involved, the less likely you are to get consensus.) and had provided the solution:  representatives.  (Hey, this kind of sounds like democracy in action, doesn't it?)

Trim down the size of the meeting to a smaller number (research has shown that five to seven people is the best, with seven being a fairly firm upper limit) and have the people in the group represent various segments of the prospective user groups.  For larger projects the same sort of segmentation and representation also works wonderfully.  Split the overall project into small teams working on a specific problem.  Have related teams grouped together where a representative from each team will have a voice in a larger collaborative group.  This has worked well for major companies like Microsoft and IBM and consulting firms like Accenture.

The root concept is the same, however, keep teams relatively small in order to reduce the number of nodes of communication that each person is subject to and to improve the the overall feeling of ownership.

Adjusting to Changes

Sometimes when we create a new application and deploy it to our clients it falls down with a resounding thud and we wonder what went wrong.  It did exactly what they asked for and it does it in the same way as their existing application, but something falls flat.  What did we do wrong?

There are so many answers to this question that I could spend the rest of my life describing them, but let me highlight some of the things that are probably big contributors to the problem:

• Lack of Evangelists.  On larger projects what you need to do is really engage the prospective clients and get them really excited about the changes.  One of the ways to do this is to recruit members of the client group to be evangelists".  Apple first made this term popular when they started hiring evangelists (marketing people) for their hardware and software.  While you may not think this has a big impact, believe me, it honestly does.  By the time the application rolls out you have people in the client group who have seen the application, know how to use the application and are enthusiastic about it.  This really goes a long way to making the roll out a success
• Lack of fun.  OK, after you've finished scratching your head, think of this:  what is the worst thing about using a new piece of software?  Finding the bugs.  But what if you made finding bugs fun?  What if you gave out prizes or even balloons, for finding bugs?  A friend of mine, when her project went into production, had some members of her project team wandering the floors of the users.  They answered questions and, more importantly, gave people helium balloons when they found a bug.  It became a fun competition to discover the bugs.  Rather than being dismayed, the users were happy.  (Needless to say it was important for the project team to fix the bugs as soon as possible so that they saw results from their discoveries.)
• Lack of Communication.  Newsletters are so '90s, but they work.  Whether it is a formal newsletter, or a web page or even an email that is sent out periodically, keeping the clients informed of the progress of the project and what is coming down the pipe is so important.  By being involved in the process, even from a passive perspective, the client is much more likely to embrace the changes and accept the new application.

None of these things are new.  Projects around the world have done them for years and have done them successfully.  The biggest problem is knowing when you need to use any of these, or a hundred more methods, for your project.

Consistency and pudding

Consistency is an often overlooked attribute.

Consistency is what keeps many franchises open - if the food is good at one location, and the franchise is consistent in terms of quality, then every location should have good food.  Consistency is important in some food - have you ever tried pudding that was not the same consistency?  Have you ever wondered what was in there?  We rely upon consistency for driving - ever seen a set of lights go from Green to Red to Yellow?  (Colour blind people need not answer this question.)

Consistency is valuable in many aspects of an organization.  A consistent approach to resolving issues normally results in the average time of an issue decreasing.  Consistency in how projects are planned allows both upper management and staff a better understanding of what is currently on the go and what is outstanding.

Consistency in projects can also increase the quality of a project.  For instance, if people consistently run unit tests when they've made changes to an application the odds of an unexpected bug slipping in are greatly reduced.  Consistency in processes between projects allows for developers to go from one project to another with greater ease and allows for upper management to aggregate data amongst the projects as the data is gathered and reported in a consistent manner.

The new version of TFS (Visual Studio Team Foundation Server 2008) is a tool which is going to help us gain some consistency in our projects.  With a common build platform, a common defect tracking system, common processes, common work flows, and common reporting, the ability to aggregate data in a meaningful manner is greatly simplified.  The ability for someone to switch projects is greatly increased as they already know the infrastructure and the processes that are in place.  With consistency we will spend less time trying to customize something into a form that no one else uses and we will spend more time resolving the fundamental business issues that we are presented with.

Internal Expectations

I've talked in the past about expectation setting with your users and how that is a good thing.  Internal to ITM, however, we have areas that provide support to other areas and there are expectations in this area as well.

Recently I had the privilege of being Yvonne's backup with regard to database administration work.  Most of it has been in the area of DeCo requests and the expectation of the end users has been, well, stunning, to say the least.  While some people did not seem to have a specific timeline in mind with regard to when the data fix needed to be done, other people, heck even the same people but for a different request, seem to have the expectation that if thirty minutes had passed since the request had been submitted then the request should be completed!!!

Is this reasonable?  Well, whether or not I think it is reasonable, it is what the end users are expecting because that is the level of service that Yvonne has been providing.  This makes it very difficult for anyone trying to even temporarily fill in for Yvonne as the odds are that this person is not trying to do Yvonne's job, just cover for her, which means that they still have another job that they have to do.  Unfortunately there is no simple solution to this as the expectations have been set and trying to claw them back now just isn't going to work.  In the long term we are looking at a solution that will allow for more self service in terms of data fixes while still maintaining the high level of auditability.  (Potentially higher, but if you want self serve then you will have to live with it.  :-) )

Conversely, however, there is an expectation on part of the DBA - OK, really just me in Yvonne's stead - that the SQL submitted for running actually, well, runs successfully!!!!I  If you submit a piece of SQL to be run as a data fix, and that SQL does not run successfully, do not expect the DBA to fix it for you.  Instead, what I am advocating to Yvonne, and what I will be following while in her place, is that if the SQL fails, immediately stop the deployment and reject the deployment request.  I mean, if we were implementing an application and the installation package failed, we would notify the coordinator and say fix it.  Why should we do something different with SQL than with an application?

Service Level Agreement

My apologies in advance if I seem a little crabby/cranky/ornery this past week but I've been Yvonne's backup, I've been sick and my darn sump pump stopped working just long enough to soak the carpet.  (Wet carpet sure weighs a lot.)  When our sump pump went we called up a plumber based on a friends recommendation.  He showed up exactly when he told my wife he would.  He advertised a service and lived up to that agreement.  He essentially met his SLA (Service Level Agreement).

Regardless of what we might believe, we all have certain SLAs as part of our role in an organization, our role on a project, our role as husband/wife/companion, as a animal owner, as a coach, as a friend, as a consumer, as an eBay entrepreneur, etc.  We have so many different SLAs that even if we tried we couldn't identify them all.  If the role you have in an organization is based on providing services to others, then you definitely have SLAs in place even though they may not be formal.

For instance, it is generally expected by the Development Teams that if they have a problem they send it to the "_EDC-Deployment Team" list and someone will take a look at it.  While this is indeed the pattern that we have been asking people to adopt, we have not specified the response time for such a request. 

Let's assume that the request is five minutes work.  Does this mean that you should expect to receive a reply in five minutes?  What happens if the person who needs to do this has just gone to a meeting?  What if they had to go to the washroom?  OK, maybe you change your expectation.  Maybe you change it to:  receive a response from the Deployment Team in twenty minutes even if they haven't actually done the work.  What about lunch?  Is someone from the Deployment Team supposed to be constantly checking email during lunch?  What about a staff meeting?  Is it going to be possible to have a staff meeting with the members of the Deployment Team if someone is constantly answering emails and responding to requests?  Do we implement tiers of service whereby Production always gets responded to within minutes (5?  10?) and the other environments on a "when available"?  But what if someone is doing some testing in UAT so that they can confirm that a hot fix for Production is working?  If this is Production related does it fall under the Production response time or the other environment service level?

While my wife and I were thrilled that the plumber met his service level, I could envision dozens of legitimate reasons why he wouldn't be able to do so.  And yet, even though those reasons would be perfectly valid, we both know that I'd be cranky that he missed his appointment, because, after all, he affected me.

Why am I bringing this up?  Well, everyone reading this not can probably tell his or her neighbour about the time the Deployment Team took x minutes to do y seconds worth of work.  While I sympathize with you and promise that we'll do better, please realize that there may be extenuating circumstances:  we may be short staffed due to illnesses, we may be with another client helping them or something as mundane as no one has had a bite to eight in 10 hours and they're hungry.

Serious Problem Here

There are three more "ideas" that I was sent that I want to discuss:

If your database needs constant data fixes (projectX, projectY), FIX THE CODE.

This is such a simple concept, but we seem to have missed it somewhere in our education.  If you need to do the same data fix, but with different parameters, over and over again, there is a problem with your code.  Rather than invest the energy in creating, executing and tracking data fixes, fix the code.  At the very least, let the users fix the data themselves.  Create a screen that accepts the required parameters and executes it on behalf of the user, with all of the proper auditing that needs to be in place.

If you are corrupting records, it isn’t the database, FIX THE CODE

It's amazing how many times the database has been identified as the culprit as to why the data is in it's current state.  "Oh, something must have made SQL Server corrupt the data" or "Maybe SQL Server can't handle the load that we are asking it to do and it failed to do something we told it".  Maybe the real answer is that SQL Server did exactly as it was told.  Our code told it what to do and it did it.  SQL Server is not an autonomous machine randomly changing data on a whim.  If the data is corrupt, we, the humans in change, did it, not the machine.

If you have a 24 hour load process, don’t blame the DBA, the hardware, the OS, maybe FIX THE CODE

We've got really large machines in our UAT and Production environment.  You would be amazed at what they are capable of doing, not in a day, not in an hour, but every minute!!!!  There is enough processing power to do physics calculations on the interaction between dozens of objects in an environment ten, twenty or thirty times per second.  There is enough processing power to correctly calculate and render a 3D environment in which these objects exist in real time and show them at thirty frames per second.  What I'm trying to say is that the machines are more than capable of playing the worlds most complex and demanding computer games in existance and provide excellent response.  The graphics and physics engines embedded in these games make thousands of calculations every second.  Tens or even hundreds of thousands of calculations every minute.  And yet, we're busy doing 100 records a minute into a database and we're happy. 

Does someone other than me, and the person that highlighted these points, see a serious problem here?

Fixing Code

I've touched a nerve and it seems to be a pretty raw nerve with a lot of people.  Fixing code.  Here's what one person said the developers need to do when fixing their code:

• take the time to understand the bigger picture and the context of the codes’ function
• take the time to understand the impacts of what they’re doing, even just commenting out a line
• rewrite their code rather than creating more spaghetti code

So, this person seems to want programmers to understand what they are doing, the impact of what they are doing and to do it correctly (no spaghetti code).  When you go to a mechanic to have him fix a problem with you car, you expect him to know what he is doing, to understand the impact of the change and to fix the problem correctly.  If you go to a doctor for an operation, you expect him to know what he is doing, to understand the impact of the operation and to do the operation correctly.

So why is it that programmers, developers, whatever you want to call them, why is it that they continually make silly changes to the code, make things more complicated and just don't seem to care to do things correctly.  For instance, in a recent project recently we observed the following code (things have been changed to protect the guilty, but the same concept is there):

IF anotherField > 3 THEN year = year +3

IF anotherField <= 3 THEN year = year + 3

These lines were back to back in the code.  The net result is that regardless of the value of anotherField, year is going to be incremented by 3.  There may have been some different logic in there previously, by whoever made the change did not bother to look up and down the code for a few lines, the immediately preceding line in this case, to see what was going on.  As it is the code is longer than necessary, takes longer to run than necessary, is more complicated then necessary, is confusing to the casual reader and just needs to be rewritten.

Yes, sometimes to fix a problem you only need to change a single line.  But the better approach may be to change half a dozen lines, or more, so that the next person finds the code cleaner, simpler and better.

Planning to Fail

I'll be honest, I kind of suck as a Project Manager, (sorry, Dawn, but you've already hired me!!!!)  My problem lies around the attention to detail that a Project Manager needs.  When coming up with a project plan they need to ensure that all tasks for the project are actually accounted for in the project plan.  Having come from a large consulting company that part was actually made a lot easier by using a standard template that had all possible tasks and you actually remove the things you wouldn't be doing in this project.  (Removing things is a lot easier than adding things.)  By adding in some default assumptions about the size and complexity of the application, viola, out came a project plan.

The next step is assigning people to tasks.  This is where 90% of project managers make the biggest mistake.  If you have 80 hours worth of work and you have a person who works 8 hours a day, how long will it take to do the 80 hours of work?  If your answer was 10 days, congratulations, you are part of the 90%.  If you said something else, you knew it was a trick question and you decided to play games with me, so consider yourself part of the 90% as well.

Contingency.  Nothing is perfect.  No task is perfectly timed.  You plan for the unknown.  You plan for things that you can't forecast.  You add a certain percentage of the task time as contingency.  This is essentially a slush bucket of time to fix things or redo things that you didn't know you would have to do when you originally scheduled the task.

Sick Days.  While this 80 hours of work may not intersect with an employee being sick the longer the project is the more likely you are going to have sick employees.  Indeed, depending upon which survey you agree with the number of sick days per year for IT workers varies considerably, but probably averages out to about 1 day for every 25 to 30 days worked.  Some people try to cover sick days under contingency while others have a separate category for it.  (If the project is long enough, consider accounting for vacation time for all staff as well as support personnel.)

Miscellaneous Project Time.  Let's not forget about the fact that you are sending emails to this employee that he has to read (skipping development), or that he needs to attend a project meeting or fill out a project report, all of which is "non-productive" time.  Indeed, non-project time can be a considerable amount of time.

Longer hours = less productivity.  Studies have shown that for short periods of time, one to two weeks maximum, people can work longer hours without decreasing their hourly productivity.  After that, however, their productivity hits a rapid decline.  I worked on a project where 55 hour weeks were mandated.  After a couple of weeks of this they got just as much productivity out of people as they would have if a 40 hour week was enforced.

So, how long will it take for 80 hours of work to be done?  11.75 days +/- 1.4 days

How many days are going to be in the project plan?  10 days.  We're already off to a bad start and the project hasn't even began.  No wonder people dislike project plans or even planning in general.  You're already in trouble and you haven't even started.

Efficiency and Simplicity

So, for the second time in less than a week I had the opportunity to use the Capital Health Region's medical services.  Last week I spent three hours in a medicentre waiting room to see a doctor for three minutes.  I would have gone to another medicentre but this was the only one that was open in a 10km radius that I could find.

On Sunday I witnessed the response to a 911 call and was pleasantly surprised at the rapidity with which the response came.  911 answered the call before I was even sure that the phone had wrung and when I was forwarded to the ambulance dispatch, the ambulance was already en route to the house.  How did they do this?  Effective use of technology.

Not just "the use of technology", because, quite honestly, sometimes technology makes this worse instead of better.  Not because the technology is not helpful, but because of the complications in using the technology or the poor implementation of the technology.  The luggage handling fiasco at the Denver Airport is one such example of how technology can be beneficial (the concept was sound and the potential payoff tremendous) but if implemented poorly the result is millions of dollars of effort wasted and, in the long run, a return to manual labour.

When I called 911 they had the reverse lookup available for them so that they knew the phone number I was calling from and the address associated with that phone number.  They had me confirm the information in order to ensure that the records they had were correct and up to date. The rest of the conversation was spent getting details of the incident so that they could be transmitted to the EMTs who were in the ambulance in order to prepare them.  The information that the dispatcher entered into the system was immediately available to the EMTs.

While we try to build our systems to take into account all of the potential problems and all of the potential solutions, we need to keep in mind that there is a very specific task we need to accomplish and that we should focus on getting that task completed in the most efficient manner possible. Not the most elegant manner and not the most all-encompassing manner, but the most efficient.  While you may not be writing an application that is designed to help save lives, that doesn't mean that the concepts - efficiency and simplicity - are not applicable to what you are doing.

Efficiency

When I went to NAIT we had an interesting instructor for one of our classes.  He was supposed to be teaching us how to design interfaces for applications.  Since this was the day of the green screen his examples were all of mainframe based applications.  Looking at those screens brought some unpleasant feelings welling up from within.  It was the opinion of everyone in the class that the instructor "didn't get it".

Indeed his screens were overly complicated, cramped and just plain confusing, but he was convinced that you needed to design screens this way in order to be efficient.  Why the disconnect?

Some people confuse efficiency with the amount of information that they present to the user in a single screen.  The more "information" the more "efficient" the screen.  This is so wrong.  Studies have shown by keeping things simple people can comprehend what is going on faster and with a higher retention rate.  In the example that our instructor gave us he was trying to demonstrate what needed to be on a customer contact screen.  He had not only the basic customer demographics, but a recent contact list, dates and amounts of previous orders, recent changes to the account and payment history.  Some of this information was denoted in codes which the person looking at the screen was supposed to have memorized in advance.

Information overload is not efficiency.  For instance, if in the majority of cases the user just wants to look at demographic information, show just the demographic information.  Tailor the information that you present to match the circumstances for the screen.  Don't be afraid to add a tab or other item for the user to get more detailed information.  IF they need it you have provided a means for them to get it.  If they don't need it you wouldn't have wasted any time retrieving it.

The purpose is to allow someone to do the same job they are doing now more efficiently and sometimes that actually means less information, rather than more.