Batch Processing

Mainframes are built to move huge amounts of data around.  IBM did a fantastic job of building giant batch processing engines.  Granted there are online options, but, to be honest, their bread and butter is huge amounts of data in and out.  Because of that mentality, the fact that the course designers at Universities and older people in the IT field "growing up" with this technology and even the integration of the term "batch processing" into the general lexicon of life, people design systems to include batch processing.

*WRONG*

OK, that may be too strong.  Let's say that the majority of the "reasons" for batch processing are actually replaceable with the phrase "because we've always done it that way".  Batch processing, the processing of huge volumes of data in a scheduled, off prime time manner, is a relic of the mainframe and needs to be replaced with the idea of a batch of one.  This idea of a batch of one is not new, as I`ve even used it on the mainframe.  The user needed certain processing done immediately as opposed to that night and we dynamically created a job, complete with the JCL and necessary parameters, in a low priority job queue and the job was processed when the machine had spare cycles.  It was batch, but a batch of one.

The idea is applicable in today's world.  When I ask people why they are creating a batch job they say something like "the printer is loaded with a certain paper at night so that is the only time we can do this".  *WRONG*  The processing can be done at anytime, the printing can only occur at night.  Instead of doing a lot of processing at night, do the processing in an asynchronous manner throughout the day and do the printing at night.  Split things into two pieces:  processing and printing.  They are two separate functions and do not need to be combined.

Let me rephrase in a more "global consulting" manner:  by separating processes into their unique steps, you create more opportunities to provide value to the business client.  Through a selective process of decoupling the steps in a business process you provide opportunity to inject new business functionality into a more flexible framework with less rework to the existing systems. Isn't that really what the business user wants?  More flexibility?  More value?

Short tempered

Some have said that my comments in the Daily Migration Note are brusque or just plain old rude.  Others have said that I am condescending in my notes.  A number of years ago a project manager actually said he found me "abusive and your attitude repugnant".  (I always keep those good emails so that I can refer to them later.)  Others call me up to see if I am still sane or whether I'm going to go postal.

I am, by my own admission, a strange duck.  Pressure doesn't affect me the same as other people.  I've been in situations where an the vast majority of an organization was sitting on their hands, waiting for me to come up with a solution. I've been the key person involved in resolving a payroll issue that affected tens of thousands of people.  While they are stressful situations, the pressure of the problem, the adrenaline that is produced, all blend together.

And yet, some things make me react quite harshly.  Yes, even quite rudely.  Why?

At a recent event there was an opportunity to evaluate myself based on a number of different characteristics.  Based on my answers to the questions, I was (drum roll please) not a people person.  This may not be a surprise to a bunch of you, maybe all of you.  What the quiz showed me was something that I knew all along:  process and logic are the key factors when I am dealing with work.

So, what does this mean?  Well, if things don't make sense to me I have a tendency of digging my heels in and saying "No".  If there is no logical sequence of events I have a tendency of saying "stop".  If people try to pressure me to do something that I think is illogical, I fight back.

Do you know what actually makes sense to me?  Something that is a logic progression when resolving a problem?  A partnership.  Not a partnership where one or more sides pay lip service to the idea of a partnership so that they can check it off their list of marketing responsibilities (there are many such partnerships amongst major corporations that fall into this category), but a true partnership where each side is working towards a common goal and where all parties are free to bring forth ideas to the table.  No finger pointing.  No "blame game" tactics. 

I have been involved in a number of different groups that were true partnerships between two or more groups and the results are amazing.  With the freedom to make any suggestion about any topic without fear of reprisal the ideas and solutions that come out of such a group are amazing.  There is tremendous difficulty in this as you need to leave your ego at the door and be willing to admit you were wrong.  You need to understand that there is more than one way to skin a cat and that yours may not be as effective.  And, in a true partnership, you don't try to pressure people to follow your lead or your suggestion, you true to influence them and persuade them that your way is appropriate to try.  As soon as you try pressuring people the partnership is gone.  And, in my case, I start resorting to my stock answers of "no" and "you must be kidding".

Partnerships.  Give them a try and you'll be amazed at what you can do.

Debugging

Don't blame the Deployment Team if your application fails and you can't figure out why.

Increasingly we have seen project teams with problems in Production (or UAT for that matter) and they are at a loss to explain what the problem is.  Sometimes there is a cryptic error message in one of the event logs along the lines of "Object reference not set to an instance of an object".  <sarcasm> Now that is a descriptive error message. </sarcasm>

There are a number of things you can do to make your life easier:

• The various .NET languages come with a feature called a Try...Catch block.  Where you try something and if things don't work you catch the exception.  Amazing how that works.  If you haven't put Try...Catch blocks in your code, particularly around any code that is calling a service outside of your control (SHR, EPS, DMS, PRS, etc.), then you score -5 points on the attention to detail rating.
• Did you know that if you catch an exception you can add your own information?  Yes, if you've caught something you can add information such as what you were doing when you got the exception or copies of the data used in the method so that people don't have to guess at what went wrong.  If you just rethrow the exception without adding any information, deduct 5 points from your "value added" score.
• Did you know that there is a Trace class in .NET?  It can do all sorts of things for you, such as allowing you to dump out detailed trace information (that is if you've written detailed trace information in your application) by changing a config file or even dynamically through code.  This is a very handy feature when used in combination with breadcrumbing so that you can see the complete route that an application has taken and, if you've added it, the data used throughout the process.  If you've never used Trace then how do you ever expect to trace anything in Production?

There are lots of things you can do to make your life easier when it comes to resolving problems in Production.  Simple things that can be useful in any environment.  Simple things that you should know how to use and how to use effectively.

Introduction to an old friend

I'd like to introduce to you, an old friend of mine.  The PRINT statement.  Granted, he's not as fancy as a GUI Debugger, nor as sexy as ANT or AVICODE, but he's still really useful.  Have anyone thought of adding PRINT statements to their T-SQL that they are executing?

For instance you could have something like this:

(14 row(s) affected)

(14 row(s) affected)

Or something like this:

Merge Learner Script
(14 row(s) affected)
Learner Merge - From SIN : 918728130 To SIN : 665171625
Start date - 2008-08-22 09:30:03.120
Create an Activity
Close the Activity
Learner Merge - From SIN : 999014679 To SIN : 654457258
Start date - 2008-08-22 09:30:03.420
Create an Activity
Close the Activity
Learner Merge - From SIN : 999011572 To SIN : 647141506
Start date - 2008-08-22 09:30:03.480
Create an Activity
Close the Activity
Learner Merge - From SIN : 917032336 To SIN : 661696161
Start date - 2008-08-22 09:30:03.527
Create an Activity
Close the Activity
(14 row(s) affected)
Done at 2008-08-22 09:30:03.590

The second version looks much better as it is clearly describing what is occurring inside the script.  If you are trying to find out what is going on the second one is so much better.  If you have an auditor on your butt, the second one is so much better. 

Why don't more people do this?  Well, it takes effort.  Typing the characters can consume multiple seconds per statement.  <sarcasm> I mean, over the course of a year this could add up to an hour or more of your valuable time. </sarcasm>

I'd like to thank the SFS team and Janice Pyle for the second example above that clearly shows the advantages of the lowly PRINT statement.

Using a hammer

Have you ever seen Canada's Worst Handyman?  I must admit that I am all thumbs, but, even so, I'm better than these guys.  There are lots of things to look at in this show because so many of the items can be translated into almost any occupation, including the IT field.  One of my favorite things is watching the people, when confronted with a screw, pick up their hammer.  Or using any one of a dozen other tools in the wrong way.

IT people have the same tendency.  We have a problem.  We have a tool.  We try to solve the problem with the tool, regardless of whether or not it is the right tool for the job.  Sometimes you have no tool, just a problem, and in that case almost any help is worth the effort.  Well, maybe not.  Sometimes the effort that the tool saves is totally outweighed by the costs associated with using the tool.  Much like a carpenter that has a plethora of tools to help him do his job, IT people have a variety of tools to help them as well.  Our problem is that there is no simple guide saying "for this problem use this tool and for this problem use this tool".

That's not to say that if we had such a guide people would actually use it.  I mean, look at Canada's Worst Handyman, they were taught certain things and still ignored what was taught.  I have no fear that even if we told people to do X they would still end up doing Y.

I'm not restricting myself to just tools, but methodologies as well.  Some people grab on to a methodology and use it like a life preserver.  Not necessarily a wise thing.  An all or nothing approach is inevitably a bad thing.  Even the Agile Methodology that people either hate or love (there we go with extremes again) understands this.  One of it's principles is:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Essentially, you evaluate what you are doing and adjust if necessary.  I guess my biggest fear is the life preserver mentality for methodologies.  A methodology is a tool and proper use of the tool depends upon proper understanding of the problem.  We sometimes adopt methodologies because they are "cool" or have the "agile" word somewhere in the title.  We don't always understand the problems that we are trying to solve with the methodology and are just so thrilled with using a new tool that we overlook some of the challenges that we have with the tool.

Don't get me wrong, I love new tools, but how many times have you ever used a new tool incorrectly the first time.  If you are going to make a big leap to a new methodology/tool, make sure that you've tested it first and that it actually solves the problems that you need solved.

Quality vs. Speed -- The Agile Way

There has always been a debate over producing a quality application versus an application that is delivered on time.  Indeed, even within the quality camp there are divisions over what exactly constitutes quality.  There have been efforts in the past to merge quality and speed of delivery so that you can get a high quality application in as short a period of time as possible.  The Agile methodology tries to merge these two, to mixed results.

Why mixed results?  Like everything else in life, more than just how you do something determines the end result.  In the case of the Agile methodology (or should I say methodologies) there is more than one way to skin a cat.  The Agile methodology is so different from what people were taught in school that there are three large stumbling blocks that need to be overcome in order for Agile to be truly effective.

• Organization.  There are forces at work in an organization to keep things moving forward.  Plans need to be made, followed and tracked.  The agile methodologies have the idea that "responding to change over following a plan" is a better way to develop software.  And, while I personally believe this, organizations tend to want plans.
• Team. While Agile has come up with some basic philosophies behind how to develop software, the larger the team the more unlikely it is that these philosophies will be effective due to outside pressures: when is going to be done? It is difficult for project managers as they are measured on progress.  Agile, because of its fundamental desire to respond to change, is havoc on plans, at least traditional "we will have this functionality delivered by that date" type of plan.
• Individuals. Everyone comes to a project with baggage:  their old projects and the "we did things this way" ideas.  Getting everyone on the same page, following the same ideas and concepts is very difficult. 

This is not to say that Agile will not work and that you will produce a quality application in less time.  The problem lies in being able to show somebody that this will happen.  Without a proven track record it is hard to show people that it will work, but you need that acceptance so that you can build that track record.  And without the right people in the right places, Agile is never going to fully accepted, fully understood or fully implemented.

Results vs. Methodology

Sometimes it is a story that strikes a chord and makes me write one of these notes.  Sometimes it is an ad on tv.  In this case it is a newsletter from Bruce Schneier, the Chief Security Technology Officer of BT.  He said, that governments should:

...legislate results and not methodologies.

That immediately showed up on my internal radar screen.  While his examples were with regard to things like legislating securing personal data, but not legislating which technologies to use.  Using my own quirky logic I equated this to:

...contract for the end result of the project, not how the project is going to get there ...

When a company puts out an RFP for a piece of work, they routinely say that the winner must "... follow all standards and processes currently in place or that may be put in place during the course of the project...".  We've all seen these as they are not specific to government.  But is it the right thing to do?  Let me go out on a limb and say "no".

In the long run, does it matter if the project team has detailed requirements tracing from the initial conceptual design stage straight through to production?  If you had two projects that delivered the same end result, does it really matter which route they took to get there?  If there are certain deliverables that you want then specify those deliverables in the contract.  Don't specify how you want them created, just that you want them created.

Coming from a programmer background and having been introduced to one of the most document oriented methodologies in existence - Method/1 - I've seen both sides of the coin.  Method/1 was created, to be honest, to minimize liability should the client ever decide to sue.  Dozens, hundreds, thousands of documents are routinely created so that an audit trail could be created in the event that a law suit was launched.  Method/1 was created by auditors, for software development, to be used in the event of a law suit.  Not exactly a glowing endorsement of a methodology.

Other methodologies are created for other purposes, but what it really comes down to is results.  You have certain expectation of results.  And whether the application was written people in blue suits with a document laden process or by a long haired geek who wrote it between episodes of Death Note, it shouldn't really matter, as long as the results required are achieved.

Your Environment is Wrong

"But, it works when I do it, so your environment must be wrong."

When I first heard that comment I took it seriously.  What could be wrong with our environment?  Why couldn't we install the application? 

Not different, because our UAT and Production environments are different than the development environments for a lot of reasons, but wrong.  I searched for something wrong.  Occasionally, I did find something wrong, but as we put more and more applications on to our UAT and Production boxes the number of things that were wrong obviously diminished to the point where their is not a whole heck of a lot that can be wrong.  The differences, however, are still there.  Differences that may prevent the application from installing or working correctly.

• clustered environment - we operate in a clustered environment, so never, ever think that this is the only application that is being installed on this server and never, ever think that it is the only instance of your application which is running
• not a clean environment - the odds are that we've installed your application before - sometimes once or twice before and in some cases a hundred times before - never assume that the environment is "clean" because the odds are that there are remnants of other installs of yours left around
• multiple installs - never assume that just because there is some manual configuration at the end that needs to be done that it is "no big deal" because in our UAT and Production environments we need to do the installs multiple times so that any manual configuration that needs to be done will be done multiple times, giving us plenty of opportunity to make a mistake

"Mistakes"?  Yes, we sometimes make mistakes when manually configuring an application.  Considering that in the UAT and Production environments we are going to be installing and manually configuring your application seven times or more, once we add more servers, this gives us seven chances to make a mistake.  Combine this with multiple releases to UAT and you've got a wonderful opportunity to introduce an unexpected bug, with the most likely problem being that the manual configuration was not saved on a specific machine.  Have the application self-configure itself where possible: prompt for data and complete the installation itself.

Oh, I forgot, you're only going to UAT and Production once, right?

More examples

Going on vacation seems to give me plenty of ammunition for these notes.  The following example of what not to do is based on a conversation with the Day Manager at the Sandman Inn in Calgary West.

My wife and I needed to go to Calgary to drop our kids off with her sister.  They were going to horse riding camp for the week and we thought we'd start it off early by going to a hotel when we got there instead of sleeping at sister's place.  On the Friday before we went we reserved a room at the Calgary West Sandman Inn through the Sandman web site.  We got a room with a King sized bed and, most importantly, a confirmation number.

When we got to the inn on Saturday, however, we were surprised because they did not have the room that we reserved.  They had a room in the same "class", but they did not have any rooms with King sized beds.  I was a little surprised because I thought a confirmation from the hotel chain guaranteed the room was available.  I was told by the clerk at the front desk that the Sandman web site and the computer system at the hotel do not talk to each other.  And that they had been trying all day to get hold of someone to have the web site stop booking rooms.

I was, quite frankly, amazed at this revelation.  Imagine building a reservation system for a hotel chain, but not hooking it into the hotels reservation system!  So what did I do?  Well, there was nothing to do about the room as there were supposedly no other rooms in the hotel.  (They did give me a $3.00 discount for my inconvenience.  The pop in the pop machine was $2.50)  When I got to the room I immediately hooked up to their Internet connection and went to the Sandman web site and filled out a form with a complaint.

After waiting 5 days for a response I called the 1-800 number for reservations.  I got the Manager of Reservations and she was going to call the Calgary West Sandman to see what the scoop was.  Three days later I got a call from the Day Manager.  She confirmed that the systems do not talk to each other and that in order to get the web site to stop reserving rooms the general manager has to fill out a form and send it in.  (He only works 9 - 5 on weekdays, so if something needs to be done outside of this timeframe, oh well.)  By the way, the 1-800 line uses the same system the web site does, so that does not guarantee you the room either.  You need to talk to the hotel itself.  Same thing is to be said with regard to Expedia or Travelocity.

So what do we have?  A hotel chain with thirty five properties that reserves rooms without actually knowing whether or not the room is available and a form based system that is required to be filled out to stop selling rooms.  Does anyone see a problem here?  As I escalate this problem up the Sandman management tree I'll let you know if this is going to be changed.  If not, unless you call the hotel directly, don't bother reserving a room with Sandman.

Planning, ad nauseum

OK, I've talked about planning in the past and I'm going to mention it again, but from a different perspective.  There are many different types of planning for many different types of people:

Programmers. These people need to know what they are doing now and for the rest of the week.  To be honest, while they might like to know what they are doing next week, as long as they are confident that their boss knows, that is probably good enough.

Team Leads.  With programmers reporting to them they need to have a much better grasp of what is currently going on and what is coming up.  A week isn't good enough.  A month may not be good enough depending on the project, so let's say that these people need 6 weeks of planning under their belt.

Project Managers.  Ah, the brains behind the project.  For any particular project that  they are working on, they need to have a good idea of staffing requirements from beginning to end.  Yes, they need to know staffing for the entire project, because if they are short, it is their responsibility to get staff.  If they are going to be over staffed then it is their responsibility to see what other projects need help.

Program Managers.  These people are responsible for the smooth execution of multiple projects within an overall program.  For example, you may have a Program that is composed of a lot of smaller initiatives, all of which are leading to a single vision, much like PASI.  Program Managers need to understand the interrelationship between the different projects and how they impact other projects and the program as a whole.  This sort of planning stretches into the years.

Organization.  Different organizations use different benchmarks, but it is considered a good idea if an organization can tell you what they are going to be doing for the next three calendar years.  This will not be to a great deal of accuracy three years down the road, but the overall plan of the organization, the inter-relationships between ongoing programs and the intended implementation date of various projects should be at least plotted out so that both clients and staff have an idea as to what is currently happening and what the short and longer term goals are.

As you can see, the level of detail goes from the microscopic, tactical information needed for day to day activities to the macroscopic, strategic information needed for long range plans.  Virtually everyone in an organization needs a different level of planning with a different focus, depending upon the roles that they are currently occupying.  The key to all of this, though, is that there needs to be someplace where all of this information can be stored.  Some place where the microscopic and macroscopic views can both find a home.  A place to go when information is needed and a place to go when you have new information.

Peer Review

Why are some people against having their work reviewed by their peers?

Commonly called a "peer review" in academia, the concept revolves around letting other people knowledge in the same field comment on your paper (design, theory or observation) prior to "publishing" it.  Specifically to the IT world a peer review is when a design is reviewed by knowledgeable people in an effort to determine any flaws in the design.  I will grant you that these reviews can be brutal on the ego, but aren't they the right thing to do anyway?

One of our primary objectives in our job should be to produce quality work.  Now, the exact measurements behind defining quality are highly contentious, but in general, the idea is to produce something that accomplishes the objective and does it as efficiently as possible.  One of the keys to creating a quality product is having a good design, one that is as simple as possible yet still satisfies all of the objectives and goals.

A good design.  How do you know if a design is good?  Well, one of the ways that the rest of the world  does it is through design reviews or peer reviews.  When a design has been completed it goes to someone else knowledgeable about the requirements so that they cam see if the design does indeed meet those requirements and does it in an efficient manner.  Seems pretty simple doesn't it?  So why don't we do it more often?

The Enterprise Architecture Working Group (EAWG) did this, to a very limited extent, when they reviewed the application architecture on various projects, but the problem was that at 50,000 feet everything looks blurry.  As a friend of mine used to say "Good from far, but far from good".  Understanding how someone is going to piece things together is important, but the details, the microscopic manipulations that occur on an individual program level are completely oblivious to everyone except for the designer and programmer.

Every critical piece should be peer reviewed to ensure that it is designed properly.  This means that if you are writing something that manipulates funds it needs to be peer reviewed.  If you are writing a mission critical import/export/reporting piece, it needs to be peer reviewed.  If you are writing a security component, it needs to be peer reviewed.

If it's never bee peer reviewed, never had an objective review of it's design, how are you every going to know if you designed something properly?  Is it fear that is stopping you?  Fear of people pointing out mistakes?  Or is it narcissistic in nature?

GUIDS and Clustered Indexes

GUIDS should never ever be used as a clustered index.

Wrong.  GUIDS can be used as a clustered index if they are not an MBUN: meaningless but unique number  I've talked about this before, but it seems to be difficult for people to understand so I'm going to take this slowly and give some examples.  For these examples, let's assume that you are running an online bookstore.

Naturally you want to keep track of orders and you have a table in SQL Server that does just this.  For each order you assign a GUID.  Should you cluster on this GUID?  Probably not and the reason is that the GUID, in this table, is meaningless.  It has no context with any other data and as a result it is truly an MBUN.  Indeed, you would probably want to sort on the customer number instead as you are more likely to retrieve all of a customers orders than you are sequential GUIDs.

Hmmmm.  We're keeping track of orders, but not the items on the orders.  Let's have an OrderItem table.  In order to connect the rows in the OrderItem table with the Order table we would probably link them via the GUID that you assigned to the Order itself.  To show the individual items within the order we would need to create another column, let's call it an ItemNumber.  This could be a GUID, but it makes more sense just to make it an integer.  For this table the clustered index should be the Order GUID and the ItemNumber.

Why?  Well,. when we create a clustered index we are telling the database that we want records that appear next to each other in the index to actually be stored next to each on the disk.  In the first example, using the GUID made no sense as there was no reason for the records to appear next to each other.  In the second example it makes perfect sense to use the GUID because we want all of the items for an order to appear next to each other on the disk.  This speeds up the retrieval of data and makes more effective use of memory.  In the second case the GUID is no longer an MBUN as it actually has meaning: it is the order number that we are tracking.

So you see, if the GUID has meaning it can be part of the clustered index.  If the GUID has no meaning, don't put it in the clustered index.  Any questions?

Database Design

How much time do you spend designing your database?  Twenty percent of your overall design time?  Ten percent?  Five percent?  Two percent?  Design?

Well, the magic number should be 15%, or thereabouts, and can be determined by the juxtaposition of the solar days as they are related to the cardinality of the days of the week, the overage as attributed to project management style and the variability imposed by Pantone colours as they are blended on a surreal palette of olives and daisies.  Just kidding, there is no magic number.

Good database developers require less time to develop a well structured and functional database.  A smaller application with a more focused purpose requires less time as do re-writes of existing systems and applications for which there is a wealth of knowledge available.  Conversely, applications that are new to the organization, new to the designers or are complex in nature require considerably more time in the design phase.

Database design, contrary to what many people believe (and contrary to what I've seen implemented) is a complex task.  Understanding the data access paths that the application takes, the appropriate indices to create and how to optimize that access is difficult for anything more complex than a simple inventory system.  Why?  Well, if you have a column that contains two values "YES" and "NO", with approximately 100 "NO" for every "YES",  and you create an index on that column, will SQL Server use it?  That depends on how many rows are in the table.  SQL Server will make the determination, at some point, that the index in its current form is useless.  How about if the column contains four values, will that help?  It will delay the inevitable, but regardless, SQL Server will, at some point, say "To heck with it" and not use the index.  The cardinality of the columns in an index is as important as the index itself, yet very few people understand this rule and as a result, indexes are created that may work well today or tomorrow or even for months.  One day, however, the house of cards will collapse.

Take some extra time and work on the design up front and I'm sure you'll be pleasantly surprised by how little pain you feel when it goes in.

Interrupted ... Again

OK, now that you've taken a look at your own life, realized that you are totally unproductive due to interruptions, what can you do to become productive again?

Work at home, or at least, away from the office..  Seriously.  Talk to your boss and see if there is a way for you to spend every second Monday/Tuesday/etc. at home to do your work.  For some of you this isn't going to work due to the work that you do.  For others, those that producing tomes, need some time to plan the next great project or even, goodness gracious, catch up on that documentation you forgot to do - six months ago - spending some solid time working on these items can be a tremendous benefit.  Now, you don't necessarily have to be at home.  Some people find that working at home is not as rewarding as it could be due to hundreds of other distractions.  An alternative location is the key.  Whether this means you book a meeting room for all day, find an abandoned office that no one is using or even use the desk/office of someone who is on vacation, getting away from your desk, the spot where people find you, is key.

Another step?  Flush your Crackberry down the toilet.  Figuratively.  Unclip it from your belt and put it in your briefcase/deskl/backpack and bring it out every hour and see what you missed.  The constant buzzing of new messages, the incessant weight just totally p***** me off sometimes and I need to just get away from it.  Why check every hour?  Well, research has shown that people work most effectively if they take a five minute break every hour.  Go to the bathroom, get something to drink, check your voicemail, etc.  Just remember that this is a five minute break and not the start of five minutes of different work.

Turn off email notification (thanks Garth).  You know that little window that comes up and says "You've got new mail and here is the subject line and who it is from..."  Turn that off or even shutdown Outlook when you don't need it.  Just remember the five minute rule once an away.

Take a look at sites like Lifehacker (Tech tricks, tips and downloads for getting things done).  (On your spare time, of course)  This site contains some useful and some useless tips on how to keep organized and get your life back together.

Unfortunately, for some of the people reading this note none of the solutions are going to work.  You know who you are:  support personnel.  Those people who support other people and who, quite frankly, need to be in constant touch because their clients need help.  An email about the best way to re-engineer a COM+ application can be done later.  An email about a crashed system needs to be handled NOW.  So, while many of you can take advantage of these and other tips, others ... oops, have to go, later.

Environment Building on ten cents a day ....

I have been giving project team members a hard time about advance notice.  You know what I mean, giving the Deployment Team more than 15 minutes notice that they need a deployment done.  Well, I'm not going to bug developers in this note, I'm heading higher up the food chain for the project managers.

I've talked about how we are short on space and short on power.  Apparently many project managers (and I am talking about a good dozen or more at the moment) think that these facts don't apply to them because they are special.  You know what?  Everyone is special.  When there are so many special projects, the appellation is meaningless.  So, let's talk turkey.

I want a performance test environment, and I need it next Monday.  OK, to be honest I haven't heard this .. too often.  Most projects give us a lot more warning, although sometimes there is not a lot I can do for them.  (Did you miss my notes on not having space and being electrically challenged?)  Let's take a look at a typical application and see what a performance test environment entails.  A smallish application that access Stakeholder Registry.

So, what do we need for this performance test environment?  Well, we could go with clustered web servers, but let's give ourselves some leeway and just pick one.  We then need a SQL Server box for SHR and a SQL Server box for the application database.  Wait, in Production these are SAN attached, should we duplicate that?  Well, that would mean buying another SAN and the storage and configuring it, etc., so for the purposes of our example let's not go there.  Oh, wait, in order for this to be even remotely representative we need to emulate the existing servers, not buy new ones.  So now we need Dell 6850s, but they're not made anymore.  Have no fear, eBay is here!!  Oh, darn, that reminds me, we need to get an Dell 1850 for the web server.  I hope we haven't gotten rid of all of them yet. 

OK, we done yet?  No?  What else do we need?  Well, if we are using SHR we need to hook in ISM, which requires another license for SSA-NAME3 as well as a Dell 2650.  (Remember eBay?  It's your friend.)  Here is an interesting fact for you:  If you restore the SHR database, you need to rebuild the database that SSA uses for searching.  This probably isn't more than a six hour job these days, but it might be longer.  I guess this means that if you run a test at 9:00 AM in the morning we might be able to rebuild everything for a 3:30 PM test.

I'm pretty sure that you're not going to be using the stopwatch facility on your Timex watch to record everything, so that means we need some tools for testing and recording.  Let's assume that we have these, but the tools we might need are Intercept Studio licenses (aka Avicode) to pinpoint some performance issues for us.  These are not necessary, but they have proven themselves to be quite valuable in the past.

There are some other infrastructure things like backup tapes, disk space for intermediate backups

So what do we have?  Four computers that need to be purchased, refurbished and installed in a computer room where we are already short on space and power.  Additional licenses for software we don't have.  A lot of effort on part of a lot of different people just to get things in place.  Just think what would happen if we needed PRS (Oracle server and license) or perhaps CGES (two servers and SharePoint) or, heaven forbid, all of those, plus more!!!!

Or, *shudder*, you have multiple projects coming to you at the same time with the identical requirements.  Welcome to my world.

Setting up a performance testing environment is very expensive in terms of hardware, software and sheer manpower needed to set it up and keep it running.  The question is, what are your really trying to test?

Don't get me wrong, I like the fact that you want to performance test.  I guess what we really need to understand is what you are trying to test.

Loss of Power

Electricity.  It lights our houses, cooks our food and allows us to watch and listen to events that occur half way around the globe.  It powers our watches and allows us to pick up a phone and call someone a thousand miles away.  It powers our lives.

It powers our computers.

The idea was that computers would become smaller, cheaper, more efficient at doing work for us.  Guess what?  In order to do this they consume more power.  The average Dell "high end" server is smaller, faster, more capable of doing fantastic things, but it also consumes about 20% more energy than its predecessor.  Unlike everything else that becomes smaller, faster, better, the power consumption of computers has gone up. People have always expected a higher density for server rooms (more computers in the same space) and server rooms that are constantly growing in size, but for the same number of computers to chew up more power?  In many respects that slipped by peoples radar.

Server rooms have always been built with some excess capacity and ours is no problem.  Indeed, if our server racks were filled to the brim with hardware from three years ago, I think we wouldn't be having this conversation.  But, power requirements for an individual server have gone up and as a result our ability to add more hardware is limited by the capacity of our server room.  We're short on capacity.  Indeed, I don't think we can add all of the new hardware we have without seriously overloading every circuit we have. 

Let's look at some examples and cry:

Example 1:  a Dell 2650 consumes 357 watts while a Dell 2850 consumes 524 watts.  That is almost a 50% increase, even though the amount of space consumed is the same. 

Example 2: a Dell 2950 consumes 307 watts and takes up 2 U of space, approximately 154 watts per U.  A Dell 1850 consumes 444 watts in a single U.  So while we consume half the space we almost triple the power consumption per U.

Newer machines are greener, consuming less power per U of space, but we don't have a lot of the latest generation of machines, those that are efficient with their use of electricity.  In the past we've upgraded machines to give clients the power they need and we've accidentally hampered out ability to expand.  The Dell '8' series (1850/2850) are huge millstones around our neck as we're trying to stay afloat in the electricity ocean.  So what are we doing?  Replacing as many 1850s/2850s as we can.  (See yesterdays note on virtualizing, evergreening and replacing).

So, we don't have a lot of room and the servers that we do have are consuming huge amounts of power.  Can't get any worse, can it?

No space at the inn

OK, for a number of months now I have been using the explanation "Sorry, but we have no room at fulfill that request."  Many of you have been the recipient of this sort of email and you may be thinking that I have just been using this as an excuse.  While there are many things that impact the "room" that is available, let me just talk about physical room today.

Within our server room we have a number of racks.  These racks are industry standard racks for holding rack mounted servers.  (Hence the name similarity).  These racks contain 42 U of space. and we have 17 of these racks in place. That's a lot of space.  We are also about 80% full. Theoretically we have enough room for about 70 servers or so, based our own current 2 U standard server.

That was theoretically. In practice we do not have this much usable space. Due to the fact that the older servers had a wide range of sizes, the fact that we put more than just servers in the racks (storage enclosures, load balancers, etc), the amount of usable space where we can install our standard server is about 72 U in size; or about 36 servers. This is pretty close to the number of servers that we need to install, but this gives us absolutely no room for growth. (Based upon meetings I've had recently, we are going to be growing.)

So, how do we free up space? Virtual machines.

We have converted about 40 machines to the virtual variety in the past few months. We want/need, to do more and we are going to that. My personal goal is to free up another 50 U of space in the next couple of months. so, that we can have that room for growth. We need some help from Microsoft at this point as we are having problems with one of their "P2V" (physical to virtual) tools that is making this transition a lot easier.  They have managed to replicate our problem and are working on a solution.

We are currently in the process of mapping out precisely what is in our server room, which rack it sits in and exactly where in the rack it is taking up space.  Based upon this very detailed map we will be able to plan out where new machines should go and what we need to do in order to consolidate some of our fragmented free space.  (Much like a disk defragmenter, but on a much larger scale.)

This is not the final solution, however, as a larger battle still looms and we will discuss that tomorrow.

Budgeting

I'm not really a big fan of budgets.  My wife and I try to make them and then we blow them out of the water within a month.  For projects, however, budgets are important as they lay down some limitations on what can and cannot be done.  Sometimes, however, we don't set the budget properly.  Let me give you an example:

You walk into your bosses office and they tell you that they have a new project for you.  They tell you what the end result of the project is going to be and what your budget for the project is.  The problem is that there has not been any detailed analysis done so the size of the project is totally unknown, but not the budget.

I'm sure that most of you have been there before, even in a somewhat different perspective.  Imagine if you will, going up to your team lead and have them tell you about a new feature you are going to implement -- in one weeks time.

it is important to understand that budget does not necessarily limit itself to dollars as budget can also be equated as people time.  If you have X dollars that equates to Y days of effort. Whether you use the top down approach to budgeting, the bottom up approach or the throw a dart at the dart board approach, you need to understand what you are going to build and how you are going to build it.  Once you figure out that you can come up with an estimate.  Based on that estimate you also include a certain percentage of time for documentation, testing, UAT support, project management, internal technimcal support and deployment time and even post implementation support. 

I know, you normally don't do this.  Maybe that explains why you feel so stressed about being late?

Process and DJ

I am a great believer in following "the law", not because it's the "right" thing to do, but because without a standard basis upon which to rest our morals and beliefs we descend into anarchy.  But you know what?  Going too far the other way is just as bad.

This is also true for IT projects.  For instance, the Project Management Institute offers all sorts of guidance on how to manage projects.  The have produced a Book of Knowledge that contains much useful information.  BUT, it is not the understanding of this information that makes a project manager effective, nor is it merely have the PMP certification that makes a project manager good at his job, it is understanding when to use the tools at their disposal and the reasons for doing so.

Over-managing a project is almost as bad as under-managing.  (Depending upon your perspective it may actually be worse, but I digress.)  Over-managing puts too much rigor into a process that does not require that rigor and that is where the skill of a good project manager comes into play.  Like a poker play you need to know "when to hold 'em, know when to fold 'em, kown when to walk away".  The effective application of the tool is the difference between a good project manager and someone who just follows the rules.

Am I being too harsh on project managers?  Maybe, but considering the wide range of project styles that I have seen and been victim to, probably not.  The project manager has more influence over the end result of the project than anyone else.  Don't influence in the wrong direction.  Give the team the freedom to innovate, the freedom to get their job done while at the same time putting in place the structure necessary to do that.

Picking on Managers ... again

It's been said that I've been picking on programmers too much recently, so I thought I would skip the low hanging fruit and go after something higher up on the tree.  Let's start with one of my favourite slogans:

Failure to plan on your part does not constitute an emergency on my part.

A project manager has a specific role on a project team:  planning and coordinating the activities of the team members.  On larger projects this may actually be the delegation of this responsibility to other members of the team.  On a large 75+ person project I worked on there was one Project Manager who ran the entire project, but four other managers who actually did the planning.  Please note that this 75+ person project had a total of 5 managers.  There were a number of team leads on the project as well bringing down the ratio to 1 to 5 for supervisors to staff.  This is by many standards a fairly low ratio, but it was typical in the organization I was with, with many of the supervisors spending only a portion of their time supervising and the rest of the time working on different tasks that needed to be done.

Each team lead knew what was expected of their team in the short term and what the long term objectives were.  Every week there were meetings amongst team leads and their respective manager and every week the managers got together to discuss the status of the project.

This same type of organization is applicable in smaller projects.  Planning is one of the key factors:  know what needs to be done now and in the future.  Too often have I seen and read about projects that do not have a clear understanding of what is required and the usual impact of this is larger, more complex projects than is required and the resulting cost and timetable impacts this has.

Worst of all, from my perspective right now, this has a tremendous impact on the Deployment Team.  Too often have we had requests like "...but the user needs it now, do we have to give you five minutes advance notice? ..." or "... it's just a small change, but we need to recompile everything and send you a complete package, if we get it done by midnight can you have it in place by 12:01? ...".  T

he reason we ask for advance notice on migrations is so that we have the opportunity to plan our own work.  If you don't give us advance notice we can't plan and that annoys me.  And everyone knows what happens when I get annoyed.