Batch Processing

Mainframes are built to move huge amounts of data around.  IBM did a fantastic job of building giant batch processing engines.  Granted there are online options, but, to be honest, their bread and butter is huge amounts of data in and out.  Because of that mentality, the fact that the course designers at Universities and older people in the IT field "growing up" with this technology and even the integration of the term "batch processing" into the general lexicon of life, people design systems to include batch processing.

*WRONG*

OK, that may be too strong.  Let's say that the majority of the "reasons" for batch processing are actually replaceable with the phrase "because we've always done it that way".  Batch processing, the processing of huge volumes of data in a scheduled, off prime time manner, is a relic of the mainframe and needs to be replaced with the idea of a batch of one.  This idea of a batch of one is not new, as I`ve even used it on the mainframe.  The user needed certain processing done immediately as opposed to that night and we dynamically created a job, complete with the JCL and necessary parameters, in a low priority job queue and the job was processed when the machine had spare cycles.  It was batch, but a batch of one.

The idea is applicable in today's world.  When I ask people why they are creating a batch job they say something like "the printer is loaded with a certain paper at night so that is the only time we can do this".  *WRONG*  The processing can be done at anytime, the printing can only occur at night.  Instead of doing a lot of processing at night, do the processing in an asynchronous manner throughout the day and do the printing at night.  Split things into two pieces:  processing and printing.  They are two separate functions and do not need to be combined.

Let me rephrase in a more "global consulting" manner:  by separating processes into their unique steps, you create more opportunities to provide value to the business client.  Through a selective process of decoupling the steps in a business process you provide opportunity to inject new business functionality into a more flexible framework with less rework to the existing systems. Isn't that really what the business user wants?  More flexibility?  More value?

Short tempered

Some have said that my comments in the Daily Migration Note are brusque or just plain old rude.  Others have said that I am condescending in my notes.  A number of years ago a project manager actually said he found me "abusive and your attitude repugnant".  (I always keep those good emails so that I can refer to them later.)  Others call me up to see if I am still sane or whether I'm going to go postal.

I am, by my own admission, a strange duck.  Pressure doesn't affect me the same as other people.  I've been in situations where an the vast majority of an organization was sitting on their hands, waiting for me to come up with a solution. I've been the key person involved in resolving a payroll issue that affected tens of thousands of people.  While they are stressful situations, the pressure of the problem, the adrenaline that is produced, all blend together.

And yet, some things make me react quite harshly.  Yes, even quite rudely.  Why?

At a recent event there was an opportunity to evaluate myself based on a number of different characteristics.  Based on my answers to the questions, I was (drum roll please) not a people person.  This may not be a surprise to a bunch of you, maybe all of you.  What the quiz showed me was something that I knew all along:  process and logic are the key factors when I am dealing with work.

So, what does this mean?  Well, if things don't make sense to me I have a tendency of digging my heels in and saying "No".  If there is no logical sequence of events I have a tendency of saying "stop".  If people try to pressure me to do something that I think is illogical, I fight back.

Do you know what actually makes sense to me?  Something that is a logic progression when resolving a problem?  A partnership.  Not a partnership where one or more sides pay lip service to the idea of a partnership so that they can check it off their list of marketing responsibilities (there are many such partnerships amongst major corporations that fall into this category), but a true partnership where each side is working towards a common goal and where all parties are free to bring forth ideas to the table.  No finger pointing.  No "blame game" tactics. 

I have been involved in a number of different groups that were true partnerships between two or more groups and the results are amazing.  With the freedom to make any suggestion about any topic without fear of reprisal the ideas and solutions that come out of such a group are amazing.  There is tremendous difficulty in this as you need to leave your ego at the door and be willing to admit you were wrong.  You need to understand that there is more than one way to skin a cat and that yours may not be as effective.  And, in a true partnership, you don't try to pressure people to follow your lead or your suggestion, you true to influence them and persuade them that your way is appropriate to try.  As soon as you try pressuring people the partnership is gone.  And, in my case, I start resorting to my stock answers of "no" and "you must be kidding".

Partnerships.  Give them a try and you'll be amazed at what you can do.

Debugging

Don't blame the Deployment Team if your application fails and you can't figure out why.

Increasingly we have seen project teams with problems in Production (or UAT for that matter) and they are at a loss to explain what the problem is.  Sometimes there is a cryptic error message in one of the event logs along the lines of "Object reference not set to an instance of an object".  <sarcasm> Now that is a descriptive error message. </sarcasm>

There are a number of things you can do to make your life easier:

• The various .NET languages come with a feature called a Try...Catch block.  Where you try something and if things don't work you catch the exception.  Amazing how that works.  If you haven't put Try...Catch blocks in your code, particularly around any code that is calling a service outside of your control (SHR, EPS, DMS, PRS, etc.), then you score -5 points on the attention to detail rating.
• Did you know that if you catch an exception you can add your own information?  Yes, if you've caught something you can add information such as what you were doing when you got the exception or copies of the data used in the method so that people don't have to guess at what went wrong.  If you just rethrow the exception without adding any information, deduct 5 points from your "value added" score.
• Did you know that there is a Trace class in .NET?  It can do all sorts of things for you, such as allowing you to dump out detailed trace information (that is if you've written detailed trace information in your application) by changing a config file or even dynamically through code.  This is a very handy feature when used in combination with breadcrumbing so that you can see the complete route that an application has taken and, if you've added it, the data used throughout the process.  If you've never used Trace then how do you ever expect to trace anything in Production?

There are lots of things you can do to make your life easier when it comes to resolving problems in Production.  Simple things that can be useful in any environment.  Simple things that you should know how to use and how to use effectively.

Introduction to an old friend

I'd like to introduce to you, an old friend of mine.  The PRINT statement.  Granted, he's not as fancy as a GUI Debugger, nor as sexy as ANT or AVICODE, but he's still really useful.  Have anyone thought of adding PRINT statements to their T-SQL that they are executing?

For instance you could have something like this:

(14 row(s) affected)

(14 row(s) affected)

Or something like this:

Merge Learner Script
(14 row(s) affected)
Learner Merge - From SIN : 918728130 To SIN : 665171625
Start date - 2008-08-22 09:30:03.120
Create an Activity
Close the Activity
Learner Merge - From SIN : 999014679 To SIN : 654457258
Start date - 2008-08-22 09:30:03.420
Create an Activity
Close the Activity
Learner Merge - From SIN : 999011572 To SIN : 647141506
Start date - 2008-08-22 09:30:03.480
Create an Activity
Close the Activity
Learner Merge - From SIN : 917032336 To SIN : 661696161
Start date - 2008-08-22 09:30:03.527
Create an Activity
Close the Activity
(14 row(s) affected)
Done at 2008-08-22 09:30:03.590

The second version looks much better as it is clearly describing what is occurring inside the script.  If you are trying to find out what is going on the second one is so much better.  If you have an auditor on your butt, the second one is so much better. 

Why don't more people do this?  Well, it takes effort.  Typing the characters can consume multiple seconds per statement.  <sarcasm> I mean, over the course of a year this could add up to an hour or more of your valuable time. </sarcasm>

I'd like to thank the SFS team and Janice Pyle for the second example above that clearly shows the advantages of the lowly PRINT statement.

Using a hammer

Have you ever seen Canada's Worst Handyman?  I must admit that I am all thumbs, but, even so, I'm better than these guys.  There are lots of things to look at in this show because so many of the items can be translated into almost any occupation, including the IT field.  One of my favorite things is watching the people, when confronted with a screw, pick up their hammer.  Or using any one of a dozen other tools in the wrong way.

IT people have the same tendency.  We have a problem.  We have a tool.  We try to solve the problem with the tool, regardless of whether or not it is the right tool for the job.  Sometimes you have no tool, just a problem, and in that case almost any help is worth the effort.  Well, maybe not.  Sometimes the effort that the tool saves is totally outweighed by the costs associated with using the tool.  Much like a carpenter that has a plethora of tools to help him do his job, IT people have a variety of tools to help them as well.  Our problem is that there is no simple guide saying "for this problem use this tool and for this problem use this tool".

That's not to say that if we had such a guide people would actually use it.  I mean, look at Canada's Worst Handyman, they were taught certain things and still ignored what was taught.  I have no fear that even if we told people to do X they would still end up doing Y.

I'm not restricting myself to just tools, but methodologies as well.  Some people grab on to a methodology and use it like a life preserver.  Not necessarily a wise thing.  An all or nothing approach is inevitably a bad thing.  Even the Agile Methodology that people either hate or love (there we go with extremes again) understands this.  One of it's principles is:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Essentially, you evaluate what you are doing and adjust if necessary.  I guess my biggest fear is the life preserver mentality for methodologies.  A methodology is a tool and proper use of the tool depends upon proper understanding of the problem.  We sometimes adopt methodologies because they are "cool" or have the "agile" word somewhere in the title.  We don't always understand the problems that we are trying to solve with the methodology and are just so thrilled with using a new tool that we overlook some of the challenges that we have with the tool.

Don't get me wrong, I love new tools, but how many times have you ever used a new tool incorrectly the first time.  If you are going to make a big leap to a new methodology/tool, make sure that you've tested it first and that it actually solves the problems that you need solved.

Quality vs. Speed -- The Agile Way

There has always been a debate over producing a quality application versus an application that is delivered on time.  Indeed, even within the quality camp there are divisions over what exactly constitutes quality.  There have been efforts in the past to merge quality and speed of delivery so that you can get a high quality application in as short a period of time as possible.  The Agile methodology tries to merge these two, to mixed results.

Why mixed results?  Like everything else in life, more than just how you do something determines the end result.  In the case of the Agile methodology (or should I say methodologies) there is more than one way to skin a cat.  The Agile methodology is so different from what people were taught in school that there are three large stumbling blocks that need to be overcome in order for Agile to be truly effective.

• Organization.  There are forces at work in an organization to keep things moving forward.  Plans need to be made, followed and tracked.  The agile methodologies have the idea that "responding to change over following a plan" is a better way to develop software.  And, while I personally believe this, organizations tend to want plans.
• Team. While Agile has come up with some basic philosophies behind how to develop software, the larger the team the more unlikely it is that these philosophies will be effective due to outside pressures: when is going to be done? It is difficult for project managers as they are measured on progress.  Agile, because of its fundamental desire to respond to change, is havoc on plans, at least traditional "we will have this functionality delivered by that date" type of plan.
• Individuals. Everyone comes to a project with baggage:  their old projects and the "we did things this way" ideas.  Getting everyone on the same page, following the same ideas and concepts is very difficult. 

This is not to say that Agile will not work and that you will produce a quality application in less time.  The problem lies in being able to show somebody that this will happen.  Without a proven track record it is hard to show people that it will work, but you need that acceptance so that you can build that track record.  And without the right people in the right places, Agile is never going to fully accepted, fully understood or fully implemented.

Results vs. Methodology

Sometimes it is a story that strikes a chord and makes me write one of these notes.  Sometimes it is an ad on tv.  In this case it is a newsletter from Bruce Schneier, the Chief Security Technology Officer of BT.  He said, that governments should:

...legislate results and not methodologies.

That immediately showed up on my internal radar screen.  While his examples were with regard to things like legislating securing personal data, but not legislating which technologies to use.  Using my own quirky logic I equated this to:

...contract for the end result of the project, not how the project is going to get there ...

When a company puts out an RFP for a piece of work, they routinely say that the winner must "... follow all standards and processes currently in place or that may be put in place during the course of the project...".  We've all seen these as they are not specific to government.  But is it the right thing to do?  Let me go out on a limb and say "no".

In the long run, does it matter if the project team has detailed requirements tracing from the initial conceptual design stage straight through to production?  If you had two projects that delivered the same end result, does it really matter which route they took to get there?  If there are certain deliverables that you want then specify those deliverables in the contract.  Don't specify how you want them created, just that you want them created.

Coming from a programmer background and having been introduced to one of the most document oriented methodologies in existence - Method/1 - I've seen both sides of the coin.  Method/1 was created, to be honest, to minimize liability should the client ever decide to sue.  Dozens, hundreds, thousands of documents are routinely created so that an audit trail could be created in the event that a law suit was launched.  Method/1 was created by auditors, for software development, to be used in the event of a law suit.  Not exactly a glowing endorsement of a methodology.

Other methodologies are created for other purposes, but what it really comes down to is results.  You have certain expectation of results.  And whether the application was written people in blue suits with a document laden process or by a long haired geek who wrote it between episodes of Death Note, it shouldn't really matter, as long as the results required are achieved.

Your Environment is Wrong

"But, it works when I do it, so your environment must be wrong."

When I first heard that comment I took it seriously.  What could be wrong with our environment?  Why couldn't we install the application? 

Not different, because our UAT and Production environments are different than the development environments for a lot of reasons, but wrong.  I searched for something wrong.  Occasionally, I did find something wrong, but as we put more and more applications on to our UAT and Production boxes the number of things that were wrong obviously diminished to the point where their is not a whole heck of a lot that can be wrong.  The differences, however, are still there.  Differences that may prevent the application from installing or working correctly.

• clustered environment - we operate in a clustered environment, so never, ever think that this is the only application that is being installed on this server and never, ever think that it is the only instance of your application which is running
• not a clean environment - the odds are that we've installed your application before - sometimes once or twice before and in some cases a hundred times before - never assume that the environment is "clean" because the odds are that there are remnants of other installs of yours left around
• multiple installs - never assume that just because there is some manual configuration at the end that needs to be done that it is "no big deal" because in our UAT and Production environments we need to do the installs multiple times so that any manual configuration that needs to be done will be done multiple times, giving us plenty of opportunity to make a mistake

"Mistakes"?  Yes, we sometimes make mistakes when manually configuring an application.  Considering that in the UAT and Production environments we are going to be installing and manually configuring your application seven times or more, once we add more servers, this gives us seven chances to make a mistake.  Combine this with multiple releases to UAT and you've got a wonderful opportunity to introduce an unexpected bug, with the most likely problem being that the manual configuration was not saved on a specific machine.  Have the application self-configure itself where possible: prompt for data and complete the installation itself.

Oh, I forgot, you're only going to UAT and Production once, right?