Best Practices

In the past year I've stated a phrase so many times that I began to wonder what I really meant.  I'm sure you've seen the phrase many times before, both in my writing, in job postings, in management briefs, all over the place, but what does it mean?  The phrase is:

"... best practices ..."

This is kind of a loaded question.  I could state that what I mean by best practices is that it is something that I think is important to do.  That would definitely boost my ego, but for the most part when I call something a "best practice" it means something more. 

For the most part when I call something a best practice it is because it is:

An established sets of practices, procedures or methods used to achieve an optimal result.

Now, who defines what an optimal result is in this context?  There are a number of different contributors:

• Academia.  Some things are called best practices because those in the academic world have studied, debated and decided that the best practice in question meets the criteria.  While those in academia do provide some input into best practices, the problem lies in the fact that the academic world is never as complex and complicated as real life.  So, take their opinion with a good dose of realism.
• Standards Groups.  These people create certain standards and they should, theoretically, know how to make the most effective use of the standards.  They are usually more accurate with their proclamations, but, once again, ideal situations are difficult to find and some of their best practices do not work in real life.
• Industry Consensus.  This is something that the majority of people in an industry think is a good thing.  It is usually decided upon by a grass roots movement by people who actually do the work and not strictly academics.  Their opinion is highly valued as it is more likely to provide meaningful results when used in the correct context.
• Organization.  Based on the way that an organization does things, there may be some specific practices that need to be done by an organization to improve the end result.  These best practices may fly in the face of other best practices read and used over the years, but because they are specific to an organization they are probably the most relevant.  (Please note that if the organization changes, but the best practice doesn't then this becomes nothing more than a waste of electronic ink.)

All of these opinions, however, must be tempered with the fact that they were created with certain pre-conditions in place or under certain technological constraints.  A best practice is only a best practice if it actually applies to your situation.  Minor variances can sometimes be ignored, but if your naming standard was based on 6 character RPG standards, they probably aren't applicable to .NET 2.0, so some (un)common sense needs to be used as well.

Some things should not be "added on"

When building an application there are some things that can be added on afterwards:  new functionality, better graphics and friendlier messages.  These are all things that add more value to the application.

There are some things, however, that should not be added on afterwards:

• Error Handling.  What?  Don't add on error handling afterwards?  No.  It needs to be done at the start.  Now, I know what some of you are saying "But, Don, we'll add this in if we have a problem."  Face it, every new application has problems and if you don't have error handling in at the beginning you are spending needless cycles trying to debug the application and you are causing me to drink Pepto-Bismol as if it were Dr. Pepper.  We recently had to help a project debug their application in the UAT environment, but they had no error handling at all, except the default ASP.NET error handling.  Thank goodness for Avicode, as it helped us pinpoint the problem quickly, just far too late in the development cycle.
• Object Cleanup.  If you create an object, kill the object.  It's simple.  It's so simple that even Project Managers can do it.  By not cleaning up after yourself you raise the potential for memory leaks to happen.  And you know what that means?  Alka-Seltzer, Petpo's cousin. I can't tell you the number of applications in which we recycle the component once it hits a certain limit because it would keep you awake at night.  (Lunesta, another cousin)  Suffice to say that many of our applications are forced to recycle after a certain period of time or when they exceed a certain size. 

The scary thing is that both of these items are considered  best practices for writing code in the first place.  I know the excuses "...the project manager won't let me do this ..." or "...I don't have enough budget to do this ..." or, the one heard most frequently "... I don't have the time to do this ...:.  Very few project managers tell their staff how to code, so the first excuse is just a cop out.  As for the budget, doing these items does not add significantly to the cost of application as it usually makes debugging faster and easier, so the budget excuse is just that, an excuse.  As for the time, if you're short on time, you need to do this as it will help you.

One of the things that many Health Organizations are putting in place is prevention of disease so that there is no need to cure the disease.  Prevention is much more cost effective.  Object Cleanup is prevention, pure and simple.  When someone has symptoms that need to be diagnosed, what does the doctor do?  Perform a seance?  Guess?  Or do they use a tool to help them out?  Ever heard of an MRI or even an X-Ray?  Think of Error Handling as a tool to help you diagnose the disease faster.  It's better than guessing.

So, object cleanup prevents problems and error handling helps diagnose problems.  So, I guess this means that I'll be seeing more applications with these items as an integral part of the overall application or do I need to go back to the medicine cabinet?

Post Mortem, but before Death

Justice Gray sent me an interesting note the other day in which he described a "post mortem" debriefing, but before the project had even started.  As Justice put it:  “It’s six months/a year/etc. from now and the project has failed.  Why did it fail?”

One of the things that people have a hard time doing is learning from their mistakes.  George Santayana said "Those who do not learn from history are doomed to repeat it" and that is very true in the IT industry.  We work on projects that are months long, sometimes years, and during each project we make mistakes, fix them and go on to another mistake.  By the end of the project we have discovered, and fixed, dozens of mistakes.  We then make the biggest mistake of all, however, and repeat those mistakes on the very next project.  We failed to learn from our mistakes.

By setting up a post mortem before the project even starts, however, you're asking the developers to think about the project, think about what has gone wrong on previous projects and apply those lessons to the new project.  In essence we are learning from our mistakes and applying the fixes to the next project. 

Does this work?  Well, from my personal experience I've had mixed results.  We did this for a couple of smaller projects and we were quite successful at identifying potential problems early on and work on risk mitigation for those items.  And that is what we are talking about:  risk mitigation.  Identifying a potential problem and what can be done early on to minimize the odds of it happening.  We also tried this same approach on a much bigger project, but the whole process blew up in our faces.  We identified so many potential problems that we spent more time mitigating risk than we did building an application.  And even after all of that advance preparation, we were hit with some nasty problems that derailed the project because of a couple of people who were not committed to the process.

Can this work, discussing potential failures before the project starts?  Most definitely, but it is really important that people think about past mistakes and problems and how to mitigate the risk of them occurring.  It also depends on the commitment of the people involved.  If the people aren't committed to the process then the results may not be what you expect, or need.

Optimization -- Part 3

There's help you can get when you hit the wall for performance?

You bet your sweet bippy there is.  The most obvious places to look for help are with the people around you.  I find that one of the most effective resources I have is my wife.  Now, my wife is not particularly computer literate.  Seriously.  She is, however, an excellent sounding board.  When I explain something to her I have to get rid of the technical jargon and explain it to her in simple terms.  Usually, while in the process of trying to simplify the explanation, my mind goes off on a 90 degree tangent and i solve the problem that I am working on.  This doesn't mean that this will work for everyone, but it works for me.

Or, try reaching out to other designers and asking for their opinion.  Give them a high level overview of the problem, what you've been looking at and let them think for a few minutes.  Don't expect an instant answer, because if something has been troubling you for a while, don't expect someone else to solve the puzzle instantly.  (Although, my wife does really well on those little puzzles where you have to take rings off a rope and untangle hopelessly tangled messes of metal.)

You can even talk to the Deployment Team.  Yes, I know, normally you avoid that, but you'd be surprised at how much help we can be.  By deploying and supporting over 80 applications the odds are that we've seen your problem before and that there are at least two, if not seventeen different ways to resolve the problem and make your life much easier.

Sometimes the answer is really obvious, but because you are so deeply involved you "can't see the forest for the trees".  This is where reaching out to others, both technically adept and technically inept, can help.  It forces you to look at the big picture, the forest, and see whether or not there is a different path.

Secret Meeting

I got my hands on a recording of a secret meeting held between project managers last Friday.  Here it is in it's gory glory.

"OK, so it's agreed, we increase all of our estimates by 10% and blame it on the overhead caused by the Deployment Team.  Right?"

"Ah, hang on a moment.  There might be a problem with that."

"What is it <too garbled to understand>?  What don't you understand about pinning the blame on the Deployment Team?  In particular that pain in the butt, Don."

"Well, don't you think that he might notice what's going on if everyone suddenly complains about his team costing the organization a lot of money?  I mean, I don't think he's a complete moron and he might figure something out."

<sound of people spitting out their beer on to the floor>

"Not again?!?!?"  <sounds like the waitress bring a mop to clean things up>

"What makes you think Jessop can figure this out?"

"Well, he has been in projects before, from a project plan creation perspective and he understands that estimates need to be complete.  And this means that they have to take into account all of the costs of the project.  Some of those costs include interacting with the Deployment Team to ensure that the application is deployed properly."

<long silence>

"You're a plant, aren't you?  You're not really a project manager, your a mole!  Get HIM!!!!"

The recording breaks up at this point as the noise of chairs being tipped over and beer steins breaking on the floor seem to take up much of the remainder of the recording.  There is the faint sound of someone shouting " ... no blood inside the bar ... ", but that could be just my imagination.

So, Project Managers, how are those estimates coming along?

Don't Worry About Performance!

There was a statement made a number of years ago in an ISWG (Integration Services Working Group) meeting which, to summarize, said "Don't worry about performance, we'll take care of that".  While that is probably going to be the epitaph of at least one person, I think it is time to set the record straight.

Worry about performance.

OK, now that we've gone 180 degrees it's time to put some parameters around this.

• Don't worry about things over which you have no control.  The speed of a DCOM call is something you have no control over, neither is the time required to create a connection for a connection pool, the time required to retrieve session state, nor the time required to process an HTTP request.
• Do worry about things over which you do have control.  While you can't do anything about the speed of a DCOM call, you can control, to an extent, the number of DCOM calls that you make.  Less chattiness is better.  While you do not have control over the speed of the resources, you have control over how effectively you use those resources.

The UAT and Production environments into which your application will eventually move has a myriad of interconnected pieces that, together, create the environment in which your application will exist.  While you cannot control those resources you control how you use them.  <Al Gore>  Ineffective use of resources and our unwavering dependency on technologies that produce greenhouse gases is threatening our livelihood and our very planet.</Al Gore>  Ineffective use of resources in any ecosystem is bad, whether that ecosystem is our planet, or our clustered production environment.  Infinite resources are not available and never will be, but effective use can be made of existing technology:

• Up until the late 1990's the computers on board the Space Shuttle flight deck were Intel 8086 chips.
• The original Xbox had a 733MHz CPU with 64MB of memory and yet it could (still can) outperform peoples desktops.
• Mission critical systems have been designed with, when compared to modern technology, primitive means.
• The first spreadsheet I used, Visicalc, ran on a 1 MHz processor.

All of these examples show that you can make something run well in limited circumstance, but you have to want to.

Identity Theft

I guess I can say that I am now a statistic.  You know, one of those millions of people who have been the victims of identity theft.  Let me tell you the story.

When I got home from work on Monday I noticed that I had supposedly been sending out emails from eBay account at lunch.  Within 15 minutes of the start of the emails I received an "A26 TKO Notice: Restored Account" from eBay UK stating that:

It appears your account was accessed by an unauthorized third party and used to send unsolicited emails to other community members, including email offers to sell items outside of eBay. It does not appear that your account was used to list or bid on any items.

The first thing I tried to do was log into my account.  Well, something either eBay UK did or something the hacker did was change my password.  I tried to enter in my answer to the Secret Question, but that didn't work either as the information on my account had been changed.  Following the various prompts on the eBay site I ended up sending them an email telling them what had happened and what the next step should be.

A couple of hours later I got another email that I apparently sent out eight hours after the first round.  Not content to sit by and wait for the email process to work its way through the system I then started scouring the eBay site for a phone number to call.  You know, that is one of the hardest things I had to do!!!!  I followed all the usual routes and ended up with forms to fill out.  I never did get a phone number, so I had to use their "Live Help" facility.  (My reluctance to go with this approach was due in part to a 45 minute wait on the weekend for "live help" from another company, which never even connected with a human being.)  In the case of eBay, however, the wait was less than two minutes, they told me my position in the queue (started at number 5) and the approximate wait time. 

The person who was on the other end of the chat could have been anyone, anywhere in the world.  The fact of the matter is, they looked at the information on my account, the notes they had sent to me and knew that I needed to talk to the Account Security division.  Within 30 seconds I was "chatting" with someone else who had the power to help.  Two minutes later things were fixed and that included changing the password on my account to a "stronger" password.

Was it brute force hacking of my account and password?  Not if this article is correct.

This particular episode was rather benign in that all that really happened was that some emails got sent and I had to change my password.  It could have been worse.  Much worse.  Think of that the next time you sign up for a web site.  Or, more importantly, think of that the next time you are building an externally facing application.  What are you doing to safeguard the information that you keep on your clients?  What are you doing to protect their safety?  Can you honestly say that you've done your best?

Validity vs. Reasonableness

While most of our applications do validity checks on data, not all of them do reasonableness checks.  Let me explain the difference.

Data Validation.  Let us suppose you have a number of fields on the screen:  name, address, birth date, phone number, and spouse's name, spouse's address, spouse's birth date, spouse's phone number and a marriage date.  Data validation would ensure that if there is a birth date, it is a valid date.  So in this case it would check to ensure that all of the date fields are valid.  It would also check to ensure that the phone number follows any one of a number of different standards, but predominantly the fact that it is numeric in nature.  You can also extend data validation to more complex tasks such as determining if the postal code is correct.   In general terms, data validation serves to ensure that a single piece of data is a valid for that data type.

Data Reasonableness.  OK, now that we've gotten the basics out of the way, there are still a number of checks that we can perform.  If there is a marriage date, then the date must be a certain time period after the birth date of both parties.  This is not just a simple "if marriageDate > spouseBirthDate then Happiness()".  We need some additional logic to ensure that even if the data is valid, it also must make sense.  Having data make sense is as important as ensuring that it is valid.

While there are many schools of thought on this, most post secondary training lumps both data validation and data reasonableness together under the "validation" banner.  This, unfortunately, has had the effect, in most cases, of putting data reasonableness checks in the background or has the checks embedded deep within the business logic of the application.  In most cases these checks can be done at the UI level, really quickly and prevent a lot of background processing that clogs the servers.  The other big problem is that some of these reasonableness checks are missed because "I thought the client was going to do that".

Just remember, there is a lot of data checking that needs to be done and reasonableness is just another item in the list.

Temporary (Expires yyyy/mm/dd)

My wife renewed her drivers license recently, as she turned 29, again, on September 24th.  She went into the Registry near our place, filled out the form, paid her money, got her picture taken and was given a "temporary" drivers license to last her until her real license was sent to her.  If the real license takes too long her temporary license is going to expire on her and that could cause no end of grief.

My bus takes a little bit longer to get to work in the morning as there is a "temporary" detour on it's normal route.  The city has dug a hole in the road, the entire width of the road, in order to do some emergency repair on the pipes.  They don't have a lot of time to get this done as it impacts a lot of traffic, a lot of buses (both ETS and school) and a lot of residents.  The only way to get to certain houses is to wind your way through back alleys.  Oh yeah, this is definitely going to be temporary.

My daughter has a temporary spacer in her mouth.  One of her baby teeth, incorrectly filled by an earlier dentist, developed some severe damage and needed to be pulled.  A temporary spacer was inserted so that her teeth would grow in their proper place until the adult tooth comes in.  The spacer is going to be removed either when the adult tooth comes in, or when the baby tooth to which it is attached falls out.  No choice, it is temporary.

We have servers, both physical and virtual, which were set up for temporary purposes.  We now have to face the task of upgrading them from NT 4 to something supported.  (OK, I lied, but you get the point, don't you?)  If things are temporary, give us a date and we will set up the system to self destruct on the day after.  If things aren't temporary, for goodness sake, tell us!!!!!  The "Oh, it was temporary, but the client said ..." story is getting old and, quite frankly, has been done better by other teams. 

Remember temporary means that it goes away.  Pick a date.  Any date.  Please....

Weapons of Mass Destruction

America went to war with Iraq because they wanted to find and destroy the Weapons of Mass Destruction.

In many respects that what I do by looking at the way applications are installed, operate and behave when encountering errors:  I'm looking for weapons of mass destruction.

My first IT job was with a construction company and my crowning achievement was an application that accurately allocated work site costs to various job codes in the accounting system on a daily basis.  It was rather tricky due to the fact that the company I worked for was a multinational company and each country, indeed province/state, had different holidays so it needed to take into account when costs should be allocated based on whether or not the previous day had been a holiday or not, in the location where the construction site was located.  Jobs on which 7x24 construction was occurring had other conditions that needed to be met.  All in all it was a masterpiece of software engineering.

Almost.

You see, I was under a tight timeframe for getting this done, as the VP in charge of construction had told the CEO that it would be in place by July 1st.  I only had 4 more weeks to finish the coding and then implement the application in the main production batch jobs.  Time was tight so I did what every rookie (and many seasoned professionals) do when faced with something that is difficult to compute:  I hard coded the answers.  I hard coded the holidays for all jobs sites for the next 18 months inside the application. 

It was simple to do and saved me a lot of trouble, because, you see, I left the company 2 months later, leaving behind a ticking time bomb in their production systems.  In sixteen months things were going to blow up, all because I took the easy way out instead of doing it properly.

Tick, tick, tick, tick ...

Groceries and Software

When you buy groceries from a grocery store, one of the things that they do for you is pack your groceries into plastic bags so you can take them home.  (OK, some grocery stores don't do that, but they will charge you a couple of pennies to give you a plastic bag so that you can do it yourself.)  When they pack the plastic bag they have certain, spoken and unspoken, rules.  For instance, you don't put a pound ... err, kilo... of hamburger in the same bag as fruits and veggies unless one of them is wrapped in an additional plastic bag.  You don't put a bag of potato chips at the bottom of the bag and potatoes on top of them and eggs are packed flat.

When creating a deployment for DeCo, think of ZIP files as being the similar to plastic bags from the grocery store.  Here are a few simple rules that you can follow to fill those bags:

• Create a ZIP file for each item type that you are migrating.  For instance, if you are migrating a web components and COM+ components, create two zip files, one for each.
• In each ZIP file put all of the pieces necessary to do the work for that type of deployment.  If it is a web zip file, include the MSI, any config files and even the documentation. 

The rules are really simple, but make life much easier for everyone.  For instance, by using ZIP files instead of a list of separate files, you reduce the number of times you need to attach a file to the request and you reduce the amount of space needed on the back end to store the files.  We current have over 9760 files that we are tracking from over 6800 deployments and these files take up over 19,680,000,000 bytes.  While some of those files are compressed, not all of them are. 

By using ZIP files you can help us keep a handle on the storage requirements for DeCo as well as making it easier for the Deployment Analyst or DBA to get all of the files they need for the deployment in one simple package.

Schopenhauer's Law of Entropy

So, just what is Schopenhauer's Law of Entropy?  Simply put, it is this:

If you put a spoonful of sewage in a barrel full of wine, you get sewage

So, what does sewage have to do with programming?  It's not sewage that I'm looking at, but rather the concept behind it.  In IT terms, what Schopenhauer is saying is that no matter how good the overall application, if one part doesn't work the whole application gets tarred with the bad brush. 

It is unfortunate that a single poorly designed, written or executed page can make someone believe that the entire application is poor.  Their perception of the application is what is important, not reality.  Kind of scary, isn't it, when perceptions are more important than reality.  But this is what happens in our business and it is something that we need to understand and do our best to influence.

So what influences this perception?  Well, consider this:  two web applications side by side on your desktop.  You push a button on the left one and you get the ASP.NET error page:  unfriendly, cryptic and somewhat unnerving.  You push a button on the right one and you get an error message in English, that explains there is a problem and that steps are being taken to resolve the issue.  Which one would you perceive to be better written and robust? 

How about another example?  You push a button on the left application and you get an error message that says "Unexpected Error.  Press the OK button".  You push a button on the right application and you get an error message that says "Our search engine is currently experiencing some difficulties and is offline.  Please try again later."  Which one do you perceive to be better?  Which one do you think your business clients will think is better?

It's not just one thing (error message or not) that gives you a feeling of confidence when dealing with an application, it is a multitude of little things.  Making things more personalized helps.  Translating from Geek ("Concurrency error") to English ("Someone else has updated the data before you") helps a lot.  Making it seem that you spent some effort to foolproof the system (i.e. don't make every error number in your application the same error number).

No matter how good the rest of your application, one bad move can create sewage.

Antacids for the PM

PMs, can you imagine the look on the faces of those developers when I said that they needed to supply you with things like estimates and time sheets?  I bet some of them were knocked off their feet!

I mean it's not like you were asking them for anything difficult.  I mean, how hard can it be to create an estimate for a project?  It's not like it's rocket science.  Every developer should be able to do it.  I mean, the National Gun Registry hit it's target.  Well, maybe it didn't.  Well, what about the Denver luggage handling system?  Another fiasco?  OK, the FBIs Trilogy Project, now that was ... a disaster?

Everybody can look into the past and come up with failed estimate, but sometimes, the Project Manager does it to themselves.  This may be a shock for some of you, but sometimes Project Managers are under different pressures than you realize.  Back when I was younger I worked for a consulting company.  My job was to come up with the technical work plan and estimates for the projects the local office undertook.  I would then, using historical data and some darn fine guessing, come up with what the effort would be on the technical staff for the project (technical staff meaning project DBA and internal technical support). 

One of my proudest, and saddest, estimating moments came for a relatively simple business project that had a number of interesting technical complexities.  My estimate of the technical cost was 179 days.  When combined with the other parts of the project it turned out that this simple application was going to cost the client a lot of money.  In an effort to reduce the impact tot he client the scope was shuffled, but this did not reduce the technical effort that needed to be spent, so in a classic PM moment the number of days was arbitrarily reduced to 79.  I am proud/sad to say that this is one of the few times in my life where I was dead on with regard to the effort.

Project Managers, if your team says that something is going to take a certain amount of time, ask them questions, make sure they understand both the problem and their own estimate, but don't arbitrarily change it unless you know for a fact it can be done for less.  The odds are they understand what needs to be done better than you and by changing their estimate you are telling them that you don't trust them.  If you still feel the estimate is high, have them sit down with you and walk you through the estimating process they used.  But, if the numbers still add up to something you don't like, take a tums.

Load Balancing Failures

It shouldn't come as a surprise to anyone when I say that our Production environment is load balanced.  I have mentioned this before and I will be mentioning it again in the future.  But, for those who may have missed my previous tirades, let me explain the impact of load balancing on Session State.

One of the features of ASP.NET is to store information specific to a browser session (aka user) into something called Session State.  Session State is kind of like the junk drawer you have at home where you have batteries, twist ties, plastic spoons, stud finders and assorted other "stuff".  Session State allows you store what you need to store in order to keep track of where the user is in the application and what data you need to save on their behalf.  The next time the user accesses the application the session state is automatically loaded and you're ready to rock.

There are a number of places to store session state:  In Process, Out of Process or in SQL Server. 

In Process means that Session State is going to be stored in the Application Pool running the web site.  So, if the application pool recycles, all session state is going to be lost.  A number of projects currently use this method and are in danger of losing Session State, if they use Session State, as we use Application Pool recycling to solve a number of application issues.  In addition, if, for some reason, BigIP sends the user to a different web server to service the request then the Session State is not going to be present, potentially causing a number of application failures to occur

Out of Process is where the session state is hosted by ASP.NET in a different process, potentially on a different machine.  While somewhat safer than storing it in the same Application Pool, a problem arises if this service needs to be reset whereby the Session State is again lost.  Indeed, if the process is hosted on the same server as the web site, moving the request to another part of the load balanced cluster is going to be a problem as Session State will not be available for the request.  If Session State is stored on a separate machine then the biggest problem is that of durability of the data.  Any problems with the service may wipe out all session state for all machines.

Storing Session State in SQL Server is the slowest method, but is by far the safest method for durability and the best method when utilized in a cluster.  Each request for Session State goes out to SQL Server to ensure that the latest and greatest version of Session State for that user is retrieved and used.

In our environment we have asked people to use SQL Server Session State, and yet, by looking trough the web.config files of a number of projects I've noticed that they have their Session State set to In Process.  If Session State is actively being used, this is a recipe for disaster.  I urge each project team to take a quick look at their web.config files and change it to use SQL Server instead of In Process.  Even if you don't currently use Session State, you may in the future and this will prevent you from having a nasty accident.

Trust

One of the biggest things we need to deal with in applications is trust in other parts of the system.  How much trust do you have in the calling application?  How much trust do you have in the common routines that you call?

For instance, you've created a method that will take an XML stream and use it to update a row in a table.  How much faith do you have that the calling application has sent you the proper types?  Should you check everything to ensure that strings are strings and numbers are numbers?  Do you double check to ensure that dates are actually valid dates and that the time listed actually exists?

I used to work for a manager that insisted that every time your method gets invoked it should double check all of the data being passed in before it did any work: verification was the first thing you did.  Being young and full of myself, I didn't follow that rule because, well, to be honest, I was writing both sides and I knew what I was passing myself!!!  Fast forward a couple of years and someone else is maintaining the code.  Well, they made some changes that didn't follow the rules and, in production, it blew up horribly because the method did not verify that the correct type of data was being passed.  Being on the support side I was called in to troubleshoot and instantly recognized what the problem was and the solution that was required.  A quick compile and test and the application no longer died horribly, but gave a nice, easy to understand error message.

With today's modern languages much of this work is taken care of for you by the development tool during design time as you need to ensure that you are calling with the correct types, or the the compiler won't even compile your application for you.  However, there is a problem when you are using XML or if you are taking in a string and attempting to use it as a numeric value.  This is of particular concern to user interfaces as pretty much everything you retrieve from the UI is a string that you need to convert and use.

A user interface should place no trust in the end user entering in the correct type of data into the text box.  But, how much trust do you place in one piece of code calling another piece of code?  I guess that depends on whether or not you are going to be the person maintaining the code for the lifetime of the application.  If you are, then I guess you can trust the code.  If you aren't then being paranoid may be beneficial.

n-Tier environment ... design

One of the biggest differences, at least in terms of perspective, between a Win32 application (aka Fat Client) and a web browser application is that the Win32 application is used by a single person whereas the web application may be used by hundreds, or thousands of people at one time.  OK, quiet down out there, let me explain.

A Win32 application is run on a desktop and is run by a single person at a time.  The application may be installed on hundreds or thousands of machines, but each machine has a single person running the application.  In this manner the workload is distributed between each workstation and the database server.  In a web based application all of that processing needs to occur somewhere.  While some of it happens on the browser, a good portion of it runs on the web server.  Therein lies the problem.

Many web applications are built in such a manner that they consume large amounts of CPU and memory in order to serve a single client.  Multiply by the number of people that are going to be accessing the application and you have a nightmare on your hands. Many years ago servers used to be much more powerful than desktops.  This was due to the advance architecture used in the servers and the very expensive chips used.  As prices have dropped, however, this distinction has almost completely disappeared.  Indeed, a new desktop may, in some circumstances, be faster than the server to which it is communicating.  Because the average developer has a fairly powerful machine, what seems to run quickly on their desktop completely bogs down on the server when multiple people try to access it.

Our current Production servers are quite large and contain a lot of raw horsepower.  We do not have a large number of concurrent users.  I would personally be surprised if we hit 40 on any one server.  This is in comparison to a server at Google that is designed to support hundreds of concurrent users.  On-line games, such as World of Warcraft, support thousands of concurrent users.  While we don't need to write our applications so that we can support thousands of concurrent users, we should always be cognizant of the fact that our application does not operate alone.  It needs to co-exist with itself and others.

Batch Processing ... Part Two

I must be honest, I am sometimes quite surprised by the reaction my notes some times invoke.  For instance, the note about batch processing generated a number of "high fives" in the hallway and a couple of 500 word responses.  The response was, as I expected, all over the place with some people telling me I was not in my right mind while other people were saying that this is what they have always believed.  Never one to let the flames of disagreement burn out, I thought I would list my personal rules as to whether or not something should be done in batch.  This is not something you need to follow, but it may help illuminate my comments from the other day.

Interfaces with other systems.  I originally had the word "external" but I replaced it with "other" as different systems internal to an overall application may not be able to handle a continuous stream of data.  For instance, our interface to IMAGIS is done through a file transfer which is done once a day.  To ensure that we get as much as possible into the system we do the transfer close to the cut off time that we have arranged with the IMAGIS team.  In this case the target system is just not designed to handle us sending them information multiple times per day.  Now, it may be the case that there are other ways to communicate with IMAGIS that we have not utilized, but with the current set up, we need to do it on a scheduled, batch basis.

Reports. This seems like a logical item to do in a batch process.  Whether this batch process is done via another tool, such as ReportNet scheduling the report, or whether it is scheduled via Windows Scheduler, reports are good batch residents. However, in my mind a report does not process and create data, it merely reports on the data.  If your "report" creates and stores data then that portion should be separated out and done in an asynchronous manner.  A report, any report, should be able to be generated quickly from data that is already stored in the database.  In addition, in many cases, reports do not need to be run on a scheduled basis, as long as the report can be generated on demand and that it will contain the identical content as if it had been generated on a previous day.  For instance, if reporting on the applications that were approved on July 12th, it would list all approvals, even if one of the application was subsequently denied.

And that's pretty much it.  It's a very short list of things that need to be done in a batch window.  For me it all comes down to this:

It is our job as IT Professionals, however, to not just do what our clients say, but to educate them as to what can be done, to show them new opportunities, and to give them something better than what they had, not just something newer.

The n-Tier world .. hardware

Does an n-tier hardware environment actually make things better?  I was asked this question recently and, to be honest, it made me pause and reflect on the promises of the n-tier world versus the reality of an n-tier world.

The n-tier push really started gaining hold, again, in the 90's when the Internet was young and foolish and so was Al Gore.  As a result, most people associate the idea of an n-tier environment as one that is web browser based.  While this is not always the case, the majority of applications currently being developed are web based, so we will run with that and assume, for the purposes of this discussion, that our n-tier environment consists of a web browser communicating with a web server that in turn communicates with a database server.

With regard to the web browser, the idea was that with a "light" front end that was downloaded every time you requested it you did not need as much processing power on the desktop (Internet Computer anyone?) and you could make changes to the application without having to distribute an application to hundreds, thousands or even millions of customers.  This has proven to be a valuable and worthwhile objective of browser based deployments and allows for quicker changes with less client impact.

Separating the main logic on to a web server or cluster of web servers (see previous notes about this) then allows the developer to change the application and only have to do it in a limited number of locations.  While this has allowed the developer to deploy applications quickly, the problem here lies in the fact that the developer(s) build the application as if it was the only thing running on the server, when in reality it is usually one of many applications.  Resource contention (memory, CPU) usually mean that going to a cluster of servers is a requirement.  It is also a common misconception that adding more servers will make the application run faster.  Adding more servers allows you to support more simultaneous users, but does not necessarily make the application run faster.  As a result, a poorly performing application will perform just as poorly one machine as on a cluster of 20, although you can annoy more people on cluster.

By placing all of the database access on a machine, or cluster of machines, there are fewer connections to the database that need to be monitored and managed on the database server.  This reduces memory usage, CPU usage and allows the database to concentrate on serving data to clients.  Unfortunately, this is the where one of the biggest problems is in the n-tier world.  Developers need to optimize their code when accessing the database.  Whether it is reducing locks and deadlocks, reducing transaction length or simply designing better applications and databases, the database back end, regardless of whether it is an Intel box, a Unix server or an OS/390 behemoth can only handle so much work.  Web servers can be clustered, but in order to get more than one database server to service requests against the same data you need to have a much more sophisticated and much more complicated environment.  Just adding another server won't cut it as the database server is constrained in terms of the memory and CPU we can use.

So, has n-tier lived up to it's promise?  Sort of.  The web browser side:  yes.  The web/application server side:  mostly.  The database side:  not as much as expected.  The problem is not the technology, rather the people.  We have created an infrastructure that can do great things.  What we need to do now is teach people how to create great things with that infrastructure.

Clusters

Clustered servers.  One of the most important reason for having clustered servers is so that in the event that one server fails the other servers will pick up the load and take over for the failed server.  However, there are some things you need to know before everyone goes rushing off to say "Clusters will solve all of our problems."

Applications must be built to function in a cluster.  So, what does this mean?  It means that if you are going to store a file being exchanges with the client, the file needs to be stored in a location that all servers can access.  It means that if session state is stored it needs to use SQL Server as the storage mechanism as this is used across all servers.  Essentially, nothing should be stored on the local machine.  Nothing.

Connections are "sticky", but don't depend on it.  By default the load balancer will try to send the user back to the same server that they were on previously.  However, in the event that the server is out of the cluster or there is a particularly heavy peak load on the server, the load balancer may send the user to another server.  This is expected and desired from a load balancing perspective, so your application better not rely on the user always going back to the same server.

Clusters do not protect against soft failures.  If a server dies, blows up, is incinerated by aliens from a distant galaxy, is cryogenically frozen in a block of nitrogen, or has it's network connection severed, the load balancing software will automatically move users to one of the other servers in the cluster.  However, in the event that the error is a soft error, an error in which the application responds, but does so incorrectly, the load balancer does not know that there is a problem and will continue to send users to that server.  Indeed, if the error is pervasive and actually causes all pages to fail, and fail quickly, the load balancer may be confused to the point where it thinks that the server is not under a large load and that it can handle more connections.

If you keep these things in mind you'll understand that while clusters do help us out a lot, there is some work to ensuring that they can help and that the help is what we are expecting.  Overall, however, the benefits of a clustered environment far outweigh the disadvantages.

Deadlocks ... Part 2

So, how do we prevent deadlocks?  Well, there are a number of different ways to go about doing it, so we'll only concentrate on a couple.

Small.  This may seem obvious, but it is one of the most overlooked items in the book.  The smaller the transaction (the fewer rows that have been updated, inserted or deleted) the less likely you are to trample on someone else.  The fewer locks you hold the better off you'll be.  One of the biggest problems in this regard is applications that update a row, even when it doesn't need to be updated.  The application follows a specific path and part of that path says that Row X in Table Y needs to have certain values.  The row is not checked to see if it has those values, it is just automatically updated.  Well, that placed a lock on that row even if the values remained the same.

Fast.  This may also seem obvious, but the shorter the amount of time you hold those locks the less likely you are to have a problem with deadlocks.  One of the classic problems here is that the application does something to the database right away, say updating a table indicating where the user is at in a process, and then does a lot of calculations or a lot of work and then does more updating at the end, interspersed with a smattering of updates / inserts / deletes.  The locks are held from the first update to the final commit and as that length of time increases, the more likely it is that deadlocks will occur.

Application Developer Guide.  OK, this one is a little esoteric, so let me explain.  One of the ways that deadlocks can happen is if one part of the application updates tables in this order - Table A, Table B, Table C - while another part of the application updates tables in the reverse order.  By explaining the order in which tables should be updated in the application's developer guide everyone on the project will know the order that they should be doing things.  Changing the order should be something that is discussed with the DBA prior to being implemented as changing the order may impact more than just deadlocks.

As usual, there are dozens of other things to take into account, but these are some of the items with the biggest payback.