The n-Tier world .. hardware

Does an n-tier hardware environment actually make things better?  I was asked this question recently and, to be honest, it made me pause and reflect on the promises of the n-tier world versus the reality of an n-tier world.

The n-tier push really started gaining hold, again, in the 90's when the Internet was young and foolish and so was Al Gore.  As a result, most people associate the idea of an n-tier environment as one that is web browser based.  While this is not always the case, the majority of applications currently being developed are web based, so we will run with that and assume, for the purposes of this discussion, that our n-tier environment consists of a web browser communicating with a web server that in turn communicates with a database server.

With regard to the web browser, the idea was that with a "light" front end that was downloaded every time you requested it you did not need as much processing power on the desktop (Internet Computer anyone?) and you could make changes to the application without having to distribute an application to hundreds, thousands or even millions of customers.  This has proven to be a valuable and worthwhile objective of browser based deployments and allows for quicker changes with less client impact.

Separating the main logic on to a web server or cluster of web servers (see previous notes about this) then allows the developer to change the application and only have to do it in a limited number of locations.  While this has allowed the developer to deploy applications quickly, the problem here lies in the fact that the developer(s) build the application as if it was the only thing running on the server, when in reality it is usually one of many applications.  Resource contention (memory, CPU) usually mean that going to a cluster of servers is a requirement.  It is also a common misconception that adding more servers will make the application run faster.  Adding more servers allows you to support more simultaneous users, but does not necessarily make the application run faster.  As a result, a poorly performing application will perform just as poorly one machine as on a cluster of 20, although you can annoy more people on cluster.

By placing all of the database access on a machine, or cluster of machines, there are fewer connections to the database that need to be monitored and managed on the database server.  This reduces memory usage, CPU usage and allows the database to concentrate on serving data to clients.  Unfortunately, this is the where one of the biggest problems is in the n-tier world.  Developers need to optimize their code when accessing the database.  Whether it is reducing locks and deadlocks, reducing transaction length or simply designing better applications and databases, the database back end, regardless of whether it is an Intel box, a Unix server or an OS/390 behemoth can only handle so much work.  Web servers can be clustered, but in order to get more than one database server to service requests against the same data you need to have a much more sophisticated and much more complicated environment.  Just adding another server won't cut it as the database server is constrained in terms of the memory and CPU we can use.

So, has n-tier lived up to it's promise?  Sort of.  The web browser side:  yes.  The web/application server side:  mostly.  The database side:  not as much as expected.  The problem is not the technology, rather the people.  We have created an infrastructure that can do great things.  What we need to do now is teach people how to create great things with that infrastructure.

Clusters

Clustered servers.  One of the most important reason for having clustered servers is so that in the event that one server fails the other servers will pick up the load and take over for the failed server.  However, there are some things you need to know before everyone goes rushing off to say "Clusters will solve all of our problems."

Applications must be built to function in a cluster.  So, what does this mean?  It means that if you are going to store a file being exchanges with the client, the file needs to be stored in a location that all servers can access.  It means that if session state is stored it needs to use SQL Server as the storage mechanism as this is used across all servers.  Essentially, nothing should be stored on the local machine.  Nothing.

Connections are "sticky", but don't depend on it.  By default the load balancer will try to send the user back to the same server that they were on previously.  However, in the event that the server is out of the cluster or there is a particularly heavy peak load on the server, the load balancer may send the user to another server.  This is expected and desired from a load balancing perspective, so your application better not rely on the user always going back to the same server.

Clusters do not protect against soft failures.  If a server dies, blows up, is incinerated by aliens from a distant galaxy, is cryogenically frozen in a block of nitrogen, or has it's network connection severed, the load balancing software will automatically move users to one of the other servers in the cluster.  However, in the event that the error is a soft error, an error in which the application responds, but does so incorrectly, the load balancer does not know that there is a problem and will continue to send users to that server.  Indeed, if the error is pervasive and actually causes all pages to fail, and fail quickly, the load balancer may be confused to the point where it thinks that the server is not under a large load and that it can handle more connections.

If you keep these things in mind you'll understand that while clusters do help us out a lot, there is some work to ensuring that they can help and that the help is what we are expecting.  Overall, however, the benefits of a clustered environment far outweigh the disadvantages.

Flow Control

Computer languages contain a number of things generically called "flow control".  These construct (words) are what allow a programmer to change the course of the application.  They contain things such as:

• if ... then


• do ... while


• while ... wend


• plenty of other examples

The one thing they have in common is that they have the potential to do something based on a condition.  But you know what?  Every time you execute one of these flow control statements you have the potential of slowing your application down.  You also have something else you need to check.  In addition, if you've chained a number of these statements in a row you must ensure that you are checking them in the proper order.

Essentially, what it comes down to is trying to avoid these statements in the first place.  If you want to get carried away, there are some object oriented purists who believe that you should be able to write your entire application without the use of an if ... then statement.  (My personal belief is that while this is an interesting thought, if ... then can save a lot of extra coding.)

This is also true of any business processes that may be implemented by the project.  Avoiding the use of flow controls makes the process easier to use, easier to follow and easier to understand.  So, whether it is an application or a business process, minimize the use of conditional statements and you'll find that things go faster, smoother and with fewer problems.

Hot Fixes

What is a hot fix?  This question seems to be coming up  more often and I think it needs a bit of discussion in this arena.  Definitions of hot fix that I have seen include:

• A hotfix is code (sometimes called a patch) that fixes a bug in a product. (Source)


• Microsoft's term for a bug fix, which is accomplished by replacing one or more existing files (typically DLLs) in the operating system or application with revised versions. (Source)

I think we can all agree that a hot fix is something that fixes a bug.  The question now arises as to the size of the patch.  The second definition is important in this aspect as it talks about replacing one or more DLLs.  So, a hot fix will fix a bug by replacing an indeterminate number of DLLs.  Darn it, I've used that word again: replacing.  That happens to be the crux of the problem that we are experiencing.

Replacing DLLs does not mean the uninstaling of the entire application and the installation of a new version of the application which has the bug fix inside.  This is simply an install of the application.  A hot fix would take the DLLs that were changed, package those up and install those on affected machines.  This is standard practice used by Microsoft, IBM, Sun, Oracle, Hewlett Packard, PeopleSoft, SAP, Symantec, Trend Micro, Adobe, Electronic Arts, Intuit, AutoDesk Check Point, and, quite literally, millions of other companies.  You don't re-install Windows every time there is a hot fix for Windows.  You don't re-install your anti-virus software every time there is an update to the software.  You don't re-install your entire application because there is a spelling mistake on a page.

If you are asking for a migration to a Shared Environment, and you are essentially asking us to install a new version of the application, don't call it a hot fix, as you are disagreeing with the vast majority of the IT world and the definition that the Deployment Team uses for a hot fix.  A hot fix replaces DLLs.  By packing everything up into a new install for the application you are potentially including other changes in your fix that are not related to the bug you are trying to install.

Deadlocks ... Part 2

So, how do we prevent deadlocks?  Well, there are a number of different ways to go about doing it, so we'll only concentrate on a couple.

Small.  This may seem obvious, but it is one of the most overlooked items in the book.  The smaller the transaction (the fewer rows that have been updated, inserted or deleted) the less likely you are to trample on someone else.  The fewer locks you hold the better off you'll be.  One of the biggest problems in this regard is applications that update a row, even when it doesn't need to be updated.  The application follows a specific path and part of that path says that Row X in Table Y needs to have certain values.  The row is not checked to see if it has those values, it is just automatically updated.  Well, that placed a lock on that row even if the values remained the same.

Fast.  This may also seem obvious, but the shorter the amount of time you hold those locks the less likely you are to have a problem with deadlocks.  One of the classic problems here is that the application does something to the database right away, say updating a table indicating where the user is at in a process, and then does a lot of calculations or a lot of work and then does more updating at the end, interspersed with a smattering of updates / inserts / deletes.  The locks are held from the first update to the final commit and as that length of time increases, the more likely it is that deadlocks will occur.

Application Developer Guide.  OK, this one is a little esoteric, so let me explain.  One of the ways that deadlocks can happen is if one part of the application updates tables in this order - Table A, Table B, Table C - while another part of the application updates tables in the reverse order.  By explaining the order in which tables should be updated in the application's developer guide everyone on the project will know the order that they should be doing things.  Changing the order should be something that is discussed with the DBA prior to being implemented as changing the order may impact more than just deadlocks.

As usual, there are dozens of other things to take into account, but these are some of the items with the biggest payback.

Deadlocks ... Part 1

Deadlocks are an interesting condition within the database.  In very simplistic terms, it occurs when two people are both trying to change data that the other person has already changed.  For instance, Person A has already updated Row 1 in Table 1 and now wants to update Row 2 in Table 2.  However, Person B has already updated Row 2 in Table 2 and is trying to update Row 1 in Table 1.  As you can see, unless someone gives in they could sit there all day.  The database is the arbiter in this case and makes a decision as to which person is going to get the error message when their transaction is canceled.

Now, you may have heard the phrase "Deadlocks are a natural part of application processing and are not a large concern."  While I do not advocate violence, the person who says this should be slapped in order to knock some sense into them.  Deadlocks are not a natural par tof an application.  I previously worked on a web-based system that had, as regularly peaked at over 650 simultaneous users, with over 250 of those being "hard core" users. This is 10 times the size of any application we have currently running.  If we got a single deadlock during the day we had to investigate why the deadlock occurred and determine if the deadlock could be prevented.  We would go for weeks, or even months without a deadlock, but when one occurred we dropped everything and investigated the problem.

Why do we get deadlocks in an application?  Sometimes it is because in one part of an application we update Table 1 and then Table 2, but in other parts we update Table 2 and then Table 1.  This is a disaster waiting to happen.  In other circumstances we have background tasks running that are not properly tuned and they try to update too much at once before committing their data.  This is also another disaster.

There are a number of surprisingly simple and, to most people, obvious strategies to use that will eliminate deadlocks to the "rare" occurrence that they should be.  We'll discuss those tomorrow.

Interim

Interim is an interesting word.  When people look at the word and use it, they might actually have a different opinion as to what the word means.  For instance, they may say that something is "an interim solution".  Based upon how most people in the IT world use the word they would assume that the solution is a short-term solution and that something is coming to replace it.  They would be wrong, in more ways that one.

Strictly speaking, the definition of interim is something like this:

... the period of time between one event and another ...

Not very specific is it?  According to the definition, an interim solution would be a solution that is in place prior to the final solution being installed.  IT people, however, haven't really been using the word in quite this manner, or rather, they have been hoping the word is not used in this manner.  They honestly believe that the phrase "short term" is in the definition.  Sorry, but it isn't there.

What this means is that if you specify an "interim solution", you had better be comfortable with that solution because there is no time limit on how long that solution will be in place.  If you are not comfortable with the solution as a long-term solution, do not promote it because many interim solutions have become long term solutions, regardless of how much we dream and pray.

Just Because ...

Just because you can do something, doesn't mean you should.

Geeks like new things.  They like playing with new technology.  Using new tools.  Trying out stuff that they've never done before.  XML is a good example.  XML has a lot of really good useful purposes.  It allows for easier portability of data between disparate systems.  It is, for the most part, human readable.  It is good for configuration files.  Indeed, configuration files are probably one of the most commonly used applications for XML.

There is one thing (okay, probably more, but this topic needs to be short enough to keep your attention) that XML shouldn't be used for:  continuously updated reference tables or codes tables.  For instance, you wouldn't put the conversion rate between U.S. and Canadian dollars into an XML for processing by an application because it would involve daily updates to that file.  You probably wouldn't put the top 10 grossing movies of the week into an XML file either, as that changes quite often as well.  You could, however, put a list of provinces in XML as that normally doesn't change that often.

We have a carefully controlled production environment.  In order for any new files to be moved into Production we first deploy them to UAT, have them tested and then moved into Production.  You can see that by putting volatile data in an XML file you are creating a lot of extra work for you, the Deployment Team and even the users who need to test the changes as every change will create a lot of overhead.

What's the alternative?  A database.  Reference tables that have the potential to be changed on a frequent basis should never be put into XML files, they should be placed into a database with the proper screens in place to change the data.  Putting this sort of data does nothing for your application in terms of flexibility or ease of use.  It was a matter of using a cool new toy when it didn't need to be done.

New Topics

There are two questions that I am always asked with regard to this daily blurb:

• How do you come up with new topics every day?


• How can you write so much about such a small topic?

Getting new topics is easy:  I pay attention to what is happening around me.  Many of the topics discussed are reflections of what is happening in and around the Deployment Team on a daily basis.

As for the content, in many respects it is the same thing that makes everyone better:  practice.  The more you do something the better you get at it.  How do golfers get better?  They practice at the course they are going to play at.  How do writers get better?  They write about a topic, almost any topic.  How do presenters get better?  Join Toastmasters and practice presenting.  How do programmers get better?  Write programs.

While it may not be intuitive, writing programs for developers is a good way for them to get better.  Now, there is a caveat:  the programmer has to want to get better.  Just like all of the other people listed, if you want to improve and you practice at it, you will get better.  Now, this isn't to say that with practice everyone is going to be the Tiger Woods of .NET, but you will improve and that's the objective.  Write programs.  If you want to learn how something works, write a program to test things out. Try to figure out how to do it yourself before relying on books and web sites to tell you.  You are more likely to grow as a developer if you overcome the challenge yourself.

But above all, write.

Everything New is Old Again

You've just rewritten your application using the next version of the development tool and your web app is up to date with all sorts of spiffy new features.  Your team is tired after the prolonged effort to get things running, but it's done and you can relax.

Unfortunately, with today's technology, you've also signed up for a continual cycle of redevelopment and maintenance.  I can hear the mainframe guys saying "Oh, we never have to do that."  Yeah, right.  The longest weekend of my life (well, maybe the second longest) was spent compiling over 1900 COBOL programs, four times, once for each of our different environments (DEVL, SYST, UAT, PROD), all because the version of COBOL changed and made certain programming approaches invalid with the new version of COBOL.

Technology changes, of this there is no doubt.  Businesses that produce that technology will only supply support for a specific period of time.  They need to continually make money to fend off shareholders and supporting old technology is a money sink, not a money generator.   Within Microsoft there is something called Mainstream Support and this is the period when hot fixes are free.  Hot Fixes are available in Extended Support, but only if a support agreement is paid for in advance.  For some Microsoft technologies that we currently use these are some dates for the end of Mainstream support:

• .NET Framework 1.1 - October 14, 2008


• Visual Basic 6 - March 31, 2005


• Visual Studio 2003 - October 14, 2008


• SQL Server 2000 - April 8, 2008


• BizTalk 2002 - July 10, 2007


• BizTalk 2004 - July 14, 2009


• Windows 2000 - June 30, 2005

Take a look at the list and you'll see that for many of your applications, Mainstream Supports end within the next 18 months.  While this may not be an issue for some of you, you should consider upgrading the development tool as failure to do so may result in incompatibilities between your application and newer technologies.

Batch Processing

Batch processing is the mainstay of mainframe processing. This is where you get to line up one, two, dozens of jobs, let them take off and then charge the various departments, divisions, companies, based on the CPU time used for that job. People would schedule their jobs for weird hours of the day because you normally got charged a lower rate for processing at night or during "off peak" processing hours. After all, the mainframe was only a certain size and you had to ensure that your online customers during the day had the processing power available to them for what they needed done.

In the Wintel (Windows/Intel) era, however, batch processing is, quite frankly, antiquated. There are few tasks that actually need to be done in a batch manner as many of them can be done in an asynchronous manner throughout the day. Now, I am not including reporting in this as some reporting needs to be done on a consistent, daily basis and can easily be scheduled (if the report even needs to be run at all!). What I am referring to is setting up Job B to run only after Job A has successfully completed.

To be honest, in 90%+ of the cases there is no good business reason why Job B must run after Job A. What there is a specific business reason for is business process B to follow business process A, for a specific, small set of data. In one of my previous lives I worked in the insurance business. You didn't pay out on the insurance until you determined the persons eligibility. You did not need to determine the eligibility of everyone who applied that day before paying a specific person, you just needed to determine the eligibility of that one person.

This is a completely different mindset than what many "old timers" are used to as we are no longer concerned about trying to schedule things for "off hours". Instead, we are processing the data on an as-needed basis. What does this do? Well, in many cases it eliminates many batch jobs as the majority of the processing has already occurred during the day. It eliminates the need for fancy scheduling between jobs. It eliminates the requirement for a batch window.

This isn't an easy mindset to get rid of. Many customers still operate in this mode as it is easier to understand and is much more familiar territory. It is our job as IT Professionals, however, to not just do what our clients say, but to educate them as to what can be done, to show them new opportunities, and to give them something better than what they had, not just something newer.

Keeping it Simple

In my second year of NAIT one of our instructors decided that we should enter a programming competition put on by ACM.  With more than just a little curiosity we entered the competition.  One of the problems was to print out a multiplication grid, using Roman numerals.  We had a difficult time with this question, but when we finished we had a program that could print a grid of almost any size and convert the numbers into Roman numerals.  It worked, but our instructor was disappointed.  He asked us why we made it so complicated.  All that was asked was to print out the multiplication grid, so why hadn't we just printed the grid with 12 PRINT statements?

Talk about keeping it simple!!!

Blaire Pascal, a French mathematician and philosopher, once wrote "If I had more time, I would have written a shorter letter."  This completely goes against "traditional" productivity measurements in IT that state each programmer should generate x KLOC (thousand lines of code) per month:  if you generate more, you are more productive.  Generating more lines of code goes completely against keeping it simple.  The odds are you are cutting and pasting lots of code, doing things in ten lines when it can be done in one and generally doing your best to add fluff to the application.

The next time you are sitting down and writing a method in the language of your choice, check to see the size of the method.  A mentor a number of years ago had a simple rule:  if you can't see all of the lines of the method on the screen at once, rewrite it.  I have tried to follow this as closely as possible for much of my professional career and have been quite successful.  By, quite frankly, putting in arbitrary limits I have forced myself to think simple.  Sometimes a single method needs to be greater than the height of the screen, but this is an infrequent occurrence.

Keep it simple and you'll be surprised at how productive you really are!

Striving for "It Doesn't Suck"

While waiting for the bus I looked across the street and noticed that someone seemed to have written a long essay on the plywood surrounding the construction site.  While I couldn't read much of it, what I could read was very thought provoking:

... should we be striving for "it doesn't suck" as our goal?

Is "it doesn't suck" ever something we should strive for? 

As I sat down on the bus, thousands of images flashed through my mind, all of them related to that comment.  For instance, did you know that of the last 24 toy recalls in the United States, every single one has been related to a product that has been shipped from China?  Due to various pressures, both in the Chinese company manufacturing the toy and the American company importing the toy, the people involved probably thought that "it doesn't suck" was a pretty good choice.

It doesn't suck doesn't necessarily mean that something is good, just that it isn't bad.  There is still a wide gap between good and bad and it doesn't suck only covers a small area of that gap.  Now, I will be the first to admit that this problem is not specific to one role within the project.  Everyone, from the project manager to the developer and everyone in between, has to commit to striving for something better.  While one weak link in the chain can have a downward affect on the overall quality of an application, that does not mean that it cannot be remedied.

When I was younger I remember the news talking about the lack of quality of North American products and how products from overseas were better built and cost less money.  Ford started an ad campaign about "Quality is Job 1".  Much has happened since that time.  Overseas products no longer have the same high regard as they used to (at least some overseas products), but the rise in quality in North American products has not been that great.

The sign of a great company, and great staff, is one where no one strives for "it doesn't suck".  The purpose behind all of our work is continual improvement towards something better.  Something good.  Something great.  In my mind, what we should be striving for is "it's awesome".

Not just IT

Cross fertilization.  No, we're not talking about agriculture.  What we are talking about is the ability to look at other areas and bring some of those concepts and ideas into your own area.  Jim Collins talks about this in the PBS show Good to Great when he showed how public sector organizations (for example, a police force) can utilize the tools, experience and lessons from a company such as Starbucks to revitalize and improve themselves  Just because something isn't in your area of expertise, doesn't mean that you can't read it, watch it, enjoy it and, most importantly, learn from it.

For instance, the whole idea behind Design Patterns for IT applications was started by a 1977 book by Christopher Alexander that established a pattern language to discuss the construction of towns and buildings.  It was a book on architecture, not IT.  The idea behind this book is what prompted a number of individuals in the IT field to look at a pattern language for designing applications.

While it is important to understand what is happening in your own field and be proficient at it, there is an alternative need to broaden your perspective and look at other areas in an effort to kick start your own mental processes.  While watching the Bridge to Terabithia with my daughters I was struck by the appropriateness of the song "Keep Your Mind Wide Open".  While it matched the underlying thematic element in the movie it was also very appropriate for this discussion.

By limiting yourself to just IT books and websites and magazines there is a tremendous amount of knowledge that people miss out on.  By opening up your perspective, by keeping your mind wide open, you expose yourself to new possibilities and new ideas that might, just might, change your perspective on a problem and make something easier to solve, easier to build and better for everyone involved.

Changes and IT

I always find it really funny during playoffs (baseball, basketball, hockey, football, etc.) when people suddenly adopt these superstitions. You know what I'm talking about. The baseball player who refuses to shave because "it might cause bad luck". Or the hockey player who taps the pads of his goalie in a particular sequence because "it's good luck". Or the developer who always does things in a particular way because he's "always done it that way before and it's worked". People don't like change.

One of the most fascinating things about the IT industry, whether you work on mainframes, minis or PC class servers, is that the technology changes. It doesn't always change quickly but you don't always change the course of an ocean liner quickly either. (Although I bet the Lusitania wished that you could.) The process of making steel hasn't really changed in a long time. Neither has the process for making plate glass. IT? Yeah, it changes.

But you know, change cause problems. Changes require people to learn new things. Changes require people to adapt. Darwin recognized this a long time ago and wrote a book about it, some of which is actually applicable to the IT field. As new technology enters the field it is up to the developer to understand the technology and adapt his personal best practices to the new technology. Yes, you may have done this dozens of times in the past, but the technology has changed and maybe, just maybe, you shouldn't be doing this in the future.

Unlike Darwin's theory, failure to adapt will not cause your extinction, but it will limit what you can work on in the future. Embrace the technology and embrace the change. Learn to take your personal best practices and apply them to the new technology. They worked once and they can work again, but you need to make an effort.

Microwaves and IT

I was trying to follow the instructions on the back of the box the other day when I cam to a part that had me stumped.  It said that in order to defrost the item I held in my hands I needed to set the power to low and put it in the microwave for 2 1/2 minutes.  Now, I consider myself to be reasonably intelligent (some may not agree with that statement) and I consider myself to be somewhat of a technology geek.  I like silly, geeky things, but, darn it, I can't defrost anything in our microwave!!!!

Now, thinking just like the instructions, I set the power to low and then tried to enter a time, but for some reason I was entering in the time for "Step 2" of the overall process instead of Step 1.  So, I hit the "Defrost" button and was asked to enter in the weight of the object in kilograms.  Tossing it up and down like the frozen pork sandwich from Costco that it was, I guessed at about 0.1 KG.  I tossed it in the microwave and crossed my fingers.  I came close but it was probably about 0.15 KG, which I couldn't enter as it expected only a single digit after the decimal point.  (By the way, the correct method was to enter in the time, then enter in the power setting.  I had this epiphany on the bus on the way to work this morning.)

We sometimes build applications the same way that manufacturers build their interfaces.  it makes perfect sense to one or two people, or even the "study group" that looked at the interface, but when you give it to someone that has never seen it before, has never been part of the privileged few that saw the development process, the interface sucks.  Sorry for the bad news, but interfaces made by geeks are ... well, geeky.  They make sense to geeks, some of them,  but honestly don't make sense to anyone else. 

You see this in many applications and still see it today.  I was looking for an application to catalog by DVD collection using information sucked out of the Amazon.com database.  I found half a dozen applications, but they all looked like they were put together by Picasso.  I then found an application that looked like it was put together very well.  They had the complete package from the functionality that I wanted to a simple, pleasing user interface.

Project Managers, don't trust developers when it comes to user interfaces.  Look at the designs yourself and see if they make sense.  if you can't figure out how to do something, get it redone as someone totally unfamiliar with the project is going to be completely lost.  If the response is "the user wants it this way" then this really means that there are too many geeks trying to cook and it might be time to bring in a professional designer to help you out.

Innovation on Demand

I know this is going to bring up painful memories, but it will soon be over.  Imagine that you're at a meeting where you are discussing some project related issues and potential solutions.  Suddenly your manager says to you, "So, now that you've heard the problems, what's your solution?"  Your mind goes blank as you try to think of a solution to a problem that you've never really analyzed before.  You start breathing quickly and shallowly as your heart hammers away at 200 beats per minute and the adrenaline flowing through your system makes you acutely aware of everyone looking at you.  Looking at you with both sympathy and glee, thankful that they were not chosen. 

I am confident that 99% of you have been through this.  The 1% that hasn't been through this  just graduated and hasn't had the opportunity that the rest of us have had.

It is widely thought amongst management types that geeks can innovate on demand.  (Boy is that statement going to get me a lot of email.)  This is, however, fundamentally wrong.  Indeed, it is wrong for most people, not just geeks.  In order to proved a good answer to a question the person needs to understand the context in which the question is being made or the issues behind the problem.  By tossing out that one liner about coming up  with a solution with no opportunity to prepare is quite unfair.  The individual in question probably does not have a suitable answer (if they did the answer was so obvious that other people should have realized it) and the added pressure of having to answer this in a more formal meeting environment is going to make the person less likely to come up with a good solution and more likely to come up with a safe solution.

This is not to say that safe solutions cannot also be good solutions, but the odds are that the reason the manager is asking this question is because none of the safe solutions seem particularly attractive.  Managers are traditionally risk adverse which means that they would have looked at the safe solutions already.  Sometimes, however, a Eureka moment arrives and an answer comes to your mind, fully formed and breathing on its own.  The rush, the thrill of coming up with an answer is fantastic.  BUT, don't expect it.  Expecting it is almost guaranteed to make sure that it is not going to happen.

So, managers, if need to get an answer to a problem in a meeting, don't single out anyone as that added pressure is probably going to do the exact opposite of what you expect.

So, when is it going to be fixed?

"So, when is it going to be fixed?" Have you ever screamed when someone has said this?

About fifteen years ago the company I was working for had just replaced the biggest application that they used. After a number of months of daily fire fighting the system had begun to stabilize and the number of daily data fixes was down to handful. I came in to work, sat down at my desk and immediately had half a dozen people asking me to help them out. It appears that our application was no longer responding. Literally the eighty percent of the organization that used this application on a daily basis was unable to use the application. Off we went to investigate the problem.

About an hour later, when I was walking down the hallway my boss asked me "So, when is it going to be fixed?" I told her that we didn't know what the problem was so I couldn't tell her. She kept pressing me for a time commitment as to when it would be fixed and I kept telling her that when I knew what the problem was I would know how to fix it and I could then tell her when it would be fixed. She kept pressing me, however, until I finally snapped ... and just walked away.

There is an important lesson here to both management and the people trying to resolve issues. For management, if the people who know the most, don't know what the problem is, they cannot tell you when it will be fixed. It's as simple as that. Come up with an answer that conveys this to your clients because you're going to be using that same phrase a lot throughout your career. For people trying to resolve the issues, if you don't know what the problem is, don't give a time estimate because all you're doing is lying to them and to yourself. Tell them you don't know what the problem is. There is no shame in that. You don't walk into a Doctor's office and expect him to have an answer right away, so why should they expect you to have the answer?

Alone in a sea of web sites

Have you ever been to the mall when it's crowded?  You drive around the parking lot, looking for a spot, but the only thing close to the doors is a handicapped spot.  You look at the vehicle your driving, realize that it is not a handicap and decide to move on.  You eventually find a spot in the same postal code and start the long walk to the door.  On the way you see this little sports car drive into the handicap stall and out pops someone who, other than his driving style, has no handicap.  What he did, and you didn't do, was take the first available spot to park in.

This is, in many ways, how most projects build their installation applications.  (Yeah, I'm wondering how I'm going to connect the two myself, but hang in there for a couple of minutes.)

Many development projects have their own servers.  These servers are isolated in that no one else uses them and the project teams can configure and administer them any way they wish.  Once they are ready for UAT, however, they need to leave this isolated environment and enter the world of the Shared Environments.  On these servers there are multiple web applications and, drum roll please, multiple web sites.  Many setup applications are designed so that they install in the first web site they come across (much like parking in the first parking spot they see).  This is such a bad idea that I shudder when I write about it.

Applications need to understand that they co-exist with many other application.  Indeed, they are running on a server that has at least one web site and probably more.  During the installation process the web application should enumerate the web sites that are available and ask the person installing the application under which web site it should be installed.  Much like asking which directory to install an application into, the web site to install into is just as important, if not more important than the physical location of the files.  If the application just blindly installs itself in a web site the Deployment Team will quite often need to move the web application between web sites and, hopefully, everything moves over properly.  Nothing like crossing your fingers and hoping that something works.

So, please, in the future, change your web application installation programs to enumerate the web sites and ask which web site the application should be installed under. It's either that or I start leaving nasty notes on your windshield.

Again, What's the Problem?

When I look back at yesterday's comments about defining the problem, I can find many instances in my career where, if I had stopped and examined the problem in more detail, I would have made different decisions and different recommendations.

With regard to technology decisions, I've usually made the right recommendation at the right time based on the information I was given.  I was asked to solve a specific problem, within specific parameters, and I came up with a solution for that problem.  The problem lies in the fuzzier "direction" requests and anything else that involved other people making decisions.  You see, I am really bad at guessing at what other people think.  I just can't do it, mainly because everyone else thinks illogically.  (Either that or I am insane.)

When dealing with other people, however, it is more important than ever to sit back and really understand what the problem is, not just your solution to the problem. Everyone has a different perspective on the issue and everyone is going to have a different answer to the problem.  If we can agree on what the real problem is, however, we can more readily agree on an overall solution that fits in with everyone's perception of the problem.

Whether the problem is with regard to a complex business issue or a procedural flaw in a process, the underlying problem needs to be identified and agreed upon by the different parties.  You may discover that what you are trying to solve isn't really a problem, but a symptom of a completely different problem that is not going to be resolved by your solution.  Implementing your solution would do nothing to fix the problem, which in itself is a problem, just not the one that needed solving when you started.

What's the Problem?

In the IT world we are sometimes enamored with a solution and try our best to implement it, regardless of the actual problem.  This is the infamous "solution looking for a problem" syndrome.  People who are particularly attached to a technological solution sometimes fail to understand that their solution, while perhaps elegant and sophisticated, just won't be effective in the long run.

There are many examples of this throughout history, but in no industry does this show up more than in the IT industry.  We geeks love our toys.  We love our toys so much that we think they can solve every problem.  Even problems we don't have.  We create components in which 80% of the code is never executed because no one uses that functionality.  We create processes that are large, cumbersome, and built to handle every possible scenario even though something smaller and faster would handle at least 80% of our work.  We put in checks, balances and counter balances in an effort to ensure that something, somewhere is "done right".

Although self monitoring is something which is very difficult to do, I think we all need to step back and look at things from a different perspective to see if we are actually solving a problem that needs solving.  The next time you are proposing a solution to someone step back and ask yourself "What problem am I trying to solve?"  Is this actually the problem?  Are you tackling a symptom instead of a problem?  Once you are convinced that you have the problem defined correctly, write it down on a piece of paper and then come up with a reason why you are trying to solve that problem.  Is this reason something that the organization would agree with?  Does it provide a net benefit to the organization?  (i.e. Does the cost of implementing the solution outweigh the benefits?) 

Lastly. take a look at your solution and see it it is actually trying to solve the problem for the reason you stated.  For straight forward technological problems the odds are pretty good that the solution matches the problem.  Once you start moving towards the fuzzier areas of standards, processes and procedures I am sure that you will find that quite often the problem, the reasons and the solution don't always match up.  If you aren't sure whether or not they match up, ask someone else to check your reasoning.

Problem definition and problem resolution are basic things, but they are sometimes hidden in the tempting glare of gradients, 3D combo boxes, AJAX and web services.  Put on your sunglasses and see if you can really see the problem.  Chances are you might surprise yourself.

A World of Difference

There is a world of difference between maintenance and development projects.  NOT!!!!

OK, this may be a controversial topic, or it may not be, but I think I need to put some parameters around my comment.  In my mind, the actual act of developing a new application, doing a major enhancement to an existing application, doing a minor enhancement to an existing application or doing a bug fix for an existing application are fundamentally the same with regard to the tasks that need to be accomplished.  How the organization wishes to perceive them, from an accounting, management or accountability perspective may be different, but the same process need to be accomplished.

Whether the scale of the work is large (project) or small (bug fix), the developer still needs to have some requirements, do some coding, do some testing (really important step) and promote it to the next environment.  How the Project Manager (remember this isn't necessarily a title, but a role that someone plays) organizes these tasks, reports on these tasks and celebrates the successes with the team (you do this, right?) is up to the Project Manager.

Some organizations have an arbitrary limit of xx days of effort means a project.  Or perhaps there is an accounting limit of $yyy,yyy dollars is operating (maintenance) but anything over is capital and needs to be a project.  But you know what?  If it takes xx-1 days or xx+1 days, from a developers perspective there is no difference.  If it takes 19 days (maintenance) to add a feature to an existing application or 21 days (small project) to add that exact same feature, the developer is doing the same work.  The processes that the organization asks for may be different, the documentation requested may be different, or perhaps it is who pays the bill that is different, these are all things that surround the core act of development, but should not impact it.

If you feel that I am wrong, please let me know, although based on previous notes I don't think anyone reading this has any qualms about letting me know that they disagree with what I am saying.

Old Programs and Children

I used to believe, when I was younger, that the programs that I wrote would last forever.  They were well written (of course), full of useful functionality (of course) and they could adapt to anything (of course).  Wow, was I disillusioned in a hurry. 

It wasn't that they weren't well written.  I got a phone call from a company, six months after I left it, and it was one of the developers calling me up to thank me for having written the program and the exception handling in such a way that he was able to solve a problem in five minutes instead of the five hours he was expecting.  I was thrilled with that comment.  It was only after I hung up that I realized that it shouldn't have died in the first place, because I could have written it differently so that the information it was looking for could have been found in a more dynamic location.

When we are writing our little part of the overall project, or perhaps we are writing the entire application, we become quite protective of our child.  Anyone that criticizes our application is not worthy of further attention from us.  We close our eyes and forget the flaws of our child and emphasize the good points.  That's not to say that there aren't flaws, but we are so caught up in the entire development of this application that we are sometimes blind to it's faults.  This is not a bad thing!!!!  This bond that a developer forms with the application makes them want to do a better job and makes them want to ensure that everything is done right.

What we, as developers, need to do, however, is step back from being a parent and objectively listen to other peoples comments.  This is probably the hardest thing to do for a real parent and it is no less difficult for you and your virtual child, but it needs to be done.  No one's application is perfect and being able to understand that a critique of your application is not a critique of you as a person is a vital step in being able to be passionate about your work, but realistic about it flaws and weaknesses.  Your virtual children will be better for the effort... I guarantee it.

Constants to the Extreme

Those good old days of COBOL. Yes, it's that dusty old memory trail. Many things can be learned from those days, including what not to do. Not everything was sunshine and roses.

Sometimes people who really didn't understand the technology were allowed to make technical decisions. Worse still when that person was a project manager. The suggestion brought forth was that in many cases you did not need to use an explicit number for a loop, you could set the limit as a constant, so if it ever changed you would only have to change the value in working storage. The non-technical manager extended this to say that you could only use constants

For example:

    PERFORM VARYING ProvinceNum FROM LT-ONE BY LT-ONE
         UNTIL ProvincName(ProvinceNum) = Province OR
               ProvinceNum > LT-THIRTEEN
    END-PERFORM

You can see that this looks like someone got carried away. In this case there is no need to make '1' a constant as it will always be '1'. The key was actually in LT-THIRTEEN. This should have been called MAX-PROVINCES or something else. This would have made the statement look like this:

    PERFORM VARYING ProvinceNum FROM 1 by 1
         UNTIL ProvincName(ProvinceNum) = Province OR
               ProvinceNum > MAX-PROVINCES
    END-PERFORM

The same thing can be said for Visual Basic, C#, Java and many other languages. Not only does judicious use of constants make the code easier to understand, but it makes it easier to change. By making it LT-THIRTEEN, you couldn't very well change that to a different number, so you would have to change the code, totally defeating the person of making it a constant in the first place.

Frequently Asked Questions

No, I'm not going to answer some frequently asked questions. Instead I'm going to make a plea, once again, to the project teams to create an FAQ.

I did get some notes from people complaining that the software (SharePoint) was not a "real" wiki package and they weren't going to do anything that didn't involve HTML markup. A couple of other people complained that the process seemed too difficult and that there was no easy way to enter the data. The odd person mentioned that their application is unique as all of the questions asked were unique and that nothing was frequently asked.

Please, sit down and think about all of the questions you are asked for your project. Have any of them been asked more than once? if so, it is a candidate for inclusion on an FAQ for your project. Summer is coming up and you don't want to be one trying to answer questions that the person across the hall normally answers, and neither do they want to answer questions that you normally answer. Save a life, give some information.

Another war story ...

One thing I do have to admit, and not just for Peter Ford's sake, but for my own, is that growing up in a mainframe world was quite beneficial. The rigor and thoroughness that was required was very good at helping new developers out.

When I was younger I was involved in a project that was creating some stored procedures for a database. The stored procedures were written in COBOL as the database was mainframe oriented and this was the preferred method for writing stored procedures. (We didn't have any PL/1 expertise so that and Assembler were out of the question.)

Because of the mainframe orientation we followed all of the rules that we normally did for COBOL programs. We documented changes that went in, both in the program and in the source control system we were using. This allowed anyone, at any time, to take a look at the changes that had been made to a stored procedure (COBOL program) and see how it had grown.

The same discipline, however, does not seem to be there in the non-mainframe world. Call it "freedom from the tyranny of those mainframe ..." or call it a "more relaxed development atmosphere" or whatever you will, some of the lessons learned from decades of mainframe usage and discipline are still valid in the Wintel era. One of those valuable lessons is putting comments in your code, whether that code is VB, C#, or T-SQL. Comments help identify what has changed in the script. These comments not only help when debugging issues, but could lead to identifying other issues, such as maintainability, that need to be addressed sooner, rather than later. (If the same lines of code are constantly being changed, perhaps that section should be refactored.)

Why bring this up? Well, the vast majority of SQL scripts that I see migrated through DeCo do not contain comments and, quite frankly, this worries me. If we don't know what changes we've made, how are we ever going to know what to test? How are we going to know what may or may not be working?

Kids and Darwin

Recently I was watching The Colbert Report and one of his guests mentioned the fact that he didn't teach his kids all the rules that you need to follow, but he taught them the framework under which those rules exist. For instance, you teach the child that warm wet things (like tongues) have a tendency of sticking to very cold things like metal. You don't itemize the complete list of metal objects like: mail boxes, soccer posts, cars, lamp posts, bus stop posts, fire hydrants, handrails, flag poles, bike racks, chain link fences, etc. You teach them the framework and they get to interpret each circumstance on their own. Darwinism at it's finest.

Rule #1 --- Redone

I love it when I’m wrong, because I learn things. As David Woods pointed out yesterday, my example for the minimal effort you should put in to a Catch was flawed. My code contained the following:

    Catch ex as Exception
        Throw ex
    End Try

WRONG

The problem with my post is that all of the stack information is lost. If you follow the example that David Woods has on his blog you will see what I mean. The correct code should have been:

    Catch ex as Exception
        Throw
    End Try

Or, if you don’t like typing, this minimalist approach:

    Catch
        Throw
    End Try

In the end, though, the point that needs to be made is that you need to either handle the exception yourself (handle does not mean ignore) or you need to bubble it up to see if someone else wants to handle it. In addition, you need to put into the exception information that will help in debugging the problem.

The Stakeholder Registry has such extensive exception information embedded within it. When their code experiences something like a SQL Timeout they display the stored procedure that was being called and even the parameters that were being passed in. This is tremendously helpful in determining what the problem might be.

Rule #2

I mentioned that Rule #1 was "If you catch an exception, you better record it or re-throw it." Today I will talk about the next rule in the series.

Rule #2: If you throw or catch exceptions, you should use the finally clause to clean up after yourself.

When you throw an exception, or catch an exception, one of the biggest things that happens is that you start to move outside the boundaries of the flow mechanism that you had in place. OK, what this really means is that when you throw or catch you skip a lot of code. If you are using resources that are not simple .NET objects, you need to clean up after yourself. The last part of the Try ... Catch ... Finally block is the Finally clause and this helps you clean up.

The code that you put in the Finally clause is always executed at the end of the Try ... Catch block. So, if you re-throw the exception, throw a new exception or, regretfully, ignore the exception, the code in the Finally clause will be executed. This give you opportunity to clean up after yourself by disposing of those resources and objects that are not simple .NET objects. This includes things like file handles, SQL Server connections, or other resources not necessarily handled by managed .NET code.

This does a lot of things, not the least of which is ensure that your applications does not run out of those resources!!! So, like I always tell my girls when they leave the kitchen table, "clean up after yourself, because no one else is going to do it for you."

Try and Catch

Please note that there is an error with this entry. See Rule #1 --- Redone for details.

Exceptions are powerful tools that give the developer the ability to understand what is going on with their application, particularly when there is a problem. Sadly, many programmers do not use this feature, or implement it so poorly as to provide no meaningful information.

Rule #1: If you catch an exception, you better record it or re-throw it.

What does this mean? Take a look at the following code:

Catch ex As Exception 
End Try

What this code does is catch the exception, then let it get away. It’s like putting cheese on a mouse trap, but then gluing down the trigger so that the mouse can get away. I mean, seriously, what are you thinking? At the least, at the very least, you should have the follow:

Catch ex As Exception 
    Throw ex 
End Try

This at least re-throws the exception so that something above you can make a decision as to what needs to be done. If I had my druthers, however, it would be more like this:

Catch ex As Not Implemented 
    Throw New Exception("Blow Up3 does not implement that functionality", ex)
Catch es As Exception 
    Throw New Exception("Unforeseen error in Blow_Up3 trying to access Don’s bank account", ex) 
End Try 

There is so much more information available to the method that initially gets the exception than the calling method has that it would be a shame to ignore that information and make life more complicated for everyone.

DataSets

DataSets are an amazing construct. They allow you to pass back an entire result set, regardless of how many tables, to the caller and have them work with the data in the fashion that they want. Relationships can be established between tables, queries can be run against the dataset and updates can be made to the table.

You remember the saying "If something is too good to be true, it is"? Guess what? It's true with DataSets as well.

First of all, let me confess up front that I am not now, nor have I ever been, a very big fan of datasets, but probably not for the reasons you're thinking. I trust you will keep this in mind as you read.

DataSets provide a wealth of functionality to the developer, but it does come at a significant cost as well. DataSets hide much of the complexity associated with databases, particularly in the area of updating and populating fields on a web page or a report. While this hiding is, in some respects, quite welcome, it lulls the developer into a false sense of complacency and prevents the developer from truly understanding what is happening. I cannot tell you how many times over the past few years I have seem developers pass around hundreds of megabytes of data in a single dataset because they could. DataSets make it easier to be blissfully unaware of the consequences of doing something because of the fact that they hide things so well.

When I was younger I learned IBM 360 Assembler, both at the U of A and at NAIT. This low level language made me very conscious of the amount of data I was using and the most effective ways of manipulating it. Even earlier versions of Visual Basic (1.0 through 5.0) were fairly good at making you aware of what you were doing and the impact. As the complexity of the lower levels of programming has been covered up and hidden by successive updates to the languages and frameworks that support them, the ability of developers to understand the impact of what they are doing has been lowered. Today we have cases where hundreds of megabytes or even gigabytes of data are routinely moved from process to process because it is so simple to do so. The underlying impact, however, can bring a server to it's knees.

While I'm not advocating that everyone learns IBM 360 Assembler, I am advocating that developers fully understand the objects they are using and the impact of using those objects. If you aren't sure of the impact of what you are doing, experiment a little more, read a little more, learn a little more. The more you understand what you are doing with the languages you are using, the more productive you will be.

Failure

Being old, excuse me, older, than many of you gives me an advantage over you in a number of ways. I will be able to get the senior rate at the movies before you and I will be able to get discounts at hotels before you. What it has also done, is given me the opportunity to fail more often than you.

One of the best teachers in the world is failure, as it shows you what went wrong and what not to do. All you need to do now is learn from that failure and try to prevent that same situation from happening again. As someone who has been in this field for 20 years I have experienced a lot of failures, both on my part and those with whom I’ve worked. Each failure has been a learning experience that has allowed me to gain some piece of knowledge such that I am able to either not fail in the same manner or at least recover faster.

Unfortunately, failure is often seen as a bad thing, and from an overall project perspective it most certainly is a bad thing. However, small individual failures are not something that should be frowned upon, but embraced. Scott Berkun, in The Art of Project Management, wrote:

Courageous decision makers will tend to fail visibly more often than those who always make safe and cautious choices.

This applies to everyone that makes decisions, from the project manager down to the developer. If a decision was made that was, at the time, the right decision, celebrate the decision, regardless of whether or not it was a success. If the decision was bad, educate the decision maker so that they can learn from their mistake. (Educate does not mean punish.) By telling people you expect them to be perfect and that you do not expect any problems, you are telling them to play things safe and not try anything new. Mankind didn’t go to the Moon by playing safe. IBM played it safe with the personal computer and lost. Risks need to be taken at certain points and we need to train all of our staff, from developers to project managers, when failure and risk, is a good thing.

Writing Enough Code

Test Driven Development talks about writing enough code to pass the test. XP (Extreme Programming) talks about writing just enough code to meet the requirements. In both cases they make a case that you should not could for things that are not required, nor should you code for possibilities that may or may not occur.

This can easily get taken to extreme, however. I worked with a young man who took the idea of “just enough code” and went too far with it. He was writing an application that was designed to accept yearly reports. It was known from the start that the reports needed to be stored by year, searched by year, printed by year, etc. However, in the first release there was no historical data that needed to be kept, everything was in the current year. So, guess what? He omitted years from everything he did, database and code, because “it wasn’t necessary for this release of the application”.

Yes, write just enough code, but also use common sense. If you know for a fact that you are going to need to do something in a future release, don’t ignore that fact just because the current release doesn’t require it. If you know that you need to handle multiple years, design and code for that early on, even if the year is always going to be the same. Writing just enough code and common sense are not mutually exclusive ideas, at least, for most people.

Branching Guidance

Sometimes there just isn’t a shortcut to the right answer. You know what I mean: instead of researching the answer yourself, you lean over, talk to your buddy for 30 seconds and he gives you the answer you need. In many cases this works when you’re trying to solve a silly little problem or you just can’t remember the name of the runner up in last years American Idol.

Other problems, unfortunately, require that you understand the background behind the solution before you can actually understand the solution itself. String theory is like this. So are some aspects of quantum mechanics. Most business problems, don’t fall into this level of complexity, although I have seen the odd case where PhDs would be confounded by the sheer complexity of what has been engineered. Not necessarily what was required, just what was engineered.

In some cases there is some simple help, but it does require a bit of reading. I was recently asked for information about when to do branching and exactly how it should be done. In this case, I went to somebody who needs to do this on a frequent basis: Microsoft. Indeed, the information at CodePlex was excellent in terms of it’s understanding of the problem and the potential solutions. For those of you who think you understand how branching should be done, and for those of you who are at a loss, I recommend this document as an excellent source of information from which you can retrieve the bits and pieces that are of particular interest to you. It is not a light read as the amount of information it contains is quite voluminous (approximately 28 pages), but it gives you some interesting insight into an arcane subject.

Soccer Referee

Paraphrasing is a lost art in the IT world, but it is an art that really needs to be emphasized more. When I was younger I was helping a friend referee a soccer match. (Football to you foreigners) He wanted me to be a linesman and he told me that “When the red team kicks the ball out of bounds I want you to point the flag in the direction that the blue team will be moving when they get the ball on the throw in.” This made perfect nonsense to me as that seemed much too complicated, so I paraphrased it “You mean, point the flag at the team that kicked the ball out?” This confused him for a moment as he struggled to reconcile what I had repeated back to him with what he had told me, but he gradually agreed that the impact would be the same.

Sometimes when we write up specifications for an application we are too deep in the details and too aware of the intent, but not fully aware of the impact. We need to step back, take a look at what we have said or written, and see if we can rephrase it to make it simpler, yet still retain the same meaning. I do this quite often when writing these one minute comments. You should see some of the stuff that I write and throw out. (Then again, you have seen the stuff that I’ve gone ahead and sent out.) For instance, I’m currently writing this note because one on testing just doesn’t make any sense when viewed from outside the original context that most of the readers will not have.

The same thing is true of specifications. Not everyone reading the specification is going to have the same background as you or is operating with the same context. Not everyone is going to be an expert in the business area involved. (Or the subtleties of being a soccer referee.) What you write for a specification needs to be easy to understand, even for those that are unfamiliar with the business process. If it isn’t easy to understand then you need to step back, clear your mind, and try again. If it is hard for someone who knows the business to write the specifications, imagine how hard it is for someone not familiar with the application to understand what you have just written.

Best Practices

There are a ton of best practices floating out there in the infamous nether of cyberspace. Many of these interesting tidbits have actually come to roost in the minds of architects, programmers, testers, and, yes, even Project Managers. The question is, how do these best practices get communicated out to everyone?

Organizations sometimes do this through the creation of standards and templates for people to follow. These can be advantageous in that they prescribe certain actions that must occur. Standards, however, have some disadvantages in that the time from the creation of the standard to the implementation of the standard can be quite long. In other cases the standard provides either too much/not enough guidance and subsequently causes more confusion than if the standard had not existed. Enforcement is also a tricky thing to implement as grandfathering old projects needs to be taken into account versus the benefit of following the standard.

Some organizations produce lighter weight guidelines for people to follow. A guideline is a water downed standard in that it has not followed the same rigorous approval process, but is still considered something that should be followed, when possible. Guidelines, however, because they are not standards and subsequently not enforced, do not always provide the structure that is necessary to take full advantage of the material in question.

At the far end of the scale are those organizations that publish standards in a much more informal manner. The mere act of a certain group publishing something gives that tidbit of information the status of an organizationally approved standard that must be followed and will be enforced. This method presupposes that the group publishing the work is granted sufficient authority to make those decisions on behalf of the organization. Sometimes this authority is granted on a wide scale (all IT standards) or on a very narrow scale (all Visual Basic Programming Standards), depending upon the comfort level that management has with the group.

All of these methods, however, share one key thing: communication. Even on a project by project basis these communication mechanisms can be used to disseminate project standards to the rest of the team. Whether this is done through formal documents (standards), informal guidelines, or by having the Application Architect send out emails or write a blog, any method of communicating standards and guidelines is better than none, so get those best practices out of your head and on to paper (electronic or wood pulp) and let other people benefit from your experience.

High Performance Teams

In another life I was busy researching the idea behind "High Performance Teams" (HPT). These teams are not NASCAR fans, nor are they hooked on amphetamines. Instead, they are a group of individuals who work with each other really well and outperform other similar groups in terms of their quality of work and the speed with which the work gets done. You've seen these teams in hockey as the coach will normally put certain players together and keep them together throughout the season with few changes.

In IT, however, the concept of a high performance team does not always seem to be understood or even implemented in many areas. A team can be as small as two people, or it can be much larger, but there are some key traits that all of these teams share. (OK, here is where I differ from conventional wisdom so if you want you can tune out, even though you may be missing some really cool stuff.)

• Trust. Perhaps the most important trait is that the members of the team trust each other to make the right decisions or at least a decision that can be lived with by everyone.


• Communication. Team members communicate with each other effectively. Different people understand things in different ways. Some people like metaphors, others like analogies while others love diagrams. In a HPT the appropriate mechanism is used at the right time to maximize the effectiveness of the communication.


• Commitment. Each team member knows that every other member of the team is just as committed as they are to producing a high quality product.


• Continuous Improvement. An HPT is not satisfied with the status quo, they want to do the next job better than they did the last job and the one before that, by continually improving how things are done.

Some organizations are not ready for HPTs as it means setting a group up as being "special". Others are not interested as they believe, rightly or wrongly, that if people just follow the process everyone would be part of an HPT. Some larger projects do implement this concept within the overall project and find that the HPT is extremely productive and crucial to the success of the project.

It may not be your cup of tea, but at least you're aware of the possibilities.

Project Managers and Bullies: Are They One and the Same?

I hope my tease yesterday kept you on pins and needles. I have had the good fortune to be on a wide number of project teams, some of which had good project managers, some of which had bad project managers and some of which had me. Throughout all of these projects there is a common theme of trying to lead the team. Different PMs had different ideas about how to lead the team with some methods being more effective than others.

One popular method of leading the team is through the use of “It’s my way or the highway”. This is rarely effective as it stifles creativity, leads to aggressive behavious between the team members and the PM and usually causes people to devalue the importance of the PM. (Sometimes it leads to team members calling the PM an “arrogant SOB from Chicago”, but that is another story.)

Another popular method is consensus. Everyone needs to agree on something before it is done. While this may work for some areas, developing IT solutions requires a vision that needs to be followed. If that is the vision of one person or the organization as a whole, there needs to be someone driving the project forward and that is the PM.

An effective PM leads through something I like to call “motivation”. You need to be able to get the team engaged and believing that their part, no matter how small or obscure, is part of the overall picture and is important to the success of the project. And you know what, it is!!! Everything that someone is doing on the team is helping to craft the final product and is important, because if it’s not, why are you doing it? The PM needs to be cheerleader, traffic cop and salesman all rolled into one. Being a bully doesn’t cut it as that devastates the motivation piece and does nothing to engage the project team members.

Are Project Managers bullies? No, but that doesn’t mean they can’t be strong willed and opinionated, just that they know when to apply these features to the tasks at hand.

(Any similarities to people living or dead is purely coincidental, unless you are referring to that arrogant SOB from Chicago in which case …)

Project Managers: Who are they?

My apologies to those people who do the work of a Project Manager but don’t necessarily have the title of Project Manager as I may have unintentionally left you out. In some of my commentaries I refer to a Project Manager (notice the capital letters) but I never actually defined what a Project Manager does or explain that this may be more of an informal role that someone on the project team is playing as opposed to a formal title and list of responsibilities.

In my mind, at least today, as I’m writing this, a Project Manager is a person who guides the team closer to reaching the ultimate goal of creating a working, quality application for a client. This may be something which is being done on a piece meal basis, in which case the Project Manager may be the same person as the developer and designer – a team of one – or it may be something much more complex like a multi-year, multi-phase project where each phase has a Project Manager and the overall implementation has an uber Project Manager called a Programme Manager. In either case the target is the same, although the scope of action may be narrower.

So, why am I bringing this up? I’m reading a book called The Art of Project Management by Scott Berkun and it has a number of nuggets of information that I will probably be sharing with you over the course of the next few weeks. I wanted to make sure that we are all on the same page so that tomorrow, when I talk about “Project Managers and Bullies: Are They One and the Same?”, we all know the type of people I am talking about.

Refactoring -- refactored

It looks like "refactoring" is indeed a four letter word for some of you, while others seem to have no problem with it. Perhaps what I need to do is define a little more closely what I mean by refactoring or, to be more precise, what I didn't mean.

1. If you are not actively involved in making a change to a particular method or function, you do not refactor it. "If it ain't broke don't fix it", in this case, is true. When you open up a program your purpose is to implement specific functionality. Modifying code that is related to delivering that functionality is part of refactoring. If you see another method that needs to be refactored, put in some comments so that the next person knows what to do, but unless it is involved in what you are doing, don't refactor it.
   
2. If your refactoring is going to end up changing interface contracts with other code, you are no longer refactoring, you are redesigning. At this point put down the mouse and tell the Project Manager. If you need to change the interface contract to deliver some specific functionality (see item #1 above) then there may be more problems and it should be looked at in closer detail.
   
3. If you refactor code, you retest code. Now, since you are only refactoring code that you are changing to implement specific functionality (see item #1 above) that shouldn't be a problem. This code should be approached as if it is fresh, newly written code, and if it has been refactored it very well may be, so a complete test is going to be required. (Automated Unit testing anyone?)
   
4. If you aren't sure if it is going to make the code better, don't refactor it. If you aren't sure if it is going to make things more maintainable, don't refactor it. If you aren't sure what constitutes better code or more maintainable code, don't refactor it.
   
5. If your only purpose is to make it look prettier, don't refactor it.
   

Refactoring is meant to improve things, but it should also be limited to what you are working on, not unrelated areas of the application. These boundaries are put in place so that people don't spend all of their time refactoring code and none of it adding new functionality.

Really long rows

I'm a curious fellow. Sometimes much to my chagrin. But every so often I come across something which kicks the old brain cells into gear and gets me to writing. Something that I think the world (or in this case the people reading this note) need to hear.

I was looking over some stored procedures the other day and discovered a stored procedure that returned a table with a large number of columns. A disturbing large number of columns. I lost count at around 240 or 250. What was particularly disturbing was that a relational set of tables was being compressed into a single row.

By way of example, image if you will a database that stored your personal information and all of the credit cards you had. For each credit card it recorded the date of the last payment and the amount of the payment. Now image compressing that relational information into a single row. You would have a column for VisaPayment and VisaPaymentDate as well as AMEXPayment and AMEXPaymentDate. While in a static world this may not be a problem, what if you get another credit card. The layout of that row would need to change because you now need to add additional columns. Indeed, every time you added a different credit card or, hopefully, paid off a credit card you would need to change the layout. This is something that is headed for disaster.

Refactoring. There, I said it. The code needs to be refactored. It may be working right now, but there is a brick wall coming up fast and the brakes are failing, so it either needs to be fixed or we need to take out a lot of insurance. Fast.

(Please note that this was a fictional example, sort of. Look at the code you're writing and make sure that it doesn't have this brick wall built into it.)

Business Tier or Database

In our current production environment, indeed, most production environments around the world, there are multiple machines front ending access to the database. In our particular set of circumstances we have a number of web/application servers sitting in front of a database server. It seems rather obvious, but I will say it anyway, this means that the database server is the "tier" which is the hardest to scale. We can add web and application servers with relative ease, but it is downright difficult to spread a single database over multiple servers without some really funky maneuvering.

So, if this is the case, and I know that most of you understand this, why are people still putting excessive amounts of processing on the database tier?

Yes, you should minimize the amount of data that you are transmitting across the tiers, but this doesn't mean that you need to move your business tier to the database!!! There is a balancing act that developers need to do in order to properly distribute the workload in their application and part of that balancing act is the understanding of where to put different pieces of logic. If you are performing a number of different calculations on the server so that you can stop sending back a single column to the client, don't bother, as it may be more efficient to do it in your code than in T-SQL. This is something that you will need to test and figure out.

Or, talk to your DBA. They've been through this before and your question is probably something that they've answered dozens of times. If they haven't then I'm sure that they are going to enjoy figuring out the answer.

A Common Phone Call

Occasionally we receive calls at the Deployment team that follow this sort of pattern:

"My application isn't working any more. It was fine yesterday, but today it's broken."

"Well, that seems a little strange as there were no changes to the environment last night."

"Oh, I forgot to mention that it's broken in UAT as well."

"When did this happen?"

"Well, this particular functionality has always been broken in UAT. We just thought that it would work when we moved to Production."

I know that some of you are laughing out there, but I also know that some of you are saying "Is he talking about me? Is he talking about ME?!?" If something doesn't work in our UAT environment, there is no guarantee that it is going to work in Production. Indeed, my bet would be that it doesn't work in Production either.

So why doesn't it work in UAT when it worked in Development? Well, unfortunately, I only have about 500 words to write an answer and in this case the answer is more like a 500 page novel. One of the biggest reasons is something we've talked about before: running something as an admin in development but with minimal rights in UAT. Just because you can run your application as an administrator on the box does not mean that you should.

I can't stress this often enough, and based on current evidence I definitely haven't, you need to run your application under the least amount of rights possible. If something doesn't work, fix it. Don't assume that there is something wrong with the environment and that it is magically going to get better when it is migrated to the next environment, as it may even get worse.

New Cobol

There is a phenomena that has risen in the last few years which, unfortunately, has had a detrimental effect on the IT industry. For the sake of simplicity, I shall call it the rise of New COBOL.

Now, some of you may not know what COBOL is, so a short history lesson is required. COBOL (COmmon Business-Oriented Language) was developed in 1959, primarily for use by the U.S. government. It quickly spread and became the de facto standard upon which billions of lines of code were written. It was also, almost single handedly, responsible for the boom in the IT business prior to Y2K because of the number of lines of code written in COBOL. While it has been criticized as verbose, it is capable of handling most business problems currently encountered and is a perfectly valid language for developing applications.

New COBOL, however, is something which people should not want. New COBOL is actually the use of an object oriented language in a procedural way. Most newer languages are object oriented in that business logic and data are encapsulated within an object and that interaction with that object is done through methods. When the developer discards the object oriented nature of the development language and essentially writes procedure COBOL, but in a new language, we have New COBOL.

This perversion of the intent of an OO language causes many problems, not the least of which is an increase in maintenance costs due to the barrier imposed in understanding the purpose of the underlying code. If the application was written in an OO style than a maintenance individual could understand it and make changes relatively quickly. If, however, the application was written in a procedural manner, but with an OO language, methods and data do not appear where they "should be", resulting in a higher learning curve for the developer.

If you've chosen to develop in an OO language, make use of the facilities of that language. Failure to do so is a disservice to your client, as well as yourself.

I look forward to your comments on this one, as I know for a fact that certain architects disagree with my stance.

Planning for Change

People have come up to me recently and expressed a concern that I am losing control of my mental faculties. To emphasize this point they point to the fact that I keep talking about "planning" and yet I am also an advocate of Agile Development. The usual comment I get is "In Agile Development, you don't plan, you just do."

Ah, the misguided conceptions of youth. Contrary to what people believe, Agile Development does believe in planning, but they are not slaves to the planning process. I once worked for a Programme Manager from Chicago who was quite ... stubborn ... with regard to his belief that if it isn't on the project plan you don't do it. Needless to say we butted heads a number of times, with him always coming out on the winning side because he was the Programme Manager.

This caused a few problems with the client and the application, however, as we were never responding quickly enough to requests. Each time something new came up we had to revise the project plan to take this into account and then confer with the client about the correct project variance that this would cause and sign off on the change request. This process took a number of weeks to complete. We were as agile as a beached whale.

After a while, we figured out how to handle the Programme Manager: insert line items into the plan, that planned for change. Indeed, that is what any agile project should do. Yes, you need to have an overall plan as to what you are going to do and when you are going to do it, but you also need to plan for change. You need to understand that change is natural in a project and you should develop your plan, and work out, in advance, how to deal with change with the client.

Change management is just as important to the development of an application as it is to the ongoing maintenance of an application.

Cutting and Pasting -- Part 2

Last week I went to one extreme and said that cutting and pasting was not necessarily a good thing. I was talking about using previously written applications as templates for your new application. Let's swing the pendulum to the other end of the extreme and talk about why you should never write certain code more than once and if you do you should slap yourself on the wrist.

Validation routines are things that every application needs. If you accept data from any source you need to verify if that data is correct. Sometimes the verification is quite simple (A9A 9A9 for postal codes) whereas in other cases verification requires database calls or lengthy computational processes involving logarithms and abstract mathematics. If your application is larger than a single screen or a single process, the odds are that there are going to be some common verification methods that can be shared between these areas.

In a previous life, my specialty was crafting routines for verifying a persons Social Insurance Number in any of a dozen different programming languages. I only wrote in once in each language, however, as I (and other people) used the same code in many different places. It varied by language and platform, but each time I used it, I essentially made a call to a utility library that did the validation for me and sent back the results.

I wrote it once and used it thousands of times.

I did not copy and paste it, but I did reuse it and I guess that is where the difference is. Copy and pasting the lines of code that did the function would reuse those lines of code, but it proliferate the number of areas where that code could potentially need to change. If you find yourself writing the same lines of code more than once, you may want to consider putting them in a utility library that you can call from anyplace. It may take some extra time to initially set things up, it saves a lot of time debugging and in maintenance.

Cutting and Pasting -- Part 1

When I first started developing applications, back in the days of COBOL and green screen monitors, one of the things that we did quite often was use other applications as templates. COBOL, in case you didn't know, is one of those older languages responsible for the Y2K rush in the late 20th century. By templates, I mean that we cut and pasted chunks of other applications into our own so that we wouldn't have to type it all over again. Does this seem familiar?

Cutting and pasting is as old as developing applications itself. Even farther back I remember reusing punch cards from other applications so I wouldn't have to wait in line at the card punch machine for my turn. ( I gambled by not putting sequence numbers on my cards, allowing them to be reused at a later date. Boy, I lived on the edge in those days.)

There is a problem with this sort of programming, however, as it has a tendency of hiding many of the complexities that programmers need to understand in order to create better applications.

For instance, what would happen if you copied some code from another application, but the code was poorly designed or performed poorly. You would be propagating that "bad code". If people copied your application they would be doing future programmers the same disservice by arbitrarily inserting code, which they do not understand and which is fundamentally not the right code to copy, into their applications for future generations to debug.

Now, I'm not saying that you need to write every line of code over and over again, but I am saying you need to understand what you are cutting and pasting. Just because it works, doesn't mean you should necessarily copy it. You need to understand what you are doing. Blindly following someone else's lead without truly understanding is one of the reasons why Sanjaya Malakar got as far as he did.

Oh, That's Easy -- Part 2

"Oh, that's easy."

As a project manager I hated those words, particularly if they were coming from my technical team. The more technically adept a person is the more inaccurate the person is at coming up with estimates. Yes, there are exceptions, but they are few and far between and should not be counted on at all times.

Someone who is familiar with the technology and lives and breathes the code is someone who is going to be horrible at coming up with estimates for other people. Heck, they are even horrible at coming up with estimates for themselves. I have had the privilege of working with some very talented people over the past 25 years (damn, there's that age thing again) and with the rare exceptional circumstance, every single one of those people has problems with estimates.

So, how do you compensate? First of all you need to ensure that the developer has thought of everything. Most of the time an estimate only includes the raw time to develop the code, but not to test it, document it and ensure that any technological innovations are compatible with the existing environments. You need to keep track of this persons estimates and use previous estimates in guiding your decision about current estimates.

No, don't create the estimate for the developer as you probably don't know the technology. (See yesterday's note.) Understand, however, that the developer is going to be optimistic about his part of the solution so you need to take his estimate, inflate it, and come up with something that is realistic for the project. And, the most important point of all, keep track of previous estimates and how they matched reality. As mutual funds state "Past performance is not an indication of future results", but they're probably darn close.

Oh, That's Easy -- Part 1

"Oh, that's easy."

As a developer, I hate those words, particularly if they are coming from the Business Analyst, Team Lead, or, worse of all, the Project Manager. To be brutally honest, and when have I not been, most Project Managers are not familiar enough with the technology being used on a project to actually make that decision. Most, not all, but most project managers are more in tune with the business aspect of a project, but not the technical parts of the project.

A number of years ago on a large project I was brought into a meeting with the client and asked whether something or not could be done. My immediate response was "I'm not sure, but I think we can." This was immediately translated into "See, it's easy". Afterwards the project manager came up to me and asked me if I could have it done by the next meeting, in two weeks. When I told him I could have the estimate ready in two weeks he came back with "Oh no, you misunderstood, you're going to have the solution in place in two weeks."

Needless to say, no one was happy. The solution wasn't in place in two weeks, so the users were disappointed. The solution wasn't in place in two weeks so the Project Manager was angry. I didn't get the proper support from the Project Manager so I was quite disillusioned. (OK, I wasn't that disillusioned because I didn't expect anything else, but I didn't fill out the patent application either.)

The lesson I learned, and that I hope I have communicated, is that Project Managers should never give an estimate of the technological complexity of a request without first asking their team. It's a wonderful recipe for disaster if you do as no one is going to be happy, least of all the client.