Myths of Innovation

We are all told to be "innovative", that we need an "innovative" solution to a problem.  We are then locked in a room and told that we have fifteen minutes to come up with this innovative solution because in thirty minutes the project is starting up and by the way here are the people working on the project and their skills so the innovative approach has to be able to be done by this group of individuals.

By the third Tuesday of next month.

OK, time to turn the subtlety off and talk about reality.  People don't think of innovative ideas on a scheduled basis.  Particularly not in a meeting room with other people.  Sure, you can brainstorm in a room with other people, but don't necessarily expect innovative ideas to come out of that session.

Too often people are told to come up with innovative solutions to problems to which they have just been introduced. Innovation requires work. Innovation requires a deep understanding of the problem. I am not saying that you will never get an innovative idea with this approach, but the odds of it happening are rather slim.  Getting a group of people together and telling them "We're going to create an innovative solution to problem X in the next 60 minutes or we're all fired", is probably not the way to go about getting a truly innovative solution.  Sure, you may come up with something that satisfies your immediate needs, but is it innovative or just copying something else?

Scott Berkun, author of Myths of Innovation has a summary of the 177 myths of innovation that he has uncovered in his own work and work done by others.  What I notice when reading the myths is that there is no single solution.  Innovation is a bolt of lightning out of the blue.  What works for one industry may not work for another.  What works for one problem may not work for the next even if the identical group of people is trying to solve both.

We also need to step back and realize that an innovative solution may not actually be leading edge technology, it may just be using what you already have in a different way.  The Apple iPod was not the most innovative MP3 player on the market.  It was innovative in how the iPod ecosystem supported all of the pieces and made the experience pleasurable.  It was not the first, but for its time it was the best.

There are so many variables that we best we can do is give people the time, space and opportunity to think.  After all, isn't thinking the start of any innovative idea?

High Performance Teams

What is a "High Performance Team" (HPT)?  At what point does a team become one?  Can a team change?  Do all the members of the team need to be "superstars"?

According to Wikipedia, high performance teams are groups that are "highly focused on their goals and that achieve superior business results.  High-performance teams outperform all other similar teams and they outperform expectations given their composition".  That last part is probably the most important and it is the part that is hardest for organizations to understand.

In a traditional organization staff are known as "resources", hence the term Human Resources for the department that deals with staff.  Within many organizations staff (aka resources) are interchangeable.  If you have an SA3 on your project and they won the lottery you get another SA3 to replace them.  Most project managers think this way as well.  In many respects human resources are thought of as interchangeable Lego pieces.  If, according to the plan, you need a 4x2 full height brick, you look for a 4x2 full height brick.  This makes a single assumption that is at the root of the problem:  people are interchangeable like Lego bricks.

People aren't interchangeable, however.  Lego bricks are, on the other hand, because the manufacturing process has extremely tight tolerances for deviations from the standard. (Up to a 0.02 millimeter variation is allowed)  People?  Well, as they say, no two people are alike.  That individualization is what makes substituting one person for another not as simple as what you might expect.

An HPT needs to fit together like Lego bricks.  If one member of the team leaves, their replacement may not fit properly.  Conversely, if you don't have an HPT and you replace someone on the team you may end up with one in the long run.  Is there a science involved in creating an HPT?  People have tried for years and have been completely unsuccessful.  What they have been successful at doing, however, is identifying when a team has become an HPT.  Once you have such a team you need to do what you can to keep that team together.  Their success as a team can have an impact on the success of the organization. 

This doesn't mean that you let them run wild.  Every team needs guidance, but instead of putting a chain around their neck and leading them where you want them to go, you point in the direction of where you want them to go and work with them so that they not only understand where you want to go and why you want to go there, but the benefits of going there.  An HPT requires more openness than other teams because that is part of why they are as effective as they are:  open and honest communication.

One of the biggest fallacies, however, is the idea that if you break up the team then each member can start their own HPT and you will have even more teams.  It doesn't work that way.  Some of the reasons for the team to be so good at what they do includes the fact that they trust each other, they respect each other and they are all aligned with regard to their work goals.  Putting five people together in a room does not make people trust each other.  It does not make people respect each other.  It does not align their work goals. 

An HPT is an organic process that cannot be predicted nor can it be forced.  It can, however, be nurtured and sustained.  If you want to be successful, just step out of the way and grab on to the coat tails of an HPT because the ride is going to be fast and frightening, but you're going to go places.

Fitting in

Have you ever written a complaint letter to an organization?  I have.  Yes, I know that may surprise you, but I have done so on a number of occasions.  I have complained about poor service (and I mean really poor service not just bad).  I have complained about getting a "deal" from the salesperson when that "deal" ended up costing me almost 20% more than the advertised price on the flyer right in front of his face.  I have complained about paying for something but never receiving it.

On the other side of the coin, however, have you ever sent a letter to an organization complimenting them on their service of product?  Not a lot of people do that.  It may surprise you to know that I have done that a number of times in the past as well.  I have complimented an organization on how the sales person / service desk handled a complaint.  I have complimented an organization on the excellent service I have received from their staff.  I have complimented an organization on how they went out of their way to make me feel important even though, to me at least, the problem was not even worth commenting about.

Most people in this world do not complain, nor do they compliment. Why do I?  Because I want to help make a difference.  I want the organization that I dealt with to be healthier, stronger and more in tune with the needs of customers after I have interacted with them.  For complaints I want them to improve their service so that everyone they deal with feels important.  For compliments I want them to recognize that individual and help foster that same attitude amongst all of the staff.

Too often people feel disinclined to speak up.  They feel that they can have little to no impact on how (insert organization here) does its business.  As a result they behave just like ants, following the scent trail of someone who has gone before them.  They don't deviate from the path unless something prevents them from following that path.

Most people in the world are like sheep.  They go where they are told because it is easier to do that then to carve a new path.  (You can see this in Winter when someone starts a fresh path through the snow.  No matter how crooked the path, people follow it because it is easier to follow than it is to lead.)  This doesn't mean that they can't excel, just that they aren't being asked to excel.  Sure, you can tell people that you want them to think out of the box, but if you subsequently put them in a box what message are you giving them?  As a culture we foster the individuals who "conform" and "fit in" while idolizing those that "think out of the box".

Thinking independently and critically is something that we need to develop, but it requires an organizational shift that most organizations, particularly Public Sector, are not willing to experience.  "Fitting in" is still more highly praised than "standing out" as the phrase "I might get fired" brings with it a lot of fear and makes people walk that crooked line in the snow.

Changing defaults

When you lick the terminals of a brand new 9-volt battery the shock you feel is Nature's way of saying "You really shouldn't be doing this."

When you approach a bonfire and feel the heat coming off in waves and singing your eyebrows it is Nature's way of saying "You really shouldn't be doing this."

When you are timing out when running a report in SQL Server Reporting Services it is Nature's way of saying "You really shouldn't be doing this."

We have received a troubling number of requests recently to expand the timeout of SQL Server or the maximum length of a transaction or enlarge the number of connections in the connection pool.  What people are doing is trying to put "fix" their problem by pretending the problem doesn't exist.  Let's blast through a number of items really quickly.

Increasing the connections in the connection pool.  The error message that people receive is "The timeout period elapsed prior to obtaining a connection from the pool."  This is almost always an application issue as the application has most likely forgotten to clean up SQL Connection objects.  Yes, .NET does automatic garbage collection.  No, it does not clean up everything.  Resources that are managed outside of .NET (for example SQL connections) need to be cleaned up by your code as .NET will not do it for you.  As a result, if you ever receive the error message listed above, look in your code first before looking anywhere else.  In our environment a well running application will only have 5 - 10 connections in use at any one time and, by default, the connection pool has 100 connections available to be used.  We've had applications that have had hundreds of concurrent users with no issues. (In an extreme case we had over 1800 concurrent users and we still stayed within the 100 connection limit of the connection pool.)  If you get the error message there is a problem.  FIX the problem.

Increasing the transaction limit above the default.  The default transaction limit is 10 minutes.  If you are trying to exceed that it means that you have a transaction which is trying to run for more than 10 minutes without a single commit or rollback in that time.  This is way too long.  You should not be running 10 minute transactions.  In reality, if your transaction is longer than 10 seconds you have an issue.  Yes, if you are importing a lot of data you can go for longer than 10 seconds, but the longer you run in a single transaction the slower your application will respond.  Keep the transaction short and meaningful.  If you need to raise the limit above 10 minutes, rethink your architecture as there are some fundamental questions that you have not answered correctly.

Increasing the session state timeout.  This one is alterable by the application team but, in all honesty, should never be altered.  If altered it should be moved down not up.  The Microsoft default of 20 minutes, in the words of many companies, is "too long".  If you are maintaining session state and the user is not doing anything to interact with your system for 20 minutes, then you need to re-evaluate your system.  Twenty minutes is a long time.  Light has circled the earth over 10,000 times. In 2011 an average of over 65 million searches were done on Google properties worldwide in a a 20 minute period.  If your user is doing nothing on your page for 20 minutes, consider them lost.  If, for some obscure reason your clients are on a single page for 20 minutes, you need to redevelop that page as it is not in the least bit effective.

In general, if you feel that you need to alter some default setting then you should be experiencing the same feeling as licking that 9-volt battery.  Stop it and step back.

There is a time for Collaboration … but this isn't it.

One of the buzz words in many peoples vocabulary is "collaboration".  We must be seen to be collaborative.  We must act in a collaborative fashion.  We must achieve our goal through collaboration.

Sometimes, not always, but sometimes collaboration is used as an excuse not to make a decision.  There are times when collaboration is a good thing and there are times when a dictator is required.  Don't misunderstand me, a dictator can be more than a Project Manager, it can be a process or policy that is created.  They dictate your actions towards achieving your goal, the goal that was created in a collaborative fashion.  However, a project is not a collaborative group of individuals striving to achieve a common goal.  It's a project team. 

If you are trying to solve something than collaborating with someone to come up with a solution is a good thing.  However, the implementation of the solution is not necessarily a "collaborative" adventure.  Collaboration comes in the act of creation, in the act of coming up with a solution for a problem.  Do project team members collaborate?  Yes, amongst each other.

Let's go with an example.  Being in the IT field there are a lot of things where I need to collaborate with other people.  Let's say that I want to implement a new standard for SQL Server.  I will collaborate with the DBAs in my group, affected project teams, subject matter experts, etc., and come up with the standard.  The implementation of that standard is not something which is a collaborative process.  The collaboration occurred during the creation, not the implementation of the SQL Server standards.

Collaborate when you don't know the solution.  Manage when you know the solution and you need to get there.

Microsoft Surface–A home run or a foul ball?

Neither.  First of all, it should be noted throughout the Microsoft web site that “Images are design renderings and not photographs.”  It is important to note that what you see isn’t necessarily what you get.

Success points.
Finally there is a tablet that is designed for Windows as opposed to a tablet designed for almost anything that just happens to be hosting a Windows operating system.  Like the Apple ecosystem that has worked so well, Microsoft realizes that getting the software and the hardware working well together is one of the features that makes the iOS line a success.  Not because it is the best, but that it works well together.
The design is both good and bad.  I think that overall it is a very clean device that looks good with only a single major visual flaw:  the windows logo on the front.  It really needs to go.  There is a big logo on the back, leave that one and clean up the front.  Other than that there don’t appear to be, initially, any significant design issues. The magnesium cover I am expecting to be both strong and lightweight.

The integrated kick stand looks like a solid piece of engineering that is required for most tablets.
The plethora of connectivity options looks like it will connect up to almost any environment and allow you to hook up most devices.  My biggest complaint about the iPad is the limited connectivity options.  While this has been sort of resolved by third parties, the fact that the options are built into the Surface is a very good thing.

Things to work on.

I can see the lawyers getting ready for this.  Microsoft needs to ensure that the price it offers for the ARM version of Surface is able to be matched by competitors like ASUS, so it is going to need to be very careful about licensing Windows 8 RT and Office at a price which allows other companies to compete.  If not, you can expect to see anti-trust lawsuits right away.  Why does Apple get away with it?  They don’t pretend to offer iOS to anyone else.  You want iOS you need to be Apple hardware.  Microsoft relies on others to manufacture hardware (yes, even with Surface) and if it can be shown that they are acting in a manner which prohibits competition then the lawyers are going to be jumping all over it.

Microsoft still likes taking choices away from people.  While the think the integrated kickstand is an interesting idea, what about portrait?  Why must it be in landscape mode and why must it be in a very specific landscape mode?  Did you notice the Windows logo on the front of the device?  That won’t look good upside down so they force the table to be only one way up for landscape.  Portrait mode?  Why bother?  There were some obvious reasons why (Touch Cover and Type Cover), but taking away choice is what makes people annoyed.  At the moment this looks like nothing more than a very thin netbook, not a tablet.

22 degree angle to the onboard camera?  Really?  I have to twist the damn thing in my hands if I want to take a picture?  This is not a user friendly option.  The tablet is meant to be used on the go and to me that means free hand or sitting with friends at a restaurant.  I see something and I pick up my tablet to snap a picture and I twist my hands to get the damn thing lined up.  Seriously?  This looks like a business item, not a consumer item.

The two different Surface models, with two completely different architectures, is going to confuse people.  “Why can’t I run the same apps as my friend?  Why can I download something from the App Store that they can’t?”  While Microsoft had to do it, calling them both Surface is going to confuse the issue.  Simplicity is the rule.

Unknowns

Tactile feedback is quite important when typing and that is one of the reasons why typing on a piece of glass can be so annoying as you can’t feel when you have pressed a key.  I don’t know how the two different covers are going to feel.  My gut feeling is that the Touch Cover is going to be too thin to provide the tactile response that I am looking for while typing.  I could be wrong but I am willing to give them the benefit of the doubt.

I am a little confused over the marketing.  It appears to be targeted at consumers (see all the pictures on the Microsoft web site), but many of the design decisions (no rotation capability, 22 degree LifeCam, keyboard that requires special connectivity) all seem to be focused on business.  Apple focused on the consumer and entertainment and entered business almost by accident.  Microsoft looks like it wants to tackle both, but isn’t sure which one is important.

iPad Killer?

Don’t know.  I think Microsoft will sell millions of tablets.  There will be a pent up demand and they will sell millions as soon as the doors open.  Will the demand be there long term?  I don’t know.  They need to straighten out their marketing.  They need to make it seamless and inexpensive to get “apps”.  They need to cannibalize their Windows desktop to sell their Windows tablet.  If you have to pay full price to run Office on your tablet and an additional license for Office on your desktop then we’re going to have a problem.
Microsoft is trying to straddle two markets:  tech savvy consumer and the enterprise.  Without really good marketing they are only going to take one, not both.  The marketing so far has been light weight and … fluffy.  Only time will tell whether or not they can achieve the right balance.

A bit of a Trend

I’m seeing a bit of a trend thee days in the Blogosphere that seems to be rising quite rapidly this year.  I’m not sure if it started with the Learn to Code challenge from Codeacademy or whether it has been building for a while and I’ve only just noticed it now, but there are a lot of bloggers out there complaining about, well let’s call it The State of Programming.

They all seem to start with the premise that there is a lot of common sense that is not being used and end up with dispensing a lot of hints and tips about how to do things correctly.  These hints and tips range from:

• Keeping the code size to its smallest possible state.  Constant refactoring should allow you to reduce the complexity of the code and increase the flexibility of that same code base.
• Don’t implement things that you don’t need.  Sure embedding Twitter or Facebook into your application might seem “neat” or “cool”, but unless it provides a distinct advantage to your user, don’t do it.
• Just because an application is complex does not mean that it is good/well designed/well written/the answer to your prayers.  The best code is the simplest code.  Does it do 100% of what you think you need?  Maybe not, but I bet it does 80% of what you need at 20% of the price.
• Water-Scrum-Fall is most likely what you are doing, not pure Agile.  Don’t look at me, read the report and see if you agree.
• The tool you are using is not the reason why your project is large/cumbersome/unwieldy/poorly tested.  “It’s a poor carpenter who blames his tools”.

Now, it may be that I am sleep blogging under a whole host of names and nationalities, but I don’t think so.  I’m seeing it from older people (Nick Bradbury turned 45 in May),  people who have been in the industry for a long period of time and even on t-shirts.  Much like everything else these days it is polarizing a group of people (programmers/developers) and causing inter-personal issues and intra-team issues within organizations.

The question behind all of this is quite simple:  is the standard for programmers/developers rising or falling?  What do you think?

Getting Help

First of all, let me state that this is not meant as an insult. 

Secondly, most people in IT suck at designing databases.

Third, it’s not their fault.

There is an art to designing a database correctly, from determining the correct number of filegroups, the indexes that need to be created, the clustered keys, the primary keys, when to use GUIDs vs. Identity columns and when to use stored procedures vs. dynamic SQL.  There is a lot to learn to become really good at it and when you come across someone who is good at it you know. 

I worked with this guy who would stare at the diagram of the tables for a long time, not moving, just tapping his fingers.  Suddenly he would start typing madly and in ten minutes he would have solved a couple of the hardest database issues you could throw at him.  He was never sure how he did it, but he said that he visualized the tables in his head and all of the primary and foreign keys and moved things around until they looked “right”.  He was very good at squeezing every last bit of performance he could out of the system.

Not everyone can do that.  Not everyone should be able to do that.

What you need to be good at is recognizing when you need help, recognizing that there is an issue in front of you that goes beyond your expertise and asking for help.  Too often our egos stand in the way of us actually getting the help we require because we believe that others expect us to somehow know everything about everything.  It doesn’t work.  Know enough to know when to get help.  That is the key to success.

Is "Learn To Code" necessary?

There have been a lot of posts in the blogging world recently about whether or not you should learn to code.  A company called Code Academy issued a challenge about learning how to code.  This was quickly followed up with the Mayor of New York, Michael Bloomberg, tweeting about how he was going to learn to code.  The site Coding Horror was so appalled by the idea that everyone needed to learn how to code that they wrote a very strident blog entry on why people shouldn’t learn to code.
Never one to back down from a challenge, I thought I would add my two cents into the argument.
Should people learn to code?  If they are learning to code in order to call themselves programmers, then the answer is “no”, people don’t need to learn how to code.  If they are learning to code in order to understand how computers work, then the answer is “no” because learning to code does not give you any insight into how computers work.  If, on the other hand, they are learning to code in order to learn something new and to expand their horizons, then “yes”, learn to code.
I think the Coding Horror site went too far in their analysis of what was happening, but to be honest, I blame that over reaching analysis to be fostered by the Code Academy web site.  By going through the list of courses it really appears as if the Code Academy web site is trying to teach people to become Javascript experts.  That is not a good thing to do.  Back in the early days of the web (many of you are too young to remember this) one of the features of HTML was the <blink> tag.  Wow, did that hurt.  Blinking text did nothing towards making the Web what it is today.  Indeed, it may have set it back a few years.  Imagine a new generation of coders thinking of blink.
What I think most people actually need is a tutorial in demonstrating how computers actually work.  It isn’t necessarily a long course.  Indeed, it may be nothing more than an hour show on PBS or the Discovery Channel.  Keeping it light, keep it entertaining, but get across the message that computers are complex machines and that it takes a lot of effort to make them do what they do. 
While I don’t totally subscribe to the Coding Horror idea that most programmers can’t program, I do subscribe to the idea that perhaps in our zeal to satisfy the hiring requirements of business we have graduated too many people that still need some work on their programming skills.  We need some way to differentiate between people that know how to write applications and people that just repeat what they learned in school. 
Is certification the answer?  Well, the Canadian ISP certification talks about how you are a “practitioner of the highest integrity.”  Integrity is important, but it does nothing towards indicating if you actually know anything and, to be honest, the certification does not actually have any impact due to the fact that if you graduate from an accredited institution you pretty much automatically get certified.  New Zealand has the Information Technology Certified Professional, but outside of New Zealand who even knows about it? And, much like the Canadian ISP designation there is a large component based on integrity and not skill.  (Self assessment was the rule until recently.)
The IT industry needs more qualified, competent individuals in its ranks.  The Learn To Code idea is an interesting challenge, but it does not come anywhere near to fulfilling the needs of the industry.  It does, however, do a really good job of advertising for The Code Academy.

Batch Processing

Mainframes are built to move huge amounts of data around.  IBM did a fantastic job of building giant batch processing engines.  Granted there are online options, but, to be honest, their bread and butter is huge amounts of data in and out.  Because of that mentality, the fact that the course designers at Universities and older people in the IT field "growing up" with this technology and even the integration of the term "batch processing" into the general lexicon of life, people design systems to include batch processing.

*WRONG*

OK, that may be too strong.  Let's say that the majority of the "reasons" for batch processing are actually replaceable with the phrase "because we've always done it that way".  Batch processing, the processing of huge volumes of data in a scheduled, off prime time manner, is a relic of the mainframe and needs to be replaced with the idea of a batch of one.  This idea of a batch of one is not new, as I`ve even used it on the mainframe.  The user needed certain processing done immediately as opposed to that night and we dynamically created a job, complete with the JCL and necessary parameters, in a low priority job queue and the job was processed when the machine had spare cycles.  It was batch, but a batch of one.

The idea is applicable in today's world.  When I ask people why they are creating a batch job they say something like "the printer is loaded with a certain paper at night so that is the only time we can do this".  *WRONG*  The processing can be done at anytime, the printing can only occur at night.  Instead of doing a lot of processing at night, do the processing in an asynchronous manner throughout the day and do the printing at night.  Split things into two pieces:  processing and printing.  They are two separate functions and do not need to be combined.

Let me rephrase in a more "global consulting" manner:  by separating processes into their unique steps, you create more opportunities to provide value to the business client.  Through a selective process of decoupling the steps in a business process you provide opportunity to inject new business functionality into a more flexible framework with less rework to the existing systems. Isn't that really what the business user wants?  More flexibility?  More value?

Short tempered

Some have said that my comments in the Daily Migration Note are brusque or just plain old rude.  Others have said that I am condescending in my notes.  A number of years ago a project manager actually said he found me "abusive and your attitude repugnant".  (I always keep those good emails so that I can refer to them later.)  Others call me up to see if I am still sane or whether I'm going to go postal.

I am, by my own admission, a strange duck.  Pressure doesn't affect me the same as other people.  I've been in situations where an the vast majority of an organization was sitting on their hands, waiting for me to come up with a solution. I've been the key person involved in resolving a payroll issue that affected tens of thousands of people.  While they are stressful situations, the pressure of the problem, the adrenaline that is produced, all blend together.

And yet, some things make me react quite harshly.  Yes, even quite rudely.  Why?

At a recent event there was an opportunity to evaluate myself based on a number of different characteristics.  Based on my answers to the questions, I was (drum roll please) not a people person.  This may not be a surprise to a bunch of you, maybe all of you.  What the quiz showed me was something that I knew all along:  process and logic are the key factors when I am dealing with work.

So, what does this mean?  Well, if things don't make sense to me I have a tendency of digging my heels in and saying "No".  If there is no logical sequence of events I have a tendency of saying "stop".  If people try to pressure me to do something that I think is illogical, I fight back.

Do you know what actually makes sense to me?  Something that is a logic progression when resolving a problem?  A partnership.  Not a partnership where one or more sides pay lip service to the idea of a partnership so that they can check it off their list of marketing responsibilities (there are many such partnerships amongst major corporations that fall into this category), but a true partnership where each side is working towards a common goal and where all parties are free to bring forth ideas to the table.  No finger pointing.  No "blame game" tactics. 

I have been involved in a number of different groups that were true partnerships between two or more groups and the results are amazing.  With the freedom to make any suggestion about any topic without fear of reprisal the ideas and solutions that come out of such a group are amazing.  There is tremendous difficulty in this as you need to leave your ego at the door and be willing to admit you were wrong.  You need to understand that there is more than one way to skin a cat and that yours may not be as effective.  And, in a true partnership, you don't try to pressure people to follow your lead or your suggestion, you true to influence them and persuade them that your way is appropriate to try.  As soon as you try pressuring people the partnership is gone.  And, in my case, I start resorting to my stock answers of "no" and "you must be kidding".

Partnerships.  Give them a try and you'll be amazed at what you can do.

Debugging

Don't blame the Deployment Team if your application fails and you can't figure out why.

Increasingly we have seen project teams with problems in Production (or UAT for that matter) and they are at a loss to explain what the problem is.  Sometimes there is a cryptic error message in one of the event logs along the lines of "Object reference not set to an instance of an object".  <sarcasm> Now that is a descriptive error message. </sarcasm>

There are a number of things you can do to make your life easier:

• The various .NET languages come with a feature called a Try...Catch block.  Where you try something and if things don't work you catch the exception.  Amazing how that works.  If you haven't put Try...Catch blocks in your code, particularly around any code that is calling a service outside of your control (SHR, EPS, DMS, PRS, etc.), then you score -5 points on the attention to detail rating.
• Did you know that if you catch an exception you can add your own information?  Yes, if you've caught something you can add information such as what you were doing when you got the exception or copies of the data used in the method so that people don't have to guess at what went wrong.  If you just rethrow the exception without adding any information, deduct 5 points from your "value added" score.
• Did you know that there is a Trace class in .NET?  It can do all sorts of things for you, such as allowing you to dump out detailed trace information (that is if you've written detailed trace information in your application) by changing a config file or even dynamically through code.  This is a very handy feature when used in combination with breadcrumbing so that you can see the complete route that an application has taken and, if you've added it, the data used throughout the process.  If you've never used Trace then how do you ever expect to trace anything in Production?

There are lots of things you can do to make your life easier when it comes to resolving problems in Production.  Simple things that can be useful in any environment.  Simple things that you should know how to use and how to use effectively.

Introduction to an old friend

I'd like to introduce to you, an old friend of mine.  The PRINT statement.  Granted, he's not as fancy as a GUI Debugger, nor as sexy as ANT or AVICODE, but he's still really useful.  Have anyone thought of adding PRINT statements to their T-SQL that they are executing?

For instance you could have something like this:

(14 row(s) affected)

(14 row(s) affected)

Or something like this:

Merge Learner Script
(14 row(s) affected)
Learner Merge - From SIN : 918728130 To SIN : 665171625
Start date - 2008-08-22 09:30:03.120
Create an Activity
Close the Activity
Learner Merge - From SIN : 999014679 To SIN : 654457258
Start date - 2008-08-22 09:30:03.420
Create an Activity
Close the Activity
Learner Merge - From SIN : 999011572 To SIN : 647141506
Start date - 2008-08-22 09:30:03.480
Create an Activity
Close the Activity
Learner Merge - From SIN : 917032336 To SIN : 661696161
Start date - 2008-08-22 09:30:03.527
Create an Activity
Close the Activity
(14 row(s) affected)
Done at 2008-08-22 09:30:03.590

The second version looks much better as it is clearly describing what is occurring inside the script.  If you are trying to find out what is going on the second one is so much better.  If you have an auditor on your butt, the second one is so much better. 

Why don't more people do this?  Well, it takes effort.  Typing the characters can consume multiple seconds per statement.  <sarcasm> I mean, over the course of a year this could add up to an hour or more of your valuable time. </sarcasm>

I'd like to thank the SFS team and Janice Pyle for the second example above that clearly shows the advantages of the lowly PRINT statement.

Using a hammer

Have you ever seen Canada's Worst Handyman?  I must admit that I am all thumbs, but, even so, I'm better than these guys.  There are lots of things to look at in this show because so many of the items can be translated into almost any occupation, including the IT field.  One of my favorite things is watching the people, when confronted with a screw, pick up their hammer.  Or using any one of a dozen other tools in the wrong way.

IT people have the same tendency.  We have a problem.  We have a tool.  We try to solve the problem with the tool, regardless of whether or not it is the right tool for the job.  Sometimes you have no tool, just a problem, and in that case almost any help is worth the effort.  Well, maybe not.  Sometimes the effort that the tool saves is totally outweighed by the costs associated with using the tool.  Much like a carpenter that has a plethora of tools to help him do his job, IT people have a variety of tools to help them as well.  Our problem is that there is no simple guide saying "for this problem use this tool and for this problem use this tool".

That's not to say that if we had such a guide people would actually use it.  I mean, look at Canada's Worst Handyman, they were taught certain things and still ignored what was taught.  I have no fear that even if we told people to do X they would still end up doing Y.

I'm not restricting myself to just tools, but methodologies as well.  Some people grab on to a methodology and use it like a life preserver.  Not necessarily a wise thing.  An all or nothing approach is inevitably a bad thing.  Even the Agile Methodology that people either hate or love (there we go with extremes again) understands this.  One of it's principles is:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Essentially, you evaluate what you are doing and adjust if necessary.  I guess my biggest fear is the life preserver mentality for methodologies.  A methodology is a tool and proper use of the tool depends upon proper understanding of the problem.  We sometimes adopt methodologies because they are "cool" or have the "agile" word somewhere in the title.  We don't always understand the problems that we are trying to solve with the methodology and are just so thrilled with using a new tool that we overlook some of the challenges that we have with the tool.

Don't get me wrong, I love new tools, but how many times have you ever used a new tool incorrectly the first time.  If you are going to make a big leap to a new methodology/tool, make sure that you've tested it first and that it actually solves the problems that you need solved.

Quality vs. Speed -- The Agile Way

There has always been a debate over producing a quality application versus an application that is delivered on time.  Indeed, even within the quality camp there are divisions over what exactly constitutes quality.  There have been efforts in the past to merge quality and speed of delivery so that you can get a high quality application in as short a period of time as possible.  The Agile methodology tries to merge these two, to mixed results.

Why mixed results?  Like everything else in life, more than just how you do something determines the end result.  In the case of the Agile methodology (or should I say methodologies) there is more than one way to skin a cat.  The Agile methodology is so different from what people were taught in school that there are three large stumbling blocks that need to be overcome in order for Agile to be truly effective.

• Organization.  There are forces at work in an organization to keep things moving forward.  Plans need to be made, followed and tracked.  The agile methodologies have the idea that "responding to change over following a plan" is a better way to develop software.  And, while I personally believe this, organizations tend to want plans.
• Team. While Agile has come up with some basic philosophies behind how to develop software, the larger the team the more unlikely it is that these philosophies will be effective due to outside pressures: when is going to be done? It is difficult for project managers as they are measured on progress.  Agile, because of its fundamental desire to respond to change, is havoc on plans, at least traditional "we will have this functionality delivered by that date" type of plan.
• Individuals. Everyone comes to a project with baggage:  their old projects and the "we did things this way" ideas.  Getting everyone on the same page, following the same ideas and concepts is very difficult. 

This is not to say that Agile will not work and that you will produce a quality application in less time.  The problem lies in being able to show somebody that this will happen.  Without a proven track record it is hard to show people that it will work, but you need that acceptance so that you can build that track record.  And without the right people in the right places, Agile is never going to fully accepted, fully understood or fully implemented.

Results vs. Methodology

Sometimes it is a story that strikes a chord and makes me write one of these notes.  Sometimes it is an ad on tv.  In this case it is a newsletter from Bruce Schneier, the Chief Security Technology Officer of BT.  He said, that governments should:

...legislate results and not methodologies.

That immediately showed up on my internal radar screen.  While his examples were with regard to things like legislating securing personal data, but not legislating which technologies to use.  Using my own quirky logic I equated this to:

...contract for the end result of the project, not how the project is going to get there ...

When a company puts out an RFP for a piece of work, they routinely say that the winner must "... follow all standards and processes currently in place or that may be put in place during the course of the project...".  We've all seen these as they are not specific to government.  But is it the right thing to do?  Let me go out on a limb and say "no".

In the long run, does it matter if the project team has detailed requirements tracing from the initial conceptual design stage straight through to production?  If you had two projects that delivered the same end result, does it really matter which route they took to get there?  If there are certain deliverables that you want then specify those deliverables in the contract.  Don't specify how you want them created, just that you want them created.

Coming from a programmer background and having been introduced to one of the most document oriented methodologies in existence - Method/1 - I've seen both sides of the coin.  Method/1 was created, to be honest, to minimize liability should the client ever decide to sue.  Dozens, hundreds, thousands of documents are routinely created so that an audit trail could be created in the event that a law suit was launched.  Method/1 was created by auditors, for software development, to be used in the event of a law suit.  Not exactly a glowing endorsement of a methodology.

Other methodologies are created for other purposes, but what it really comes down to is results.  You have certain expectation of results.  And whether the application was written people in blue suits with a document laden process or by a long haired geek who wrote it between episodes of Death Note, it shouldn't really matter, as long as the results required are achieved.

Your Environment is Wrong

"But, it works when I do it, so your environment must be wrong."

When I first heard that comment I took it seriously.  What could be wrong with our environment?  Why couldn't we install the application? 

Not different, because our UAT and Production environments are different than the development environments for a lot of reasons, but wrong.  I searched for something wrong.  Occasionally, I did find something wrong, but as we put more and more applications on to our UAT and Production boxes the number of things that were wrong obviously diminished to the point where their is not a whole heck of a lot that can be wrong.  The differences, however, are still there.  Differences that may prevent the application from installing or working correctly.

• clustered environment - we operate in a clustered environment, so never, ever think that this is the only application that is being installed on this server and never, ever think that it is the only instance of your application which is running
• not a clean environment - the odds are that we've installed your application before - sometimes once or twice before and in some cases a hundred times before - never assume that the environment is "clean" because the odds are that there are remnants of other installs of yours left around
• multiple installs - never assume that just because there is some manual configuration at the end that needs to be done that it is "no big deal" because in our UAT and Production environments we need to do the installs multiple times so that any manual configuration that needs to be done will be done multiple times, giving us plenty of opportunity to make a mistake

"Mistakes"?  Yes, we sometimes make mistakes when manually configuring an application.  Considering that in the UAT and Production environments we are going to be installing and manually configuring your application seven times or more, once we add more servers, this gives us seven chances to make a mistake.  Combine this with multiple releases to UAT and you've got a wonderful opportunity to introduce an unexpected bug, with the most likely problem being that the manual configuration was not saved on a specific machine.  Have the application self-configure itself where possible: prompt for data and complete the installation itself.

Oh, I forgot, you're only going to UAT and Production once, right?

More examples

Going on vacation seems to give me plenty of ammunition for these notes.  The following example of what not to do is based on a conversation with the Day Manager at the Sandman Inn in Calgary West.

My wife and I needed to go to Calgary to drop our kids off with her sister.  They were going to horse riding camp for the week and we thought we'd start it off early by going to a hotel when we got there instead of sleeping at sister's place.  On the Friday before we went we reserved a room at the Calgary West Sandman Inn through the Sandman web site.  We got a room with a King sized bed and, most importantly, a confirmation number.

When we got to the inn on Saturday, however, we were surprised because they did not have the room that we reserved.  They had a room in the same "class", but they did not have any rooms with King sized beds.  I was a little surprised because I thought a confirmation from the hotel chain guaranteed the room was available.  I was told by the clerk at the front desk that the Sandman web site and the computer system at the hotel do not talk to each other.  And that they had been trying all day to get hold of someone to have the web site stop booking rooms.

I was, quite frankly, amazed at this revelation.  Imagine building a reservation system for a hotel chain, but not hooking it into the hotels reservation system!  So what did I do?  Well, there was nothing to do about the room as there were supposedly no other rooms in the hotel.  (They did give me a $3.00 discount for my inconvenience.  The pop in the pop machine was $2.50)  When I got to the room I immediately hooked up to their Internet connection and went to the Sandman web site and filled out a form with a complaint.

After waiting 5 days for a response I called the 1-800 number for reservations.  I got the Manager of Reservations and she was going to call the Calgary West Sandman to see what the scoop was.  Three days later I got a call from the Day Manager.  She confirmed that the systems do not talk to each other and that in order to get the web site to stop reserving rooms the general manager has to fill out a form and send it in.  (He only works 9 - 5 on weekdays, so if something needs to be done outside of this timeframe, oh well.)  By the way, the 1-800 line uses the same system the web site does, so that does not guarantee you the room either.  You need to talk to the hotel itself.  Same thing is to be said with regard to Expedia or Travelocity.

So what do we have?  A hotel chain with thirty five properties that reserves rooms without actually knowing whether or not the room is available and a form based system that is required to be filled out to stop selling rooms.  Does anyone see a problem here?  As I escalate this problem up the Sandman management tree I'll let you know if this is going to be changed.  If not, unless you call the hotel directly, don't bother reserving a room with Sandman.

Planning, ad nauseum

OK, I've talked about planning in the past and I'm going to mention it again, but from a different perspective.  There are many different types of planning for many different types of people:

Programmers. These people need to know what they are doing now and for the rest of the week.  To be honest, while they might like to know what they are doing next week, as long as they are confident that their boss knows, that is probably good enough.

Team Leads.  With programmers reporting to them they need to have a much better grasp of what is currently going on and what is coming up.  A week isn't good enough.  A month may not be good enough depending on the project, so let's say that these people need 6 weeks of planning under their belt.

Project Managers.  Ah, the brains behind the project.  For any particular project that  they are working on, they need to have a good idea of staffing requirements from beginning to end.  Yes, they need to know staffing for the entire project, because if they are short, it is their responsibility to get staff.  If they are going to be over staffed then it is their responsibility to see what other projects need help.

Program Managers.  These people are responsible for the smooth execution of multiple projects within an overall program.  For example, you may have a Program that is composed of a lot of smaller initiatives, all of which are leading to a single vision, much like PASI.  Program Managers need to understand the interrelationship between the different projects and how they impact other projects and the program as a whole.  This sort of planning stretches into the years.

Organization.  Different organizations use different benchmarks, but it is considered a good idea if an organization can tell you what they are going to be doing for the next three calendar years.  This will not be to a great deal of accuracy three years down the road, but the overall plan of the organization, the inter-relationships between ongoing programs and the intended implementation date of various projects should be at least plotted out so that both clients and staff have an idea as to what is currently happening and what the short and longer term goals are.

As you can see, the level of detail goes from the microscopic, tactical information needed for day to day activities to the macroscopic, strategic information needed for long range plans.  Virtually everyone in an organization needs a different level of planning with a different focus, depending upon the roles that they are currently occupying.  The key to all of this, though, is that there needs to be someplace where all of this information can be stored.  Some place where the microscopic and macroscopic views can both find a home.  A place to go when information is needed and a place to go when you have new information.

Peer Review

Why are some people against having their work reviewed by their peers?

Commonly called a "peer review" in academia, the concept revolves around letting other people knowledge in the same field comment on your paper (design, theory or observation) prior to "publishing" it.  Specifically to the IT world a peer review is when a design is reviewed by knowledgeable people in an effort to determine any flaws in the design.  I will grant you that these reviews can be brutal on the ego, but aren't they the right thing to do anyway?

One of our primary objectives in our job should be to produce quality work.  Now, the exact measurements behind defining quality are highly contentious, but in general, the idea is to produce something that accomplishes the objective and does it as efficiently as possible.  One of the keys to creating a quality product is having a good design, one that is as simple as possible yet still satisfies all of the objectives and goals.

A good design.  How do you know if a design is good?  Well, one of the ways that the rest of the world  does it is through design reviews or peer reviews.  When a design has been completed it goes to someone else knowledgeable about the requirements so that they cam see if the design does indeed meet those requirements and does it in an efficient manner.  Seems pretty simple doesn't it?  So why don't we do it more often?

The Enterprise Architecture Working Group (EAWG) did this, to a very limited extent, when they reviewed the application architecture on various projects, but the problem was that at 50,000 feet everything looks blurry.  As a friend of mine used to say "Good from far, but far from good".  Understanding how someone is going to piece things together is important, but the details, the microscopic manipulations that occur on an individual program level are completely oblivious to everyone except for the designer and programmer.

Every critical piece should be peer reviewed to ensure that it is designed properly.  This means that if you are writing something that manipulates funds it needs to be peer reviewed.  If you are writing a mission critical import/export/reporting piece, it needs to be peer reviewed.  If you are writing a security component, it needs to be peer reviewed.

If it's never bee peer reviewed, never had an objective review of it's design, how are you every going to know if you designed something properly?  Is it fear that is stopping you?  Fear of people pointing out mistakes?  Or is it narcissistic in nature?