Interrupted ... Again

OK, now that you've taken a look at your own life, realized that you are totally unproductive due to interruptions, what can you do to become productive again?

Work at home, or at least, away from the office..  Seriously.  Talk to your boss and see if there is a way for you to spend every second Monday/Tuesday/etc. at home to do your work.  For some of you this isn't going to work due to the work that you do.  For others, those that producing tomes, need some time to plan the next great project or even, goodness gracious, catch up on that documentation you forgot to do - six months ago - spending some solid time working on these items can be a tremendous benefit.  Now, you don't necessarily have to be at home.  Some people find that working at home is not as rewarding as it could be due to hundreds of other distractions.  An alternative location is the key.  Whether this means you book a meeting room for all day, find an abandoned office that no one is using or even use the desk/office of someone who is on vacation, getting away from your desk, the spot where people find you, is key.

Another step?  Flush your Crackberry down the toilet.  Figuratively.  Unclip it from your belt and put it in your briefcase/deskl/backpack and bring it out every hour and see what you missed.  The constant buzzing of new messages, the incessant weight just totally p***** me off sometimes and I need to just get away from it.  Why check every hour?  Well, research has shown that people work most effectively if they take a five minute break every hour.  Go to the bathroom, get something to drink, check your voicemail, etc.  Just remember that this is a five minute break and not the start of five minutes of different work.

Turn off email notification (thanks Garth).  You know that little window that comes up and says "You've got new mail and here is the subject line and who it is from..."  Turn that off or even shutdown Outlook when you don't need it.  Just remember the five minute rule once an away.

Take a look at sites like Lifehacker (Tech tricks, tips and downloads for getting things done).  (On your spare time, of course)  This site contains some useful and some useless tips on how to keep organized and get your life back together.

Unfortunately, for some of the people reading this note none of the solutions are going to work.  You know who you are:  support personnel.  Those people who support other people and who, quite frankly, need to be in constant touch because their clients need help.  An email about the best way to re-engineer a COM+ application can be done later.  An email about a crashed system needs to be handled NOW.  So, while many of you can take advantage of these and other tips, others ... oops, have to go, later.

Environment Building on ten cents a day ....

I have been giving project team members a hard time about advance notice.  You know what I mean, giving the Deployment Team more than 15 minutes notice that they need a deployment done.  Well, I'm not going to bug developers in this note, I'm heading higher up the food chain for the project managers.

I've talked about how we are short on space and short on power.  Apparently many project managers (and I am talking about a good dozen or more at the moment) think that these facts don't apply to them because they are special.  You know what?  Everyone is special.  When there are so many special projects, the appellation is meaningless.  So, let's talk turkey.

I want a performance test environment, and I need it next Monday.  OK, to be honest I haven't heard this .. too often.  Most projects give us a lot more warning, although sometimes there is not a lot I can do for them.  (Did you miss my notes on not having space and being electrically challenged?)  Let's take a look at a typical application and see what a performance test environment entails.  A smallish application that access Stakeholder Registry.

So, what do we need for this performance test environment?  Well, we could go with clustered web servers, but let's give ourselves some leeway and just pick one.  We then need a SQL Server box for SHR and a SQL Server box for the application database.  Wait, in Production these are SAN attached, should we duplicate that?  Well, that would mean buying another SAN and the storage and configuring it, etc., so for the purposes of our example let's not go there.  Oh, wait, in order for this to be even remotely representative we need to emulate the existing servers, not buy new ones.  So now we need Dell 6850s, but they're not made anymore.  Have no fear, eBay is here!!  Oh, darn, that reminds me, we need to get an Dell 1850 for the web server.  I hope we haven't gotten rid of all of them yet. 

OK, we done yet?  No?  What else do we need?  Well, if we are using SHR we need to hook in ISM, which requires another license for SSA-NAME3 as well as a Dell 2650.  (Remember eBay?  It's your friend.)  Here is an interesting fact for you:  If you restore the SHR database, you need to rebuild the database that SSA uses for searching.  This probably isn't more than a six hour job these days, but it might be longer.  I guess this means that if you run a test at 9:00 AM in the morning we might be able to rebuild everything for a 3:30 PM test.

I'm pretty sure that you're not going to be using the stopwatch facility on your Timex watch to record everything, so that means we need some tools for testing and recording.  Let's assume that we have these, but the tools we might need are Intercept Studio licenses (aka Avicode) to pinpoint some performance issues for us.  These are not necessary, but they have proven themselves to be quite valuable in the past.

There are some other infrastructure things like backup tapes, disk space for intermediate backups

So what do we have?  Four computers that need to be purchased, refurbished and installed in a computer room where we are already short on space and power.  Additional licenses for software we don't have.  A lot of effort on part of a lot of different people just to get things in place.  Just think what would happen if we needed PRS (Oracle server and license) or perhaps CGES (two servers and SharePoint) or, heaven forbid, all of those, plus more!!!!

Or, *shudder*, you have multiple projects coming to you at the same time with the identical requirements.  Welcome to my world.

Setting up a performance testing environment is very expensive in terms of hardware, software and sheer manpower needed to set it up and keep it running.  The question is, what are your really trying to test?

Don't get me wrong, I like the fact that you want to performance test.  I guess what we really need to understand is what you are trying to test.

Loss of Power

Electricity.  It lights our houses, cooks our food and allows us to watch and listen to events that occur half way around the globe.  It powers our watches and allows us to pick up a phone and call someone a thousand miles away.  It powers our lives.

It powers our computers.

The idea was that computers would become smaller, cheaper, more efficient at doing work for us.  Guess what?  In order to do this they consume more power.  The average Dell "high end" server is smaller, faster, more capable of doing fantastic things, but it also consumes about 20% more energy than its predecessor.  Unlike everything else that becomes smaller, faster, better, the power consumption of computers has gone up. People have always expected a higher density for server rooms (more computers in the same space) and server rooms that are constantly growing in size, but for the same number of computers to chew up more power?  In many respects that slipped by peoples radar.

Server rooms have always been built with some excess capacity and ours is no problem.  Indeed, if our server racks were filled to the brim with hardware from three years ago, I think we wouldn't be having this conversation.  But, power requirements for an individual server have gone up and as a result our ability to add more hardware is limited by the capacity of our server room.  We're short on capacity.  Indeed, I don't think we can add all of the new hardware we have without seriously overloading every circuit we have. 

Let's look at some examples and cry:

Example 1:  a Dell 2650 consumes 357 watts while a Dell 2850 consumes 524 watts.  That is almost a 50% increase, even though the amount of space consumed is the same. 

Example 2: a Dell 2950 consumes 307 watts and takes up 2 U of space, approximately 154 watts per U.  A Dell 1850 consumes 444 watts in a single U.  So while we consume half the space we almost triple the power consumption per U.

Newer machines are greener, consuming less power per U of space, but we don't have a lot of the latest generation of machines, those that are efficient with their use of electricity.  In the past we've upgraded machines to give clients the power they need and we've accidentally hampered out ability to expand.  The Dell '8' series (1850/2850) are huge millstones around our neck as we're trying to stay afloat in the electricity ocean.  So what are we doing?  Replacing as many 1850s/2850s as we can.  (See yesterdays note on virtualizing, evergreening and replacing).

So, we don't have a lot of room and the servers that we do have are consuming huge amounts of power.  Can't get any worse, can it?

No space at the inn

OK, for a number of months now I have been using the explanation "Sorry, but we have no room at fulfill that request."  Many of you have been the recipient of this sort of email and you may be thinking that I have just been using this as an excuse.  While there are many things that impact the "room" that is available, let me just talk about physical room today.

Within our server room we have a number of racks.  These racks are industry standard racks for holding rack mounted servers.  (Hence the name similarity).  These racks contain 42 U of space. and we have 17 of these racks in place. That's a lot of space.  We are also about 80% full. Theoretically we have enough room for about 70 servers or so, based our own current 2 U standard server.

That was theoretically. In practice we do not have this much usable space. Due to the fact that the older servers had a wide range of sizes, the fact that we put more than just servers in the racks (storage enclosures, load balancers, etc), the amount of usable space where we can install our standard server is about 72 U in size; or about 36 servers. This is pretty close to the number of servers that we need to install, but this gives us absolutely no room for growth. (Based upon meetings I've had recently, we are going to be growing.)

So, how do we free up space? Virtual machines.

We have converted about 40 machines to the virtual variety in the past few months. We want/need, to do more and we are going to that. My personal goal is to free up another 50 U of space in the next couple of months. so, that we can have that room for growth. We need some help from Microsoft at this point as we are having problems with one of their "P2V" (physical to virtual) tools that is making this transition a lot easier.  They have managed to replicate our problem and are working on a solution.

We are currently in the process of mapping out precisely what is in our server room, which rack it sits in and exactly where in the rack it is taking up space.  Based upon this very detailed map we will be able to plan out where new machines should go and what we need to do in order to consolidate some of our fragmented free space.  (Much like a disk defragmenter, but on a much larger scale.)

This is not the final solution, however, as a larger battle still looms and we will discuss that tomorrow.

Budgeting

I'm not really a big fan of budgets.  My wife and I try to make them and then we blow them out of the water within a month.  For projects, however, budgets are important as they lay down some limitations on what can and cannot be done.  Sometimes, however, we don't set the budget properly.  Let me give you an example:

You walk into your bosses office and they tell you that they have a new project for you.  They tell you what the end result of the project is going to be and what your budget for the project is.  The problem is that there has not been any detailed analysis done so the size of the project is totally unknown, but not the budget.

I'm sure that most of you have been there before, even in a somewhat different perspective.  Imagine if you will, going up to your team lead and have them tell you about a new feature you are going to implement -- in one weeks time.

it is important to understand that budget does not necessarily limit itself to dollars as budget can also be equated as people time.  If you have X dollars that equates to Y days of effort. Whether you use the top down approach to budgeting, the bottom up approach or the throw a dart at the dart board approach, you need to understand what you are going to build and how you are going to build it.  Once you figure out that you can come up with an estimate.  Based on that estimate you also include a certain percentage of time for documentation, testing, UAT support, project management, internal technimcal support and deployment time and even post implementation support. 

I know, you normally don't do this.  Maybe that explains why you feel so stressed about being late?

Process and DJ

I am a great believer in following "the law", not because it's the "right" thing to do, but because without a standard basis upon which to rest our morals and beliefs we descend into anarchy.  But you know what?  Going too far the other way is just as bad.

This is also true for IT projects.  For instance, the Project Management Institute offers all sorts of guidance on how to manage projects.  The have produced a Book of Knowledge that contains much useful information.  BUT, it is not the understanding of this information that makes a project manager effective, nor is it merely have the PMP certification that makes a project manager good at his job, it is understanding when to use the tools at their disposal and the reasons for doing so.

Over-managing a project is almost as bad as under-managing.  (Depending upon your perspective it may actually be worse, but I digress.)  Over-managing puts too much rigor into a process that does not require that rigor and that is where the skill of a good project manager comes into play.  Like a poker play you need to know "when to hold 'em, know when to fold 'em, kown when to walk away".  The effective application of the tool is the difference between a good project manager and someone who just follows the rules.

Am I being too harsh on project managers?  Maybe, but considering the wide range of project styles that I have seen and been victim to, probably not.  The project manager has more influence over the end result of the project than anyone else.  Don't influence in the wrong direction.  Give the team the freedom to innovate, the freedom to get their job done while at the same time putting in place the structure necessary to do that.

Picking on Managers ... again

It's been said that I've been picking on programmers too much recently, so I thought I would skip the low hanging fruit and go after something higher up on the tree.  Let's start with one of my favourite slogans:

Failure to plan on your part does not constitute an emergency on my part.

A project manager has a specific role on a project team:  planning and coordinating the activities of the team members.  On larger projects this may actually be the delegation of this responsibility to other members of the team.  On a large 75+ person project I worked on there was one Project Manager who ran the entire project, but four other managers who actually did the planning.  Please note that this 75+ person project had a total of 5 managers.  There were a number of team leads on the project as well bringing down the ratio to 1 to 5 for supervisors to staff.  This is by many standards a fairly low ratio, but it was typical in the organization I was with, with many of the supervisors spending only a portion of their time supervising and the rest of the time working on different tasks that needed to be done.

Each team lead knew what was expected of their team in the short term and what the long term objectives were.  Every week there were meetings amongst team leads and their respective manager and every week the managers got together to discuss the status of the project.

This same type of organization is applicable in smaller projects.  Planning is one of the key factors:  know what needs to be done now and in the future.  Too often have I seen and read about projects that do not have a clear understanding of what is required and the usual impact of this is larger, more complex projects than is required and the resulting cost and timetable impacts this has.

Worst of all, from my perspective right now, this has a tremendous impact on the Deployment Team.  Too often have we had requests like "...but the user needs it now, do we have to give you five minutes advance notice? ..." or "... it's just a small change, but we need to recompile everything and send you a complete package, if we get it done by midnight can you have it in place by 12:01? ...".  T

he reason we ask for advance notice on migrations is so that we have the opportunity to plan our own work.  If you don't give us advance notice we can't plan and that annoys me.  And everyone knows what happens when I get annoyed.

It's not always a technical solution ...

One of the interesting things I learned at NAIT was that technology is not always the best solution.  Sometimes simple, every day solutions, non-technological solutions, are actually better.  For instance:

I take the bus to and from work.  While on the bus if I get a brilliant idea, or even a not-so-brilliant idea for a daily migration note I need to capture the idea so that I can later cogitate on it and then spew it forth.  I carry with me, a Blackberry.  Seems like the appropriate thing would be to type it into the Blackberry using my thumbs, but I've found that, for me, it is the wrong solution.  A pen and a small notepad are what I need.  Faster.  More intuitive.  And I'm less prone to correct my spelling.  Later I can look at it and determine if I want to save it electronically.  Initially though, paper is king.

While I consider myself a geek, I also consider myself to be open to alternatives and, quite frankly, paper is a valid alternative for me.  While I like electronic toys and I like building little applications that do something, I also understand that sometimes pen and paper, a new process, or even revised expectations, are all valid solutions to a problem.  What we, as IT professionals, need to understand is that when a client comes to us with a problem it is in our best interest to come up with a proper solution, regardless of what technology we are or are not using.

If you talk to a user and persuade them to change their process to resolve a problem, rather than create an entirely new system, then you will have done everyone a service.  Over fifteen years ago I was asked to build a system for a client that did some inventory work for them.  Our recommendation was that they change one of their existing processes to handle the requirement, but the client was adamant that he wanted a new computer system built.  We built it in VB4 in about 20 hours.  It didn't have any error handling and the interface was kind of clunky, but it did everything he wanted.  Within two weeks of rolling it out to his staff they changed their business process and dumped the application.  Sometimes you need to experience the pain to understand the truth.

Casual clothes = casual work?

I was watching a report on CNN that talked about "Casual Fridays" in England - London to be specific - and how people did not want to "dress down" for work.  The reason that was given was that having a dress code made it simpler to know what was required and actually made things easier for the worker.

There is another underlying fear, as made quite evident from my days with Andersen Consulting, that if employees dress casually, they will take their work casually and not produce the same quality of work they would otherwise.  While one of the unwritten rules was that you were to follow your clients standards with regard to what you wore, another unwritten rule was that you would wear a tie at all times.  So, while you could dress casually (i.e. take your jacket off), too casually was a no-no.

Being part of the generation that grew up with suits and has subsequently ushered in the "Casual-era", I've seen both sides of the coin.  While I believe that going casual has reduced some of the discipline that is required, I believe it has been more than made up for by the atmosphere in which we work.  A suit and tie impose a certain level of discipline when you're working and losing those pieces of clothing has, apparently, reduced the level of discipline that people seem to have. (Please note that there is a limit to being casual, a limit that some people just don't understand.)

However, I also think that the work atmosphere now is more conducive to change, more willing to accept new challenges and new ideas and, well, more fun than it used to be.  The fact that I don't wear a suit and tie makes me more comfortable at work and, as a result, makes me more attentive and able to focus on a problem.  (Who can fix something when you're convinced that the next time you cough your tie is going to spontaneously tighten itself around your throat and stop you from breathing?)

Back in the suit and tie world the Internet would never have been a success: it breeds contempt for boring; offers success for being different; allows people to do things in amazingly different ways; and was built by people without ties.

The kids nowadays ...

I was talking with someone the other day and we were both commenting on how we both had high expectations of developers.  Perhaps too high.

When I was just starting out in this business the mainframe was still the big thing.  I remember going to the launch of the IBM PC that IBM had in town and thinking that if they did this right IBM had a gold mine on their hands.  Well, they blew it, but the industry did not let the personal computer drop and the world was changed.  I learned lots of stuff on the mainframe and then applied it to the PC world.  I learned how to debug properly when you could only do one compile per day.  I learned how to design things properly when you had 4K of memory to use for your application before you had to start worrying about addressing problems.  I learned how to adapt the mainframe mindset and use it in an intelligent fashion on the PC.

Kids these days don't do that.  Schools don't teach it and the kids have no opportunity to learn what I learned.  While I may have owned my first computer before some of you were born (March 15, 1981 was when I bought my Apple ][+) I've had so much more varied opportunities.  I've worked on large mainframe applications, large PC applications, large web applications,  I've worked on machines were space was at a premium, both disk space and memory and you were challenged to be as efficient as possible.

The kids these days are challenged to be efficient.  If it works on their dekstop then it must work on the server for hundreds of people and, if not, it's the deployment team that needs to make it work.

I think it's a shame that mainframes are being relegated to the back room because I think the lessons that they taught are still applicable today, but there is an entire generation that will not learn those lessons.

Integrity

One thing that continually amazes me about Hollywood, I guess about their writers, is that they sometimes choose the most idiotic, but convenient plot devices because it saves them the effort of actually thinking about something.  In this case what I am talking about is the infamous "back door". 

You know what I'm talking about:  a hacker mysteriously manages to access a top secret application through a simple userid and password that he hard coded into the application when he built five years ago.  You can imagine the sort of code that he put into the application:

if userid = "Fred" and password = "flintstone" then
access = "SuperAdmin"
endif

The trouble is most access systems work this way.  Any access system for a top secret application is not going to use something that the programmer wrote because they want it to be more secure than what they're paying this guy who is later going to hack into their system.

But the thing that annoys me most is the idea that this is commonplace in the IT industry.  I have never written such code for a production application, neither do I know of anyone who has written such code.  I've been involved with systems that dealt with hundreds of millions of dollars in payments, yet there were no secret back doors into the system.  I am annoyed by the fact that Hollywood, and the image they present of the IT industry, is that of a bunch of cowboys with no integrity.  Why don't they show social engineering in action?

Hacker: Excuse me, ma'am, but I'm from the IT department and we've noticed that there is a problem with your ID.  Could you please tell me your password and when we've resolved the problem we'll reset your password and you can go back to your job.

Social engineering is by far the easiest way to gain access to a system because people fall for the lines quite easily.  If you don't believe me just look around and you'll see stories about hundreds, thousands of people who have fallen for the Nigerian 419 scam.  So instead of showing how innocent people can be easily duped they decide to show that the bad guy was evil long before the movie started.  Yeah, like I'm evil.