Advice: preparing for production launch
I was talking to my friend and ex-colleague today. He is a team lead, and we were discussing a forthcoming launch of a product his team is working on. He shared his worrying with me – launching a product and being so responsible for it is a new experience for him.
I had this experience. More than two years ago I have successfully launched a system for accepting and processing online immigration applications for one of Canadian provinces. Our system has several components, and I was not only writing a code for one of them, but also integrating all the components into one platform. I wasn’t head of the project, but I was technically responsible for the launch. And what I can tell now after that launch and bunch of other update launches afterwards is that it’s not that scary as it might seem.
I think it’s clear that the most worrying thing is possible errors. Especially those which can corrupt data. Well, don’t worry. It’s just this simple. Instead of worrying, take preventive actions. I already hear you opposing me that you have been fixing errors all the time, but they just keep appearing from nowhere. That’s the point of this post – it’s unlikely you will release your software without errors. I love this joke (originally it was in Russian and probably sounds better):
There were 100 bugs in the code
One we have fixed
Is it less now?
So in addition to fixing bugs in your code, you need to do something else in order to prepare for the launch.
In order to prepare for launch, not only keep fixing errors, but prepare instruments for fixing errors AFTER the launch.
Let me explain. I did couple things before the launch and they saved a bunch of my nerves because I was ready:
- I introduced a very sophisticated logging system and was writing every user’s action into it. A lot of records were redundant and I am still cleaning them, but log records saved significant amount of time while investigating into issues, trying to understand user’s actions and even restoring some data, because log records could contain some IDs of objects which had been deleted at the moment of investigation. By the way I don’t have direct access to production server.
- I made efforts to make things as smooth for user as possible. I know that hiding errors is not a good approach. Empty catch section is something a programmer often should be thrown out of the office. But making things smooth for end user is another story. User is not responsible for errors in your code. Hide errors from user and instead expose them to your development branch. Log records, status reports, integrity checks – create and use instruments to monitor your system, but don’t bother user with your problems. If users are not affected, you win time for possible investigations, because users won’t overwhelm your support, so your support won’t overwhelm you.
And to put your worry to the ground completely, I can tell you – that if you do your work well, chance of critical errors appearance will be low enough. Do your best to prevent critical errors impacting end users and/or system stability, then prepare yourself for fixing all other issues. Because they will happen anyway. There is nothing to worry about. It’s just part of our job.
P.S. If you are from NASA, Airbus, Boeing or any other company working on a product which people’s lifes and well being depend on, neglect everything you just read. You don’t have right for errors, sorry.