This will be a a long one…
After an year of hard work, our whole team were anxious and waiting when the customer finally decided to buy the new product. It wasn’t easy for anybody, neither for us nor for the client - New product means a lot of change, and a lot of questions about performance and reliability and business usually doesn’t like ’new’ changes.
However, I landed in Tokyo on Mar 4, 2007; to represent my team and help the client transition over to the new world. I had a room in Keio Plaza hotel in the heart of Shinjuku area. If japanese gave me such hospitality, I knew they would be expecting something unnatural from me. My US managers chose me out of everyone (I thought its because they trust me so much which might be true, but now I have another perspective of looking at it).
Well the D-day had begun on 5 march, with a public relations manager from the client office waiting to escort me precisely at 7:00 a.m. It was a beautiful day, people walking briskly to their destinations with formal suits and overcoats, carrying little briefcases - uncommonly professional. I had worn my first suit too which I had bought a day before for 10,000 bucks. I just arrived at the office and went straight to work, the only social moment being performing the Japanese ritual of handing over the visiting card and bowing.
Everything was set, I had a private DSL line, company laptop, VPN Id, everything. The users were already on the new system, and for first 2-3 hours everything was going fine. The system was very friendly, highly responsive and people were happy. But I was anxious and so was the team. Everybody were up, though it was in the middle of the night in US. We had a quick conference call, all the AVPs, the COO on the client side and US managers. It was a path they had decided to take which has no return ticket. Once they are on the new system, once the processing completed for that single day - that was it, they have to live with change, the new system.
The anticiaption was killing me. Though I am very much impressed with what we had accomplished and proud about where we have reached considering how it was in the begining, I still knew places which required attention and according to me were still less than perfect. But, that was a day 2 concern and the client didn’t care about it anyway.
I had got atleast 100 calls in 2 hours, requesting security changes, account unlocks and stuff like that, but all that was expected. No major issues still.
And damn, then it started. One of the users couldn’t see any fields (Japanese) displayed on one of the transaction screens and the AVP started running around, trying to describe the problem in his English which is usually difficult to understand. I thought it was a major issue, but within few minutes could find out that it was a language setting problem on the user’s PC.
Nothing much happened, but I still had the AVPs and CIO standing behind me and watching everything I do on the system. It was criminally freaky.
Phew, another such problem. A user was getting a creep error message, which didn’t give me anything useful whatsoever. Then it started happening with all the users, then all the real time transactions gave the same creep error message. And I knew, we had it coming down on us.
I started surfing through the amazingly huge logs, comparing time stamps of when the error occured.. it was like a thriller movie. Just after a minute, I had all the above mentioned people standing behind me, breathing down on my neck and asking me every 30 seconds whether I fixed it or not. Its a beauty of the computer technology that, it doesn’t give a rats ass even though the management is stark ravingly mad at you and yelling. To make the situation more interesting, they told me to solve whatever it is in 35 minutes. Of course, I accept they had every right in the world to tell me so; it was their business which was going to dogs.
I wanted to call my US manager, and these guys never let me speak in private; The AVP himself dialled and kept quiet during the whole time trying to smell if theres anything fishy, till I specifially mentioned that he too is in the call. Ok., now its up to managers, let them bark at each other and wonder why pluto is no more considered a planet or something; I was back to work. But its really funny and a great personal dilemma on what information to disclose to the client and what to be kept secret. One wrong word, and the deal goes to hell. Though everybody there asked me everything, all I said was it needs to be discussed with my manager and team; I was being loyal and was ready to pay the price for it.
I came back to my system, and all logs were looking great to me. Nobody could determine what the damn issue was and I had only 20 minutes left in my pocket. I wanted to recycle the server (Of course its a windows machine, and there’s no fix better than restarting the application). But anyway, they had decided to backout to the old system in 10 minutes. In the interest of the client everybody agreed to cut back, and in 5 minutes they were back to business. I restarted the application and tried to recreate whatever happend and good lord, things were perfectly fine!!!
Anyway, we had already blown it and the root cause for the issue still refused to reveal itself till we get few sherlock holmes to the scene. But the issue never reoccured once we went back, and everybody told me it was really unfortunate that such a thing had happened. I was amazed on how calm I was, and nothing stuck me the whole day. I was wondering what was there to take personally offending when one of my managers gave me some soothing words of not to take anything personal.
It hit me quite late, not until the next day I went to office. I didn’t have anything to do there, my mission there had no more relevance and meaning. Of course, we had managed to find the problem and I could explain to them clearly what had happened, but thats all I had to do. And all the people there were so good and friendly which made me feel very guilty to waste time and do no good to them. They had spent so much on me and I couldn’t ignore that fact. Its at such times that one will realize that Business is what that drives technology. Our brainchild didn’t see the world and it was really painful to have that sink in slowly.
I wanted to come back and had already requested to reschedule my tickets. But it was much more painful to hear the AVP coming and telling me that he doesn’t like to waste my time here and indirectly indicating to get the hell out of here. It does make perfectly logical sense but its painful.
Of course I did learn some interesting lessons and made some observations, may be I’ll post it in another blog.
Well enough is enough, I am now eating a bowl of cornflakes, staring at the skyscrappers from my 27th floor room and wondering how the japanese language evolved and whether it had anything to do with the system not working.
-- There’s no free lunch.