During the first 8 years of my career, I was working on an extremely interesting, super large (more than 8M LOC), mission-critical and legacy system (so much for adjectives). It was a real-time data acquisition system the company invested to build to replace two old systems – a plausible ambition, state-of-the-art technologies, top-notch talents in a dream office (much nicer than many of the big techs today) and lots of $$ investment. However after many years of effort and $$$$$$ investment, the result diverged seriously from the original ambition (do not ask why, there are zillions of reasons – we actually wrote a long document summarize the lessons learnt). Then it becomes a legacy system the company leadership team at the time decided to ship to another location to give a last try (and closed the site, which is a pity – such a beautiful place), that was where we came in. We were extremely excited about it (and we were young 😎), legacy or not, it is a super interesting/critical system and technically very challenging. In a blink of eyes, 8 years passed, and we have dedicated our best years in career to this system. We went through ups and downs, and also made many life-time friends because of this system. Well, back to the topic, I want to write down a few sensations and experiences I had in those 8 years savoring the system, in memory of the best few years of my career.
Have no fear and take the pride
When we were assembled to go take over the system by the leadership team at the time, we have very little clue about what we are getting into. A group of young “pops”, with zero experience, landed in this “too good to be true” place in US, ready to “conquer the world”. Yes, we have nothing but we were fearless! That’s exactly what get us to where we were many years later to successfully deploy the system and finally helped the company to return its ages of investment. We were assigned a group of awesome mentors and they have played crucial roles to get us onboard with the system, many have been fostered to become life-time friends. We have went through many ups and downs (saying it’s like roller coast is not exaggerating at all), burned many night oils, but we have never lost the spirit and just keep pushing forward. At one point, the project became this “taboo” project that you are cursed to be working on it. But even that, we were still quite proud of working on it, one colleague (wise guy) said: “We dedicated our youth to this system, and we will continue dedicate our kids and grandkids as well.” That’s the spirit! Experience tells, sometimes thing appears to be really really hard, but have no fear, take the pride and keep moving forward, you will be surprised (pleasantly) how it turns out at the end. More importantly, along the way, you pick up many treasures in life which will be carried on with you forever, how rewarding that is!
The golden strategy: divide and conquer
The best way to solve complexity is, no surprise, divide and conquer! 8M LOC, it is a monster system at first sight, particularly there were so many runtime issues with it. Together with our mentors, we have divided the system into subsystems, which further divided into components. Each one of our names are assigned to a few components. This mapping exercise breaks down the overwhelm first impression to a much more manageable experience. 8M is hard to chew but 10k you can probably chew over a few days. Collectively we have increased our understanding of the system drastically over a few weeks. We became much more comfortable of troubleshooting the issues for our responsible sub-systems/components, and eventually get very comfortable of adding new capabilities. Later one, we continued with a lot of cross-pollination trainings so that many of us can take on the system level activities. This also grows a generation of system architects.
Get it out of the door quickly but safely…
Finally, it comes to the dilemma of ship it or not. The system has been cooked for at least 5 years when we took over, and none of the real user has actually used it. It has super rich functionalities trying to cover many use cases, however none of the functionality works flawlessly. That’s the problem. Working with business team, we have focused the effort on a few “most used” workflows and get them “bullet proof” relentlessly. At the end, it turned out that this is a very effective strategy. The system can fulfil the most fundamental workflows with super high quality (this is crucial for mission-critical system as the cost of error sometimes can be unbearable), so that we can start deploying it and get it into the real end user’s hands. This has to be followed up by super efficient response process and critical issues encountered can be addressed instantly. It poses a lot of pressure to the team but on the flip side, finally what we have been working hard on for years is actually being used, what is more rewarding than that for software professionals! Here it goes, it took off and allowed to incrementally increase the capability of the system, and eventually reach to a global deployment.
Looking back, it seems that we have only worked on one project for so many years, but the experience gained through such a multi-discipline, mission-critical and complex system is invaluable. It developed a generation of highly-skilled system architects and multi-skilled leaders for the company. Most importantly, the people you work hard with for an honorable cause becomes life-time friends and made “a valuable dent” in each one of our lives.
Bin's Blog 