The One Where I Lie To The CTO

This was several years back. Keep in mind that early in my career, my father had told me that doing a good job often meant doing what needed to be done in spite of your boss. And by that he meant that you can either make your boss successful and happy or you can run every decision by your boss. In which case no one is successful or happy.

I was working at the time for a Fortune 500 company and our CTO had signed up to deliver a big project for an important client with whom he had personal connections. He also decided to outsource a key part of it to a large tech services firm who claimed they had a product that would do most of the heavy lifting for us.

As has been typical in my career, when the vendor said they had a product, what they really meant was they had something vaguely resembling a product that vaguely matched what we needed, and with heavy customization they could torture it into doing what we needed. Of course by customizing their “product” we cleverly combined all the downsides of vendor software with all the downsides of custom software. We simultaneously achieved the holy grail of bad ideas: an inflexible vendor package that would have to be forced into doing something it wasn’t designed to do but would also be forked from their main product codebase - guaranteeing sooner or later it would be end-of-lifed once the vendor realized how expensive it was to keep maintaining. We grumbled to each other about what a horrifically bad idea this was, especially considering the vendor’s proven track record in not delivering anything on time.

Because the CTO had a yearly turnover of his direct reports, every status call about the project took some variation of “great idea, boss” even though literally no one involved thought it was even a good idea.
Or even a mediocre idea. It was a bad idea.

The rest of the project required heavy development in-house for all the other pieces, so we had our own challenges keeping us busy. But even so, as project dates through the summer slipped and the vendor promised that any day now their product would be ready to integrate for an October launch date, it became more and more obvious to everyone but the CTO that the project was in trouble. Finally in August the vendor delivered their “product” and we began the death march to integrate with it.

In September we encountered a show stopper bug. The vendor product stored every customer transaction as a json record in a giant json document. So as test data accumulated, performance of the product got slower and slower. Adding a new transaction involved reading the entire json document out of the database, then appending the new record to the end. The vendor claimed they could fix this by indexing the transaction fields, and that seemed to help for a bit until we ran into problem number two.

The database they chose was MongoDB and at the time Mongo had a record limit of 16MB per document. So in October when the conversion team started putting real customer data in, we started hitting the 16MB limit and things took a really interesting turn. A decision was made to hide this limitation from the client and go live a month late, but in the meantime start a skunkworks project to replace the vendor integration. Without telling the vendor, either. So we were simultaneously lying to our clients and our valued technology partners.

Grumpy Old Dev at the time was more Enthusiastic Young Dev and so he set his team to building the replacement. The vendor had roughtly 70 people engaged on the project. Grumpy Young Dev assigned 3 people to replace it. One to design the database, one to build the back end to interface with the database, and one to build the business logic/web services.

The client was told we would have a new version in January for them to test. It would fix the most critical defects they had accepted in the original go-live. But they weren’t told we were rewriting the entire core system. In just under 2 months. When the original project had taken over a year to launch. With only 3 people. Over the holidays. (You see where this is going.)

And so around about middle of December everyone working on the project was told (not asked) to work through the holidays.
Mind you, most of us had already been working 60-80 hour weeks for the past 6 months just to make the original(ish) launch date.
Everyone was burnt out.

At this point, if you’re reading and you’re not a delivery driven technical person, you’re probably thinking this is crazy and it’s time to quit. And you’d be right. But. Many of us who really enjoy software development feel a bit like rock stars. You spend months or even years putting the show together, and then your launch date is like a performance. And you want to hit your launch date. Part of it is like theater people: the show must go on. But you also want to feel like a rock star when all your hard work hits real users for the first time and you feel that thrill of I did that. People like what I did. I overcame the impossible. A software launch is like performing live theater for introverts.

So, by this point it’s almost Christmas. The team has basically built the replacement software in one month of work. There are still some features to hash out. But these are clever developers and they’ve been hitting their marks and I know we are going to make the testing date if they don’t burn out.

So when the CTO comes to me and says holidays are cancelled, I say “OK.”…

And then for one of the proudest moments of my life, thinking back to my dad’s advice about getting the job done in spite of the boss…
I tell my three guys “take the week off. I got this.”
And I dial in every morning for the mandatory death march status call with the CTO and I lie.

• “The team is working hard. Today we hit milestone integration point #73.”
• “The team made good progress yesterday, we finished another web service.”

Every day I showed up and told the big boss that we were hard at work on stuff that we had already completed over the previous month.

The guys came back a week later, refreshed.

And we hit our dates in January, went live with a great launch, and were rock stars for a bit. Maybe more like Herman’s Hermits than The Beatles. But it still felt good.

And that’s the time I lied to the CTO.

• from the trenches
• CTO