War is too important to be left to the generals.

• Georges Clemenceau, upon learning of another military failure during World War I

I say this with utmost respect for the engineering profession, to which I have dedicated my professional life. But the idea that “engineers know best” is so foolhardy that I feel I must speak against it, over and over again.

An anecdote from years ago: company I was working for acquired another company. This other company had two distinct systems that were not integrated at all - they might have been from another planet even though the new system was built much later and with specific purpose of integration.

On one side was the original product which was the thing actually serving 99.99% of the customer base but that also had its own set of problems. In particular, it had a significant missing feature that all the other players on the market had and that would significantly increased the number of use cases that we could solve for our customers. I was surprised by this obvious gap and called it out early but I got push back from the subsystem lead - it wasn’t clear to him why we needed this at all and no one was willing to convince him. He wasn’t in my reporting structure and I had no direct influence over the roadmap so it went nowhere.

On the other side was this new system that took a team of 12 engineers full three years to deliver and was not doing well as a product. By the time this system came under me, it had a single (purportedly very important) customer, that the company would not fire even though we were losing money on them hand over fist because <strategy> (read: lack thereof). The workloads that this customer was running on the system immediately struck me as odd: it was built for high throughput I/O but of course it could also do high CPU workloads so it was (ab)used for that. Furthermore, 100% of these workloads were actually being produced by the old system and then shipped to the new system for processing. I couldn’t figure out what was the original reasoning behind it but the dependency was already created and we had to make a plan to migrate this customer before shutting down the system.

The good news was that the feature gap I raised on the old product would fit exactly the workloads that the new system was being used and thus it would allow us to migrate the customer. We would close the feature gap with the competition, we would make the overall system work better (original system would just do everything through its already existing subsystem), we would migrate the single customer of the new and shut it down, freeing the team to pursue more productive endeavors.

Of course I was the only one that saw it that way.

For the lead of the subsystem on the original product, I was forcing on him a new capability that he didn’t like (this is not a joke - he would use words like “disgusting”). And for the team that build the new system, they didn’t want to let go of their system - they spent years building it! And of course, to make things all the more fun, the teams despised each other, were always talking down all the other teams and had no intention of cooperating.

What I would find out only later is that the only reason that single customer was even running on the new system was because the lead of the subsystem team has already refused to build the feature. And the leadership let it. Let me spell this out: the engineering leadership left it up to engineers to decide what they wanted and what they didn’t want to build and rather than leading and optimizing for company goals, the leadership optimized for not upsetting the engineers! And it then became apparent to me that the new team was also in the exact same situation: they were allowed “to engineer” for three years without a customer in sight and then finally a customer was made up for them. It didn’t make any business or engineering sense to use the new system instead of building the feature in the original one. It only made sense in minimizing the number of upset engineers.

And this upsets me to this day. The amount of human years that were wasted because the leadership let the engineering teams build their own little stable-but-barren turfs rather than doing the hard work of coordinating and globally optimizing, is just staggering to me. The original system could have just been augmented with an actual, obvious, feature gap that would have strengthened the product on the market and allowed other companies to build their own products on top (critical for a product that wanted to become a platform!) It was a failure at stewardship but even worse, it was the staggering failure of the folks that were essentially pampered by leadership directly into failure mode (of course it all got shut down)

And so when I am in the situation of leadership, I try very, very hard to optimize globally even when that is upsetting. And when I am in the situation of being lead, I try very, very hard to understand that my local optima is not necessarily the global optima, not only within the most immediate team but also within the larger organization. We are paid to build products, not “to engineer”, and we must not lose that from our site. We must be able to both argue for our position but also be cognizant of the larger context and be able to disagree-but-commit.

Last modified on 2025-01-30