How to handle complexity
But first, a confession
Let’s start with a confession. I didn’t love the way this post turned out:
All my posts are full of hand-waving and attempts at focusing on one thing without going down too many it-depends side streets, but this one did too much of it. The situation was contrived to show one possible outcome of miscommunication between Product and Engineering. I got a few emails that essentially said “this just sounds like bad product management”1
And it was bad product management, but it was also bad engineering. If you think hard enough, you’ll find most problems are a mix of both. It was a classic three-step dance of badness:
Product management doesn’t share essential information with engineering.
Engineering thinks they know everything, doesn’t ask enough questions, and designs accordingly.
Both sides are left more entrenched in their views. i.e.
“This is why I don’t tell engineering everything, they overdo it”
“Product is so dumb, they forgot to tell me the most important part”
The post focused on #2 and ignored #1, and for this reason I got a reaction from readers, (who are most #2). So let’s talk about how #1 and #2 combine, because I’d like to express my core theory about how complexity works across Product and Engineering. Let’s talk about #3.
Complexity
I went to college and spent most of my time trying to figure out what to declare as my major. I remember hours and hours of pouring over the course catalog looking for options that seemed interesting. I declared each of these majors (not in this order): English, Business, History, Math, Management Information Systems2, Computer Science, Linguistics.
The classes I remember most fondly are the linguistics classes. They were in a older, prettier part of campus, and the professors seemed like they had already accomplished something and were mysterious, deep, and broadly wise.3 They had either 1) really cool origin stories4 2) great facial hair or 3) weird clothing choices5. One of the lessons from one of those classes is that languages have a roughly consistent level of complexity. If verbs in Spanish are complex, with many edge cases, then there is simplicity elsewhere in the language. The average intelligence of human beings caps the level of complexity of a language.
Complexity in Software
There is also complexity in software, which is capped by how complex the domain, the build, and the organization are. What the software is supposed to do is the primary complexity. Tasked with “build a system to track what happens to a bank as interest rates increase”, the primary complexity is modeling the bank’s activity, figuring out how to report results, the math behind time value of money, etc. Secondary complexity arises within your solutions, like keeping React up to date if you decide to use this in your UI, the complexity of AWS, or your home-grown build tool that keeps breaking.
A deep understanding of the primary complexity drives and minimizes the secondary complexity.
Understanding how banks work helps you design your domain and data model in a way that represents what is needed without extra parts. Choosing Rust over Python, if driven by a real need for speed in the primary layer, reduces complexity over time in the secondary layer of complexity.
Complexity in Software Teams
Your team structure also brings out additional complexity.
Communication is never perfect, so there are many partial solutions to separate and focus it. A larger software team will have a separation of Product (build the right product) and Engineering (build the product right)6. Over time, the primary complexity will be handled by Product, and Engineering will handle the secondary complexity. The relationship between them is what you are both trying to manage. Ideally, you are both seeking to minimize overall complexity.
Secondary complexity is driven by Product’s ability to understand and communicate the primary complexity.
If product stops talking to engineering, secondary complexity grows.
You can state this more simply: Over the long-term, the less you tell the engineers, the more complicated the system they will build.
If you reduce the amount of information that Product gives engineering, what happens? Secondary complexity rises exponentially over the long term. If you are just spoon-feeding them, then tell them your long-term plan. They react to it, and secondary complexity rises in the short term.
Why Product doesn’t communicate everything
This is actually a very common Product Management misstep, in my opinion. I don’t need to tell the engineers things that are coming in the future because it will lead to a more complicated system, and those things might not happen.
Whatever your current project is, if you tell the engineers just about that project, the project is shorter. If you tell the engineers about your potential sales funnel, and that idea you had, and your in-pencil product roadmap, they might do a little bit more work here and there to abstract something away that they think might change, now that they know a bit more about the primary complexity. But your current project will take a little longer, so Product doesn’t do it. It’s classic short vs. long-term reward behavior.
Why Developers Create Complexity
And engineers love complexity; they stand at the fridge drinking it from the carton smiling, and so they are happy to build a more complex design for something that _might_ happen. They (we) can’t help it. It is fun. A new, simple system is more fun than an old, complicated one. A new, complex system is best. Developers like novel things; we work in an industry that grows new skin every 5 years; we have to like change as the baseline price of admission into the role.
We also don’t get to design many new systems and are thus looking for chances to do so. Getting a new requirement for something and fitting it neatly into the current design makes us feel unique and smart. Getting a new requirement and having it go against the grain of the current design makes us feel like we should have seen it coming, especially when we find out the company has been planning it for years and told us about it the same day they told us the deadline.
Bad Products
Product and Engineering keep dancing together, with Product telling Engineering 80% of what they should know and Engineering over-designing that 80% because they are starved for information and too eager to build the right way. Thus, the product starts to drift towards more complexity until it is a bad product.
A bad product is one in which secondary complexity has taken over, and Product and Engineering are both constantly having to manage it. All the cards talk about are Angular updates, or how to fix the core component v2, and if you need to do it before v3 ships, and how to coordinate between teams since those two teams hate each other since their leads stopped dating. New features, or improvements that customers care about are small. Keeping the system from collapsing in on itself is just secondary complexity management.
This is commonly called Technical Debt, which is accurate only up to a point. Sometimes you can label it as other things like Technical Mistakes or Technical Navel-Gazing or Busy Work or Critical Infrastructure Work or Production Support.
The Core Complexity
If what the software needs to do is primary complexity, and how you build it is secondary complexity, then how your organization works, and how well it shares information and pushes in the same direction, is the core complexity. Startups focus on getting primary complexity figured out, and take out loans to get some sort of system up with secondary complexity that they can hopefully get a handle on later. Large companies are only worried about the complexity of coordination - the core complexity.
Conclusion
What to do about this? Work as a team to minimize complexity. Product: be honest about roadmaps. It is perfectly OK to say “we might do this” or “this roadmap will change, its just the current rough direction”. Engineering: talk to Product away from your keyboards. Listen to all the information before jumping to a solution. Think of yourselves as both wearing hard hats - you are both on-site working on the same software product, which should be the right product, built right.
And zero emails that said “This just sounds like over-engineering”. I expected > 0.
I still remember the one-sentence description of this major: “This major prepares students to be Chief Information Officers.” Even then, I was like, “Sure.”
On the other side of campus, where I ended up majoring in computer science, the professors seemed to be working hard towards something and too busy for questions. The entire building smelled of sodium from Ramen cooked in the hallway microwave. There were passive-aggressive New Yorker cartoons on the professor’s doors.
One of my linguistics professors told us he liked to participate in and run psychological experiments. Another was a local (male) theater actor who only played female roles. Another kept dropping details that seemed to indicate he studied in Mexico for four years because he got lost on a mission trip.
On the other side of campus, the professors all seemed to be wearing something they looted from Kohls in an emergency a year ago, having spent the last year hand-washing it from the condensation on the car they were living in. Actually, I have no confirmation, but I think many of them lived in the engineering building. At least one did.
This separation is so fundamental that we have a mental model of even a two-person startup as being a “tech person” and a “business/product person”, like Jobs and Woz et al.