Give it another go. Try a better, more recent tool. Give it subtasks. Give it complex things that you aren’t sure it can figure out1, and when it surprises you, feel the feelings, but then be optimistic. Read what others are doing2, and what they feel about it.

Keep learning.

Don’t turn it off.

Don’t run from it.

Try again.

Guess what. The AI models have also read those books, and can pull them into their context as needed. They are also actively out-reading you. They can generate code very quickly, creating large systems in a way that you didn’t.

Sure, they make a lot of mistakes, and vibe-coding an unstructured 20K LOC is a Very Bad Idea. They need to be guided and taught, and you need to serve as the architect and product manager to keep it on track to meet implicit and explicit requirements. There are times when it will act just like a junior developer, and all your skills in mentorship will pay off. There are times it will generate very slow code, and your years of experience tuning will be very valuable. In many ways these coding tools are serve as a mech suit for someone like you.

While many of your skills carry forward, there are some that don’t, and only one that really matters. Attitude. Because although they are a mech suit for you, they are also a tool for non-programmers.

Your whole life you have been told and it has been reinforced that software engineering is a high art and a difficult science that you are therefore very special. Some of us having taken this to heart more than others. But we have all been treated with special handling by product managers, QA staff, and other members of our team because Only We Can Change the Code.

And now that isn’t quite true anymore, is it?

Much like doctors are taught within a culture where they are viewed all very powerful, better even, than their coworkers, programming has the same sin.

The I’m-special attitude is dangerous, professionally and personally. These tools should enable product managers to build clickable prototypes quickly and easily, and help QA analyze the code looking for bugs in a new way. We can’t be gatekeepers of this, we need to instead help them turn these efforts into what we always do - shippable, supportable high-quality systems. Without the attitude.

system: a set of things working together as parts of a mechanism or an interconnecting network.

Software developers work on systems. For example, if you work for a a loan approval company which determines if a person should be offered a loan, you work on the code that takes in credit history and runs a series of rules to determine if a person should be offered a loan and if so at what rate.

Your job is to make the system work, make it work correctly, and make it work quickly. You spend most days working on one of those things - support, bugs, production incidents are making it work, adding new features is making it work correctly, and any scaling or performance work is making it work quickly. This involves a lot of complexity as each of these three tasks can also break the system or interact with another attribute. For example, adding features can slow down a system, fixing bugs can accidentally make the system no longer work, and of course making a system faster can make it no longer work correctly.

This system lives within a larger system which is how people interact with it. This greater system includes loan officers, the customer, the support team that field questions, etc. The greater system includes your competitors, other tools that your customers use, the market, and many other things that are interconnected with your small technical system.

What system are you really working on?

That’s Not My Job

Traditionally, computer programmers had a clear answer to this question which is, um, I’m working on the system that the computer works on, because I’m a computer programmer and I don’t understand all the nuance of the greater system. That’s not my job, my job is the technical system.

And for the most part you would be correct. But, a misunderstanding of the greater system can make you build the wrong system:

• make it work: it can literally be the wrong system - nobody cares about it.

• make it work corectly: it can work to spec, but the spec be wrong, not make sense to the user, or behave incorrectly given context.

• make it work quickly: you can tune the heck out of a part of the system nobody cares about, while not addressing something that is the real bottleneck.

Traditionally, understanding the greater system was the role of the product manager, or “the business”. It was somebody else’s job to do this, not your problem.

Cost of development

The cost of development is lowering with the AI winter ending. Computer programmer’s jobs are shifting from being the only ones that can understand and code to being managers of tools and agents that can code very quickly. Our jobs are shifting from making something from nothing to be judges of whether or not something has been done correctly by an agent.

And this means that we need to understand the technical system, and the greater system. We are moving “up stack” so to speak, and becoming higher level builders.

It is a wise investment to spend time learning the greater system.

urgent

adjective

1. (of a state or situation) requiring immediate action or attention.

2. (of action or an event) done or arranged in response to a pressing or critical situation.

3. (of a person or their manner) earnest and persistent in response to a pressing situation.

Most of your work is 1 or 2, and you need to be a person who is urgent in the third sense. You need to be perceived to be a #3.

Typically, the ticketing system will provide clues on how urgent something is, but normally, your company has a scale that protects the recording and accountability1 of a decision, so the priority system is something non-sensical like:

Super High Critical - you are expected not to eat or sleep until it is fixed. Updates must be provided every 15 minutes so that people who don’t understand what is happening know what is happening. Your parents are mad.

Critical - Someone is actively yelling at someone else about this, and updates must be provided frequently. You can eat and got to the bathroom on a typical schedule. You parents aren’t mad, they are disappointed.

High - This used to be critical, but was changed to High once a week passed, the yelling stopped, or new louder yelling started. This is the majority of work.

Medium - This will never be worked on, and contains 1% of all known tickets. This is where product managers put tickets that nobody seems to understand, but they are afraid to cancel or close. This is where cards from a previous administration go to retire.

Low - this shouldn’t have even been recorded and will be either not brought into the next ticketing system when it migrates, or closed to help KPIs as part of an audit 3 months before layoffs. In a poorly run technical organization, this is a synonym for the tech debt queue.

How to Make Sure You Work on Top Priority Cards

For all the same reasons, it is hard to figure out which product to build and to determine the urgency of tasks competing for time. But luckily, you are but a small cog in this enormous machine that produces software for cat moms, and you can just ask your boss what your top priority is.

Here’s the thing, though: you don’t have just one boss. Nobody does. Sure, sure, there is one person who tells you that you didn’t get a raise this year because of how pro-dog you were on social media and how you were late on a few key deadlines, but I mean the people you report to are never singular.

For example, your boss has a boss. Sometimes, their priorities override what your boss just told you to do. Sometimes, the boss of the boss of the boss shifts what everyone is working on. You also have other non-work bosses, like your family or your cat at home. It’s the age-old tale: you work your hands to the bone to make a cat blogging platform successful and forget to raise your cat. And you have an internal boss, with brains and ideas and stuff, and sometimes you feel that there are better things you could be working on.

How to Handle Constantly Changing Priorities

So here are a few techniques to handle this:

• At work, ask your official boss what they think your priorities are. Do this every few weeks. Get them to edit a shared to-do list, product backlog, or some other sneaky in-writing format.

• Keep track of everything that you were told to work on. If something was an emergency and now it is less of an emergency but you know it will be an emergency again soon - the smoldering items - keep track of this as well.

• Rarely express your opinion on their view of priorities. Save any disagreement for when you really don’t agree. Be agreeable to change: We are done with that large boat you wanted - oh planes need to land on it too? Ok, we will retrofit it as an aircraft carrier, no problem! We embrace change!!. This is quite difficult to do without building a prejudice towards the other roles in your organization, but it is possible. Just think to yourself: man, there must be a lot of details I don’t know about for them to decide in some meeting at the last minute that planes should land on this freaking boat.

• When you have a really good idea, pitch it to your boss after you have fleshed it out a bit. Since you have been so agreeable in the past, your voicing of a strong opinion now means more (maybe).

How to Communicate That You Are Aware of the Priority of Something

You might have been led to believe that keeping calm in a tense situation is a strength, but this can backfire. If the CEO, CTO, VP of Software Development, Product Manager, Staff Engineer over Scary Changes, Head of Risk, and Executive Chef are all on a production incident with you, and you act casual, you will be viewed as NotSerious, NotProfessional, and NotToBeTrusted, and they might add the Head of HR to the call. Instead, you need to use the fact that people’s nervous systems work in packs to communicate that you get it.

Start the call by FREAKING OUT. Then, calm down and solve the problem. Normally freaking out works in short bursts, because it shows that you are aware of the consequences, and gives a voice to what others are feeling. When you calm down, you show great levels of control and emotional intelligence. Even if you are faking it:

BAD EXAMPLE

(joins production call, Party Rockin is on in the background)

YOU: “Whattup”

INCIDENT MANAGER: “The system is completely down, and nobody can login”

YOU: “Again? OK…. I guess I can take a look”

GOOD EXAMPLE

(joins production call, Party Rockin is on in the background)

YOU: “What’s up”

INCIDENT MANAGER: “The system is completely down, and nobody can login”

YOU:

screaming “OH NO PLEASE NO”

(sound of music being turned off, glass breaking, and a cat maybe being kicked)

more screaming “THIS IS AWFUL, LET’S FIX IT, WITH GOD AS OUR WITNESS”

more calmly “Ok, let’s start by looking at the logs and gathering more information.”

These lessons probably apply to your cat mom blogging platform because they are best practices. I mean, the people you work with are so smart, and they get bored really easily. In tech you need to keep the smartest people busy, and working at your company instead of going to a competitor.

Build it yourself, you have plenty of time

One of the beautiful things about software teams is that even if it takes 10 people to build the first version of an application, it takes 10 people to maintain the second version. It is pretty amazing, and very different than how, say, highways are built. Imagine if the entire construction crew just stood around near highways, ready to fix huge sinkholes that we all expected to appear.

Sometimes, if you let this dynamic play out, you will be on the 8th version of a piece of software and there are still 10 developers1, somehow. Nobody says it outloud, but you run out of things to do. Not that anybody is aware of it, but it happens. Everybody is happy and the new critical features just slow down and you start nitpicking. It starts out slow and low risk - maybe you make a Visual Studio Code extension that helps you with some custom code generation in your particular project. Then you look at the vscode extension library and think - it would be nicer if….

The Absence of Cash Rules Everything Around Me

Next thing you know you are writing TerryScript, which compiles down to ECMA, but only via TerryStudioPlus, which is a custom editor with specific features like JIRA and Slack integration for your particular project, so that you can attach the requirements you received via Slack messages into the comments of the code. In addition, everyone is using a house font that is specifically recognized by the compiler.2

And then suddenly you can’t keep any developers. You can hire super smart ones, and you have a group of super smart ones - I mean one of them built their own language! And another one built their own editor! And they hate each other, just a little bit.

But every time you hire someone new, the pattern is the same: they are very excited by how hard the interview is (some of that extra time was spent coming up with a great interview process), are excited to learn about all the new custom tech, but they leave right at 8 months. Everytime.

Keep Growing, Always Be Hiring

Why are you hiring anyway? There's still a lot to do. The application works great and the developer experience is fantastic, but the application crashes - Segmentation Terry’s Fault3, the team has t-shirts that they think are so funny - after lunch everyday when everyone logs back in.

Terry said something about garbage collection, and his desk is a freaking mess so that sounds right but you aren’t sure what it has to do with anything and you have already talked to the custodians and they still refuse to go into Terry’s office. You have been told that we are weeks away from a real fix for the “daily system siesta” as the users call it. Terry just needs to finish the new version of the runtime, which is compatible with Version 3 of TerryScript, which he says is really close. But only Terry can work on that, so I guess the reason you are hiring is that the business thinks that it takes 10 people to run the team and you only have 9, so you are just trying to find someone to fill the team out. And somebody has to write and maintain The Restarter, which is the watcher program that restarts the primary program when it crashes. The restarter doesn’t work if the primary program crashes explorer.exe, but that’s just a Windows thing which will be fixed once Terry’s friend Tim finishes up Timdows 98 which is due any minute now.

DEVELOPER: We need to go back and fix our architecture

FOUNDER: Architecture? like a building? But we have a 1.0 shipped and making money

DEVELOPER:Yeah, but we aren’t going to be able to ship version 2.0 in this state, it will take so long and will likely not work.

FOUNDER: But you said you knew how to build things well; your whole thing when we hired you was that you build things the right way once.

DEVELOPER:That doesn’t sound like me. Anyway, yeah, well I didn’t expect all these constraints to be thrown at me like they did; it was really hard to ship something for the long-term, when it needs to work in the short-term, and we didn’t know everything yet.

FOUNDER: So again, why do you need to rebuild everything?

DEVELOPER:We don’t, not everything, its just that our system is writing checks that it can’t cash…... You know when you overextend yourself and then have to calm down and rest? Like when you go into debt?

FOUNDER: Oh yeah, I know about that. I have 4 mortgages on my house, it is how we got money to hire you for the Version 1.0.

DEVELOPER:Yeah, well its like that - we need to pay off some of the debt.

FOUNDER: OK, I guess.

Technical Debt Projects Do Not Get Love

If you separate technical debt from other work, then you have narrowed your chance of ever actually working on it.

This picture does a better job explaining part of the dynamic:

The Problem with Technical Debt

There are other, more complex, reasons that technical debt projects won’t get worked on.

First, let’s be clear, projects that are:

• well-understood

• have executive sponsorship

• can be justified with a business case

• come from a customer

Have a higher chance of being worked on versus projects without those attributes. Technical debt projects don’t meet these criteria often because they:

• make it look like you didn’t do a good job building things

• can be hard to explain, and it has to be explained by developers

• start out as grassroots by the people with the most low-level knowledge of how the code works (programmers, support)

• are confusing, because to product management they feel constant:

  • a bad developer always wants to upgrade the system to a newer, more attractive technology , for selfish reasons - work is more interesting and it might be better for your career

  • a good developer always wants to upgrade the system, for valid reasons locally that might not be valid globally (i.e. the system would run better if rewritten in Rust, but it doesn’t make sense for the business to spend the money to do so)

• are not started by customers explicitly1

Add all these factors together, and you’ll find that “technical debt projects” have a harder time getting started and completed.

Technical “Debt”

The term technical debt is a failure of imagination. Sure, it is an accurate way to describe something that happens from time to time:

You:

• don’t have time to do it the Right Way

• need to ship something quickly

• build something that works for now, with the present knowledge that it isn’t the best solution for the future

• were exactly right about the future, so when it arrives you now need to go build it the way you wanted to in the first place

The problem is that once the future arrives, in hindsight, most technical decisions look like technical debt. This is why people call a payroll system that has generated hundreds of millions of dollars in revenue for decades a legacy system, riddled with technical debt.

These are more common situations:

You:

• don’t have time to understand the problem

• need to ship something quickly

• build something that works OK now

• the future arrives and it needs a better solution

You:

• understand the problem well

• need to ship something

• build something that works well now

• the future arrives and it needs a better solution

You:

• understand the problem sorta-kinda-maybe

• need to ship something

• build something you aren’t sure is going to work (like a startup’s product)

• it works enough for the future to arrive

• the future arrives and it needs a better solution

In all cases, in hindsight, the problem and solution seem obvious and like you made an intentional debt decision in the past. Or someone did. But guess what: they didn’t.

The World Around Your System is Changing

“the future arrives and it needs a better solution” can be a long list of things:

• it is now harder to find good COBOL programmers

• it is cost prohibitive to host it the way it was built, or find people to keep it running

• it is no longer legal to run the system the way we do, or for it to make the decisions in the way it does

• the system does not scale - it worked great, then worked fine for awhile, then OK for awhile, then we fixed something, but now there is visible smoke coming out of the back during business hours

• the system works well for one domain, but the business wants it to work for another domain

• One of the system’s big assumptions is wrong:

  • Years can be represented in two digits

  • Flights only depart from their gate once

  • Daylight Savings Time always starts around spring and continues until fall

  • Cars only having one engine at a time

  • A street number in an address will never be 02

• The vendor of one of the technologies has raised their prices so much that you have to rebuild without them or go out of business and change your identity

• One of the key open-source components you rely on has a maintainer that wishes to afford a reasonable lifestyle that includes not being able to see her kitchen from her bathroom

• etc. (this list is essentially infinite)

Some uplifting metaphors

Think of it this way: every relationship in your life is actively decaying. If you don’t make an effort to spend time with your wife or you kids or your friends, they drift away. Everyone you stop calling drifts away faster. You are losing everyone that you don’t fight not to lose. Would you call this relationship debt?

If you own a home, the house is decaying around you and trying to kill or bankrupt you. You need to spend money on simple things like new paint and cleaning, but also on large things like replacing plumbing, resealing your deck, and fixing the driveway where the tree roots have pushed it up. Is this debt?

Likewise, all computer systems are decaying, becoming more irrelevant and less functional as the world around them changes or what is required of them shifts, drifts, expands, or contracts.

This isn’t debt, its entropy.

Your fight is not with debt, it is with chaos. Just like you need to be making an effort towards the relationships that matter in your life and fixing things in your house, you need to constantly be making an effort to keep your systems working as conditions change. This should be part of your normal backlog, and not a separate thing.

Share

Technical Debt Leadership

As technical leaders, we need to treat technical debt as a part of ongoing work, and not separate from it. By isolating it we bring a spotlight to something that is just a reality in our field, and a bit hard to explain to those slightly outside it. You need to constantly upgrade your infrastructure and architecture for the future, with a clear understanding of the present needs. That’s it, that is all there is. And this has better ways of being explained than “technical debt”.

Some examples:

• When you estimate building Version 2.0, you include changes to frameworks and infrastructure that are actively causing issues or will in the near future.

• We need to use this new framework because its hard to make changes now, and we need to be able to derisk the components breaking each other, and we need to ship faster.

• As part of a new project to add features, we need to upgrade our infrastructure to handle the 30% estimated new customers that this feature will bring in, without increasing hosting costs.

• When we fix this bug, we should work on the design to ferret out the assumption that caused this bug, not just fix this instance.

And when you need big things (like a weekend away with your family or a new central air conditional unit) they should be tied to business goals not treated as a separate technical debt project.

Sometimes you need to express this as an objectives that are always true. The justification of these changes is not always new-new functionality, its just keeping on as you are now:

• Have a happy marriage: I’d like to continue to be married, so let’s spend some more time together.

• Live in a safe home: I’d like to sit at the dining room table and not have rain get on me this winter, so let’s fix the roof now.

• Have a system that can be changed: Changing the system for one new business rule should not risk breaking other parts of the system.

• Having a system that customers can depend on: If our reliability decreases another 10% we will lose customers and reputation.

You aren’t paying down a debt, you did not take on debt. You built things the best you could with what you knew and could do then, and now you need to improve the system with what you know now, to support your business goals. That’s it.

You might think that we outgrow biting, but it is still in us and can be manipulated.

This is called the “aspirational model” and is the foundation of a lot of advertising. Remember where it says in the bible: I wish I was a little bit taller, a baller, I wish I had a girl who looked good I would call her1

We aspire to be better: to direct one's hopes or ambitions towards achieving something. Advertisers tap into this when they show us commercials of people who look like professional athletes taking weight loss medicine or people who look like bicycles riding Pelotons, and the implicit message is “if you buy this your life can be like mine is”. Every car commercial says: if you drive this car, then you will park it in an impressive house full of all your dreams come true with fridges with eggnog in June and pantries stocked with all the types of Skittles, and video game consoles with all the versions of Megaman2 which you can play when you get tired of playing basketball in a full indoor gym with a stereo system playing Outkast 24/7.

No car commercial shows regular people driving them, with creaky knees and large heads straining to get out after parking at an Applebee’s on all-you-can-eat-ribs-night.

Something this low-level is hard to not carry around with you, so it is effective in getting you to jump at work as well. But at work you might look up to different people. You might want to do what some tech influencer does, or someone that just gave a talk at that tech conference that work sent you to last year. And your company might be made up of people who wished that they worked at Google. If they worked at Google, then they could tell their mom and dad that they work at Google, and then they could tell their friends that their offspring works at Google and somehow things would be OK with you, on the inside.

Best Practices

So anyway, this is what best practices are, at their worst. They are a series of commercials by people to try to get you to do the Right Thing, in a way in which they profit. At best, they are a series of helpful tips from craftsmen and craftswomen3 that help you learn your chosen profession.

Speaking of craft, sometimes craftspeople like to use their own tools, and they like to pull the right tool for the job out. This doesn’t apply to software. For example, you might have heard about Kubernetes, which is a very strong name for a system that coordinates large groups of machines to do what you want them to do. Think of it as a conductor of an army of paper shredders.

You might think - man you would need to be a really big company to need a way to manage a lot of machines that shred paper. This is wrong. Its a best practice. I mean you probably don’t need a murdered-out gun metal black BMW M3 to drive to your ex-girlfriends’s house, but don’t you sort of want to.

Kubernetes is actually right for everyone, because it is the right way to do things, from the beginning. If you deploy it at your startup, you will be cool, and you can tell your mom and dad (before the startup goes out of business) that you might not be working at Google, but you are doing what they do by deploying 65,000 clusters at scale using AI. Sure, the clusters are basically drones that drop money into paper shredders, but it is a best practice.

Subscribe now

Interesting Links

I have been trying to study AI along with four other things, so today’s links are focused on AI:

The Prompt Engineering Playbook for Programmers - coming from Addy, this is something to pay attention to even more closely. It matches my experience using Claude on a recent project.

My AI Skeptic Friends Are All Nuts - a nice shot, chaser with the last link.

Kelly Vaughn, a worthwhile person to follow in general, has started working at Zapier, and posted their AI competency matrix:

Also: Apple’s widely-discussed and infrequently read recent paper: The Illusion of Thinking

Misc

Better than Random

If you are useful, it doesn’t mean you are valued

As you progress in your career, understanding the difference between being useful and being valued is very important. At first glance, they might look similar because the signals you get are more or less the same: a promotion, a higher than expected bonus, a special stock award. This is why it’s important to dig deeper an…

Read more

a year ago · 82 likes · 8 comments · Better than Random

Unsolicited Book Review

Beautiful Code: Leading Programmers Explain How They Think

I started this book, not kidding, 15+ years ago (published in 2007). I would read a chapter, and then put it down and think about it.

It is a fantastic book, with each chapter written by a different author about a project they thought had some “beauty”:

• Discussion of the NASA dashboard uses for the Mars rover missions (J2EE)

• MapReduce

• NumPy’s C implementation

• JUnit’s design choices

• BioPerl (!), and gene sorting

• Different approaches to writing an XML verifier

• Python’s dictionary implementation in C

• A long discussion about code that was beatiful because it was NOT written

My notes are too extensive to give a good summary, but here are a few:

• Is all cool software in my early career just the result of someone being really good with C? (discussion of NumPy C implementation, Python dictionary, 4+ other chapters)

• NASA on their large J2EE CRM implementation for the Mars rover mission: “In any large application, the key to success is integration, not coding”

• “There is no permanent place in the world for ugly mathematics”

• “As software engineers, we are responsible for our own stress tests. Those that don’t believe this - those have some patrician notion that writing such tests is too coarse for the delicate hands of a Gentleman Engineer - will deliver chronically broken software.”

• “Unfortunately, interactive debuggers as they stand do not support the scientific method. To be sure, debuggers are great tools to poke around and investigate code and its results at will. This is a good thing - but only for skilled programmers who know how to use debuggers systematically. I’d rather see programmers trained in systematic debugging methods than in fancy debugging tools.”

Share Bad Software Advice

A bug is when somebody who is paying money for your software, or not paying money for your software1, uses your software and then:

(what they expect to happen) != (what actually happens)

There are a lot of different scenarios based on the following general role archetypes:

1. The Customer: actual users of the software, the people paying them, the people paying for the software, the people married to these people, people who might use the software in the future, people who keep promising you they will use the software, that the check is in the mail, et al.

2. The Product Manager: the product owner, product manager, project manager, subject matter expert, support staff, or any other non-software-building entity that talks to customers or makes decisions on what you don’t work on.

3. The Developer: technical staff in charge of building and keeping the software running, typically Product Development, or Application Development, which includes Quality Assurance.

Different combinations of agreement between these three roles create confusion. A bug report is meant to solve this confusion.

Sometimes, the Customer and Product Manager agree on the bug, but the developer doesn’t:

• misunderstanding or missed requirement, the developer did not know that the large boat we just built is expected to have planes land on it. The customer (General McStuffins) seems upset that this requirement was missed, and that this misunderstanding led to a poor beta test landing.

Sometimes, only the Customer believes there is a problem:

• It is a major missed requirement, in which the customer expects something because all other competing software has it, but yours does not. The large boat should have large guns, all the other navy aircraft carriers have large guns, its so obvious that they are needed.

• It is a misuse of the software, a training issue, in which the customer puts a piece of bologna into the CD-ROM drive and then files a bug report.

• It is a mistake by the customer in which they tells you that something happened that didn’t. Or the developers says “that can’t happen”. This is referred to as denial of reality or gaslighting.

Sometimes, only the Developer believes there is a problem:

• It is technical debt, which is when the developer thinks it is a problem, but the customer and product manager don’t agree because they don’t know what you are even talking about.

The Product Manager very rarely has a situation in which neither the Developer nor the Customer agrees with them. If this happens, the manager is no longer a manager but a visionary (or very wrong).

If Customer, Product Manager, and Developer all agree:

• There is a fault with the software, meaning that the developer agrees that it shouldn’t be doing that.

  • Each fault leads to multiple failures. Like when you let your cat drive your car: each mailbox that they hit is a failure, born from the fault in letting them drive.

Sometimes, all parties disagree on what is happening. A customer receives an error that the product manager and developer have never seen and cannot resolve in the system. This is called failed to reproduce2.

How Reality and Truth Work

Here is a table3 representing these realities:

As you can see, the people paying you tend to define your reality, unless you actively prove them wrong. Customer-facing folks usually avoid telling customers they are wrong, or soften it “well, bologna is round, so I can see where you are coming from - did it heat it up at all after you put it in the drive?”.

Customers => Product => Development

Decisions flow from left to right, and reality is defined left to right. Sometimes, truth and science flow from right to left, but you must manage it carefully.

What a bug report actually is

I say all of this to explain what a bug report is. A bug report is meant to bring all parties into agreement. It is a written record of a shared reality.

Its primary function is to give enough information to a developer so that they agree on the problem's existence, urgency, and exact details. A good bug report prevents anyone from denying or dismissing the problem and makes it easier to fix it.

Primary sections of a bug report

Version Affected: software doesn’t have versions anymore, ya heard? Well, there are only two versions: Not Shipped and In Production. Make sure you don’t put anything stupid like “Current” in the bug report - remember this bug report might outlive your children.

Evidence: Screenshots, spreadsheets, a transcript of whispered conversations at a bar during a user conference, recordings of emotional customers proving the issue exists, etc.

Comments: This is where people put their fan fiction, including proof that they are human and can do simple pattern recognition:

• “This is the same issue from last Fall”4

• “Could this have anything to do with year end numbers?”5

Emotional Commentary: This is where customer-facing people need to declare hedges and prove that they are working hard on the issue and understand its urgency:

• this is IMPORTANT, need to fix PERMANENTLY and then determine root cause ASAP6

• We need a software system free of issues, we talked about this last fall - MUST-FIX

• There must be a big difference in this issue and what we just tested, because I tested all scenarios that I think will provide cover for my career, and they are all attached with checkmarks next to them this is not on me please don’t let this be on me

Reproduction steps: this is when exact steps to trigger the problem are detailed. This is the most helpful part of the bug report for a developer, and having it greatly increases the chance of a fix happening this year. Think of this as steps to create new evidence. This is rarely filled out, unless the bug report is created by a QA staff member. Most of the time it is filled out with hundreds of words of nonsense that boil down to: “I launch system (screenshot of me launching), and then the problem is so obvious I can’t explain it omg please fix ltr bai”

Reading a bug report

The person who gets assigned the bug is expected to figure out what all the previous programmers overlooked or didn’t see, understand what foreign code is doing, and then fix it, for all time, immediately, dammit. The person who writes the bug report must click some buttons and then hit save.

The quality of the software your organization produces is a function of a few key behaviors, and one of them is high-quality, detailed bug reporting. The average bug report is filed by someone with a cursory examination of an issue who thinks it wise to get it into the hands of a developer as fast as possible, like a hot potato.

Ideally, bug reports detail the issue (what looks wrong, what it should look like) and include steps to reproduce it. This only happens sometimes, like when a stopped clock is right twice a day.

Image 1:

Image 2:

Which image shows a good employee? That’s right, its Image 2, because the employee is wearing a suit within their designated work area.

There are other problems with Image 1:

• The employee is using personal headphones, not company branded audio gear assigned by their local IT support representative

• The employee is working from their home - not an approved work location

• The resource is not displaying any of our corporate values1 in their cord management, reflecting poorly on our department

• The things’s workstation is not locked

• The thing is sleeping

How to tell if your employees are working

Many new managers struggle with determining if their employees are working hard, and they are insecure about it. The way to battle insecurity is to get ahead of it, go on the offensive - assert dominance. Like a stupid, dumb dog that gets scared when there are fireworks - they don’t understand something, get scared, and trash your house. Fireworks are your employees deep technical understanding of software development, you are a dog, and the house is your company’s culture.

Now you might be thinking that this is a silly comparison to make. Fine, let’s look at some additional reasons companies assert control over their employees.

1: They think they are supposed to, based on their training

Many software managers can draw direct lessons from their time running the afternoon shift at a Chick-fil-A2, while others were individual contributors at a software company. You can tell the difference quite clearly in how they think that employees are motivated and rewarded.

CFA manager:

• Free chicken: you motivate employees with childish, delicious rewards.

• Cleaning toilets: you punish employees if they make a mistake.

Software manager:

• Reward: grant well-used autonomy with more autonomy.

• Punishment: discourage bad behavior through reputation and temporary reduction in autonomy.

2: They think they are supposed to, based on their heroes

Many think that the best leadership is military leadership.3 The goals of your cat mom blogging platform is to make money and delight users, not defend a position, but many draw direct comparisons and use what is called command-and-control leadership styles, which work great when you want to get someone to do something that their instincts say is dangerous. It doesn't work as well if you want people to decide what to do and then work together as a group to do it. The military is the best leadership style for their needs; your needs are different.

3. Managing any other way is much harder

Coordinating work amongst knowledge workers and giving them space to figure stuff out is harder than controlling them. Trying to guide them to make the right decision, get them unstuck on a problem, or offer novel solutions requires you to understand the work very well and have additional people skills. Being a software development manager is closer to being a manager of craftsmen than a manager of manual labor. And this is just simply harder.

All that matters it output

You hire a carpenter to work on your house, then make them work how you want them to work, with your tools, on your schedule, and using your methodology. And you have never been a carpenter. Or you are a former carpenter, give them space, and offer up a tool if you see them getting stuck.

Which way of working ends up with the best result?

I want you to picture the following:

office fire marshall

I’ll create some space for you to not see something:

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

You likely thought of someone like this:

If you aren’t familiar with this character, their properties are:

• Good at planning and organization

• Takes everything very seriously; things must be done the Right Way

• Enjoys having power and uses it to full effect to make sure everyone is following The Rules

• she says that she "enjoys being judged" and that she participated in beauty pageants as a child

Fire marshalls are good people that work hard to prevent or react to something that is unlikely to happen, like a fire at your office, or a security incident, or a fault in a computer program setting a bunch of money and careers on fire.

Now I’d like you to imagine this:

fire fighter

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

tick tock

You likely thought of something like this:

• Hero

• Cool outfit

• Mysterious, can’t see face, might carry an axe

• Reacts to something that has happened when others aren’t sure what to do

The Truth

Fire marshals (and inspectors) are made fun of behind their backs, and firefighters get sexy calendars. If you prevent problems from happening or make everyone 10% less likely to create a problem, you are not rewarded like the people who clean up the mess. This is because management acts on incomplete information; they work from afar and cannot hear whispers.1 Only people deeply involved in the day-to-day work know about problems that were prevented, avoided, or solved while they were small.

This is the world you live in, and you need to get used to it.

The Shape of Teams

But on software teams, the team that inspects and builds the house is also the team that fixes it when there is a problem. This would be like a fire inspector, who also shows up if there is a fire. Overall, this is A Good Thing and should lead to fewer fires, as they learn from their oversights and mistakes.

Software teams:

BUILD: write features and bugs.

TEST: find bugs, make sure features work.

SUPPORT: fix faults that make it to production, and recover from failures2.

Different teams have different strengths and weaknesses across these three things. Let’s look at three different teams:

Team 1: Ships a lot of features, weak testing, lots of support

Team 2: Ships less, and tests really well, little support

Team 3: Ships almost nothing, vigorous testing prevents most errors, but support is still a nightmare

Anti-hero

Team 1 has an anti-hero. A flawed hero. A firefighter that maybe starts some fires by their carelessness and their confidence in being able to fix things.

If you have someone on the team who is very good at fixing problems (or more than one), you can have a lot of fires. Like, why are there so many fires. The next team over doesn’t have this many fires. They are terrible fire marshalls, but great firefighters. People play to their strengths subconsciously, even creating the situations in which they can shine, without thinking about it.

This person can be very dangerous to certain teams, and absolutely essential on certain teams. Where you place such people matters a great deal.

Extra Employees

You might be thinking, “but David, what about systems with a lot of faults that require greater support?” or “when is lunch?” or “Team 3 doesn’t make any sense, why is there so much support?”.

In my example from above we assumed that all teams had some baseline that they started with that was equal. But teams support different things, and the team needs to match the task. Existing code has a huge impact here - building a Version 1 of something is very different than building Version 38.

Legacy systems have grumpy employees that you cannot fire. They are called The Existing Code. They affect the shape of your team and how you should design your teams.

Let’s use some terrible metaphors.

• Your existing code is a large nearby planet that provides gravity, affecting all your actions.

• It is geography, changing where you can live because of floodplains.

• It is a starting point that is the 4th bar on our graph:

Not that we have made the code quality poor on all three teams, which team will fail?

• Team 3 has terrible starting code quality, and has a lot of support already, and weak testing. What should it focus on? Improving testing and support to make sure the fixes are real fixes.

• Team 2 is more mature and can ship features, because they have strong testing.

• Team 1 is in a death spiral - they have bad code, ship a lot, and have poor testing. Their support could double overnight with a bad release. Uh-oh. Only one thing can stop a death spiral. Time for a hero.

Building Teams

Personal opinions on how these interact vary, and it is a fascinating thing to stare at and think about.

• What do you need to increase if you move from senior developers to junior ones? More testing, or more senior support?

• What are the relative sizes of these things? If initial code is 10 million lines, and you have two developers supporting it, what do you need more of to keep two developers and not have to hire more people?

• At what rate can you improve these things? Hiring stronger developers: how quickly does this make an impact?

• Which of these items requires the most, highest-quality teamwork?

• If you have a pre-existing, high-quality team, which situation do you drop them into?

• What is the effect of making the goal to reduce support, with no other changes - more testing, less features? More experienced staff?

• What happens when you cover up weak QA with strong development and automated testing?

• How do good documentation and training help?

• If you have to be very weak at one, which one would you choose?

• Great taste, less filling?

One thing you can do immediately in almost any team scenario: make sure to have a balance of fire marshalls (that are rewarded) and firefighters (that are also rewarded). Perhaps they rotate jobs; you need a balance of both. If you are just starting a company, most people are going to have the firefighter personality and be weak planners. But the firefighters need to establish the first broad rules: no grilling on wooden decks, no smoking at gas station pumps, and no flamethrowers on hayrides, etc.

Established companies are going to have more existing control, process, and organization. That is fire inspector/fire marshall territory. Moving from one to the other requires rotating people around so that firefighters don’t work on the systems that become stable thanks to their help.

Take an honest look at what system you are dealing with, and what plans you have for it. Do you need to save this building? Do you need your best builders/testers/fixers here?

There is much more to building a team than deciding who doesn’t like sitting next to each other.

All my posts are full of hand-waving and attempts at focusing on one thing without going down too many it-depends side streets, but this one did too much of it. The situation was contrived to show one possible outcome of miscommunication between Product and Engineering. I got a few emails that essentially said “this just sounds like bad product management”1

And it was bad product management, but it was also bad engineering. If you think hard enough, you’ll find most problems are a mix of both. It was a classic three-step dance of badness:

1. Product management doesn’t share essential information with engineering.

2. Engineering thinks they know everything, doesn’t ask enough questions, and designs accordingly.

3. Both sides are left more entrenched in their views. i.e.

   1. “This is why I don’t tell engineering everything, they overdo it”

   2. “Product is so dumb, they forgot to tell me the most important part”

The post focused on #2 and ignored #1, and for this reason I got a reaction from readers, (who are most #2). So let’s talk about how #1 and #2 combine, because I’d like to express my core theory about how complexity works across Product and Engineering. Let’s talk about #3.

Complexity

I went to college and spent most of my time trying to figure out what to declare as my major. I remember hours and hours of pouring over the course catalog looking for options that seemed interesting. I declared each of these majors (not in this order): English, Business, History, Math, Management Information Systems2, Computer Science, Linguistics.

The classes I remember most fondly are the linguistics classes. They were in a older, prettier part of campus, and the professors seemed like they had already accomplished something and were mysterious, deep, and broadly wise.3 They had either 1) really cool origin stories4 2) great facial hair or 3) weird clothing choices5. One of the lessons from one of those classes is that languages have a roughly consistent level of complexity. If verbs in Spanish are complex, with many edge cases, then there is simplicity elsewhere in the language. The average intelligence of human beings caps the level of complexity of a language.

Share

Complexity in Software

There is also complexity in software, which is capped by how complex the domain, the build, and the organization are. What the software is supposed to do is the primary complexity. Tasked with “build a system to track what happens to a bank as interest rates increase”, the primary complexity is modeling the bank’s activity, figuring out how to report results, the math behind time value of money, etc. Secondary complexity arises within your solutions, like keeping React up to date if you decide to use this in your UI, the complexity of AWS, or your home-grown build tool that keeps breaking.

A deep understanding of the primary complexity drives and minimizes the secondary complexity.

Understanding how banks work helps you design your domain and data model in a way that represents what is needed without extra parts. Choosing Rust over Python, if driven by a real need for speed in the primary layer, reduces complexity over time in the secondary layer of complexity.

Complexity in Software Teams

Your team structure also brings out additional complexity.

Communication is never perfect, so there are many partial solutions to separate and focus it. A larger software team will have a separation of Product (build the right product) and Engineering (build the product right)6. Over time, the primary complexity will be handled by Product, and Engineering will handle the secondary complexity. The relationship between them is what you are both trying to manage. Ideally, you are both seeking to minimize overall complexity.

Secondary complexity is driven by Product’s ability to understand and communicate the primary complexity.

If product stops talking to engineering, secondary complexity grows.

You can state this more simply: Over the long-term, the less you tell the engineers, the more complicated the system they will build.

If you reduce the amount of information that Product gives engineering, what happens? Secondary complexity rises exponentially over the long term. If you are just spoon-feeding them, then tell them your long-term plan. They react to it, and secondary complexity rises in the short term.

Why Product doesn’t communicate everything

This is actually a very common Product Management misstep, in my opinion. I don’t need to tell the engineers things that are coming in the future because it will lead to a more complicated system, and those things might not happen.

Whatever your current project is, if you tell the engineers just about that project, the project is shorter. If you tell the engineers about your potential sales funnel, and that idea you had, and your in-pencil product roadmap, they might do a little bit more work here and there to abstract something away that they think might change, now that they know a bit more about the primary complexity. But your current project will take a little longer, so Product doesn’t do it. It’s classic short vs. long-term reward behavior.

Why Developers Create Complexity

And engineers love complexity; they stand at the fridge drinking it from the carton smiling, and so they are happy to build a more complex design for something that _might_ happen. They (we) can’t help it. It is fun. A new, simple system is more fun than an old, complicated one. A new, complex system is best. Developers like novel things; we work in an industry that grows new skin every 5 years; we have to like change as the baseline price of admission into the role.

We also don’t get to design many new systems and are thus looking for chances to do so. Getting a new requirement for something and fitting it neatly into the current design makes us feel unique and smart. Getting a new requirement and having it go against the grain of the current design makes us feel like we should have seen it coming, especially when we find out the company has been planning it for years and told us about it the same day they told us the deadline.

Bad Products

Product and Engineering keep dancing together, with Product telling Engineering 80% of what they should know and Engineering over-designing that 80% because they are starved for information and too eager to build the right way. Thus, the product starts to drift towards more complexity until it is a bad product.

A bad product is one in which secondary complexity has taken over, and Product and Engineering are both constantly having to manage it. All the cards talk about are Angular updates, or how to fix the core component v2, and if you need to do it before v3 ships, and how to coordinate between teams since those two teams hate each other since their leads stopped dating. New features, or improvements that customers care about are small. Keeping the system from collapsing in on itself is just secondary complexity management.

This is commonly called Technical Debt, which is accurate only up to a point. Sometimes you can label it as other things like Technical Mistakes or Technical Navel-Gazing or Busy Work or Critical Infrastructure Work or Production Support.

The Core Complexity

If what the software needs to do is primary complexity, and how you build it is secondary complexity, then how your organization works, and how well it shares information and pushes in the same direction, is the core complexity. Startups focus on getting primary complexity figured out, and take out loans to get some sort of system up with secondary complexity that they can hopefully get a handle on later. Large companies are only worried about the complexity of coordination - the core complexity.

Conclusion

What to do about this? Work as a team to minimize complexity. Product: be honest about roadmaps. It is perfectly OK to say “we might do this” or “this roadmap will change, its just the current rough direction”. Engineering: talk to Product away from your keyboards. Listen to all the information before jumping to a solution. Think of yourselves as both wearing hard hats - you are both on-site working on the same software product, which should be the right product, built right.

Life is good. You have a domain, you are in control of it, and you know its boundaries.

So then its a bit of surprise when you open up JIRA and get your next support assignment: Fix BI cat content export. Odd, you haven't heard of that before in the last two years. You feel like you have heard of every single component of the system, as when you joined it seemed like the system had trouble with everything - slow blogs, inconsistently slow image loads, timeouts when saving long cat blog entries, cat blog reporting engines, insane cat data audits1 etc. This cat content export sounds new. Everyone from the team before is either in jail or on another team and actively denying ever working on the Cat Owner Content Office Administrator (COCOA) application.

You search the wiki anyway and find nothing.

You do have the contact information for one of the older engineers that has moved into management and so you send her a nice little direct message asking what the content exporter is, and get an immediate reply:

• she isn't really sure

• oh yes she sort of remembers building this

• she thinks its a scheduled task on one of the servers

• she sends you a search string to find in the code repository: cc --help

You do as she instructs you, pay via Venmo for the trouble, and begin your search. The code does in fact exist, and it has a README.md file that explains that it is a simple command line program cc that exports all cat content to a third party API. It has a backoff retry policy and when it fails it writes to a logfile and pages someone, otherwise it logs that it ran to the Windows application event log. It runs twice a day, and has been running without error everyday for as far as the log goes back.

Until today.

You start to draw some conclusions:

• This program has stopped working and must be fixed.

• By you, today.

• The urgency on the JIRA ticket is VISIT TO VET, the highest level of emergency. This is a critical component to the business; they are asking for every-30-minutes updates.

• You have never heard of it, which means it hasn't broken in two years, and has been running successfully for even longer.

• Actually, the person who trained you never heard of it, because it isn't in the old wiki, and they never mentioned it.

• This means that it hasn't broken or needed changes in 5 years.

• This means that it has survived:

  • The company move from on-prem to the cloud

  • The company move from Azure to AWS

  • All the python, .NET, npm and Linux kernel upgrades

  • One full US presidential term

  • 17 seasons of Survivor, 5 seasons of The Bachelor, the creation of The Golden Bachelor, and the first two US seasons of Love is Blind.

Oh god no. You have discovered an unkillable mythlogical creature, and this creature has come into your hospital and needs life-saving measures.

You have stumbled onto the dark matter of our industry - our ultimate goal and worst nightmare: working software.

Working Software :: Our ultimate goal

• We want to work on new things, not polishing the same thing

• The only way to truly be done with something is to create a system that works without us

• The only way to get back to playing Ghost Recons: Wildlands is for the software you work on to be boringly stable

Working Software :: Our ultimate nightmare

• When it breaks, it likely means that something it relies on has broken, which can be tricky

• If software is running unattended for too long, all the tribal knowledge disappears and you are left with only the documentation which means you are likely left with nothing

• If software can be this drama free, why can’t we seem to produce it consistently?

What to do

First, deal with your fear. Being a software developer is about holding multiple things in your head, and two of them are I can build really strong reliable software while also thinking they really need me because all the software is so fragile. Software that works even as the ground shakes beneath it is frightening because it directly contradicts your worldview, which is that all software needs help and that you are a great helper of software.

Seeing software that is this stable is unnerving to programmers, typically all we work on are things that don't exist or are wrong and we are meant to build them or fix them. To fix something that is some type of perfect already fills programmers with existential dread, like when an expert chess or Go master is suddenly beaten by a computer program and realizes his life has been spent perfecting something that is not meant for humans to perfect, or like hearing a talking dog.

We do not study working software, and its features confuses and amaze us, although we do recognize it. Imagine a cat sitting at a window and seeing a tiger walk by.

For example, try not to think too deeply about the following questions:

• What if the current system you are working on reaches this level of strength? Will you lose your job?

• Could all software work this way, impervious to the ecosystem around it? Why doesn’t it?

• What happened to the man or woman who created such software? Are they immortal, or was the problem easier or they worked on it for a long time?

Think only of the following:

• How can I rescue this beast, and leave it better than I found it, before the truth gets out of its mythical power?

There is only one answer to this question: to get it working again - you better hope the source code is still around - and to tell its story. Leave documentation, patch its wounds, and let it roam back into the forest.

Say a little prayer as it leaves.

Subscribe now

When this happens, there is usually an effort to fix it. However, there is a gap, called reality, between the person who sees something odd and a software developer assigned a detailed description of something to fix.

Choosing the victim

The person responsible is chosen using a series of heuristics based on how the organization works:

• Often, the teams are created so that routing bugs is easy. If the UI has a problem, assign it to the UI team.

• Other times, an on-call person is assigned the problem. They review it and then curse.

• Other times, mentioning a word or phrase invokes a person’s name. “It looks like the tax is wrong” => “Didn’t Terry work on that?” => Terry’s phone starts to ring.

• Sometimes, it goes through a few less experienced people before the right person, which takes more time. This is called a mature support organization. These typically have Levels - Level 1 can’t figure it out and then Level 2 gets involved. The levels map to the current heart rate of the reporter of the bug as they wait on a fix.

What To Do If You Are the Responsible Party

Once you find the right person, you have also seen the person most likely to have caused the problem. This might be you.

If you work on the tax invoice engine (v1)2 team now but didn’t 3 years ago, you might be a proxy person for the person who wrote the bug. You are now the most experienced person in this area of the system, so any issues with it are mostly your fault. You might be thinking, “well I’m responsible, but I’m not at fault”, which indicates that you think language matters in the workplace - a common fallacy.

You may have never heard of the part of the system that contains the bug, understand the bug, or be aware of the component that is failing, or what it is supposed to do, or how urgent it is, or that it saying “17” instead of “19” even means in terms of impacts to users, or any other number of logical questions. These do not matter and frankly sound a lot like excuses springing from a bad attitude on your part.

Providing Resolution

The person responsible then set off to determine what happened and work towards a solution. This process is called Debugging. Writing bugs is called Software Development. Half-fixing a problem is called Rebugging. Ignoring a problem in the hopes that it goes away is called Unable to Reproduce. Anything that causes people to stop talking to you about a problem is called Resolution.

After you fix the problem, you might think that some effort should be made to figure out how it happened and how it could have been prevented. This is naive, and a little cute, but allowed.

Finding the fix

We have written about how to actually debug before:

• How to avoid debugging software

• How to ignore issues and keep secrets

Documentation

I don’t know why I put a section here; it is a bad idea to write stuff down.

After this happens, which only happens because you worked very hard, months will go by before real customers use the system. This is a little annoying, because the timeline is roughly:

As you approach death, you work very hard to build a system, under time constraints that seems unreasonable. Why didn’t you take more time to build the system The Right Way during the nothing happens phase? Then you wouldn’t have all this stupid tech debt to worry about for the next year.

Perhaps a smaller example would make it clearer what is happening:

1. A product manager asks you to change something that they have promised their boss will be done by the end of the month.

2. Given the time constraint, you design something that you can complete in a week. It doesn’t feel great, if you had more time you would design something that would scale more effectively.

3. It ships and the product manager is very happy and tells you that they owe you a beer and invite you to a party to celebrate.

4. When you arrive at the party you realize it is a Superbowl party and that your product is going live at the Superbowl.1

5. Suddenly the unspoken requirement of being able to scale seems important, and you watch in horror as the system handles a few thousand requests and then fails. This seems very predictable, and you feel like the system has failed.

6. When you return to the office you immediately get to work on a version which scales better, now that you know what the real requirements are.

7. You complete this in a week and ship it. You are emotional about it as you do this; it is utterly unimaginable to you why the product manager wouldn’t mention the scale requirements. What an idiot.

8. During a retrospective with the product manager, you utter the following phrase: “There is never enough time to do it right, but there always seems to be time to do it over”.

The entire playing field

As a software developer we build systems of all types, and we start to think that we see the entire playing field. We think that other roles are not as important, as we only see their failures. In the first example, the do-or-die demo doesn’t seem so do-or-die to us and so it seems like a fire drill that cost us a lot of sleep.

In the second example, it seems like the product manager is terrible at their job and/or we didn’t ask the right questions. We start to think that all deadlines are stupid, and everyone other than us might also be stupid.

What actually happens when you aren’t involved

The reason developers don’t see the entire playing field is that they only work on one piece of it, and most developers spend their entire careers in a few rooms of a house, never seeing anything else. They are like pharmacists drawing conclusions on doctor behavior based on prescriptions, never seeing times that doctors solved problems without recommending a visit to the pharmacy.

There are important activities that go on outside what you see.

In the first example, the demo was do-or-die, as the conference was the best place to launch the product to:

• get it into the hands of real customers, who all attend that conference

• get immediate feedback on a quick and dirty “version 1”

• to determine if there was a real market for it

• to get it into the minds of buyers of the software product, who think in year long sales cycles

A tech debt-ridden version of the product had to exist on that day, that worked, in order for the nothing happens to kickoff in time, which was when the sales and product people learned that there was a market for it, and set into motion a lot of things that end up being $$$ in our picture. A brochure or fake demo of a product would not have worked, a half-broken demo showed customers that you really are building this thing. A large amount of work happened in the “nothing happens” phase, but none of it was technical commits.

In the second example, the learning is also what matters. The product manager wasn’t sure if the product would work, and the product that you work on is exclusive and high-end, so the product failing to fulfill orders was a requirement, not a bug. The product manager wanted to create buzz, and they wanted to find out if people cared through an expensive version of an A/B experiment. You just weren’t aware of it, and you went off and built a better version of the product just in time. From their perspective all went according to plan, you were just a pawn in the plan. In fact, the system no longer needs to scale, it never did.

But you were in your feelings. Your view is:

“There is never enough time to do it right, but there always seems to be time to do it over”.

Reality is more like:

“If you know exactly what to build, then you know exactly what is right, and you should spend time building that, because otherwise you’ll have to build it right later. Sometimes you don’t know what to build, so you need to throw something out there and learn what is right, and then go build that.”

That quote is not as catchy, but I’ll offer another version from the developer world:

Plan to throw one away; you will, anyhow.

Sometimes, a product manager intentionally tasks you to build a crappy v1, because a crappy v1 is exactly what is required to ever have a v2 that makes money. Building a perfect v2 is simply not possible, but feels like it is only in hindsight.

Do not assume that you see the entire playing field or that those in other roles are idiots. We are all idiots, trying to coordinate towards a successful business.

Smuggling Documentation into Work

As someone who is sometimes mildly annoyed when I document things and then can’t get others to do so, I found a story about people trying to restore missing documentation to a company they used to work for - reverse corporate espionage - funny.

Success sometimes limits your options

There is no separate term for the type of technical debt that you create between you and another company with a bad interface or accidental behavior. Oh, actually there sort of is: Hyrum’s Law.

Something, something, remote work good|bad

Not that often that you find yourself referencing a single comment on Hacker News, but I thought this one was quite good on the remote work good|bad debate: (emphasis mine)

Yes, as a well-paid, introverted, technical contributor who is internally motivated by their craft, with the luxury to afford good working space and at a moment in one's life where home haunts feel secure and supportive, you can't beat it. Like any tradesman in history keeping up their own shop, it's really quite empowering. I've been doing it for pretty much all of a very long career.

But it's worth keeping in mind that there are a lot of implied constraints there, and that the industries that drive the society we live in often rely on making the best of people who can't meet all those constraints.

There are people whose jobs need them work with other people dynamically, extroverts who need to be around others with a common aim to thrive, people with compensation to meager to carve out an effective home office, people who need on-site facilities, people with chaotic or draining home lives, etc

It's very easy to talk about why remote work can be extremely rewarding for some, but the big picture of a business or an industry needs to balance a whole bunch of other concerns -- some intrinsic and some simply inertial.

It's just not a single, simple topic where we can project our own experience as if it was universal.

I think this is a great, personal, take, and find the remote work debate difficult because it is deeply personal. For years I was caring for an illness in my family and had to work from home, but never told anyone at work. Some people cannot work from home safely, do not have the space, and cannot stay alone and be mentally healthy. Others just want to maximize their rent. This is why we speak past each other so often, we are all holding our cards back.

Speaking of remote work, I’ve open-sourced by book: Navigating Remote Work, and made it free for all. Its a personal guidebook for developing the skill and systems to be successful working remotely, and is based on 17 years of mistakes.

Advantages of Incompetent Management

The main disadvantage of incompetent management is its definitional inability to set and achieve key goals, which can endanger the survival of the organization. Incompetent management can only thrive in situations where basic survival and even growth are somehow taken care of, and any major changes in that situation create an existential risk.

Well, that’s one way to look at it.

Unsolicited Book Review

One of my strategies to stop myself from going too far down the AI rabbithole was to read about:

• AI ethics, or lack thereof

• Ways that AI will ruin the world

Before learning how learn why not and oh no god no. The hope was that this would stop me from pursuing a masters in AI. While this strategy was not successful, it did expose me to a few scary books about AI that I’d recommend:

• Code Dependent: Living in the Shadow of AI : good catalog of areas in which AI can help, and areas where it is currently hurting and changing society, throughout the world.

• You Look Like a Thing and I Love You : sort of silly book about how AI “solves” things and can be very wrong and funny.

• Not with a Bug, But with a Sticker : adversarial approaches to AI, and ways you can make it fail that are quite unexpected.

• Superintelligence : a dense book that has “WTF” written in the margins dozens of places if you buy it used.

Share Bad Software Advice

The card describes a bug that your system is experiencing, with an attached video clearly showing the issue and exact steps to reproduce it. You try to open it, without success. It looks like your company's tool for video capture has implemented single sign-on, but it says you don’t have access and that your system administrator has been emailed. You wonder who that is; over the years you have met multiple people with beards and strong opinions about rare and unimportant things—it could be any of them.

You take a moment to figure out what to do. Your thought process is roughly:

• Wondering if there is a system administrator

• Wondering if he or she ever checks their email for this system

• Wondering how much trouble watching this video is going to be

When you woke up this morning, you didn’t know it, but you were not going to arrive at work today. You arrived at an IT Escape Room, and your mission today is to get out of it before the end of the day (EOD).2

Organization

The tech company you work for is structured, like many companies, into different groups that interact with each other. Some would say in a power dance, or checks and balances type fashion:

1. Application Development, Product Development, Development, Engineering, or Innovation Lab, or The Smelly Room: the people that build the software.

2. IT, Security, DevOps, Infrastructure, Computing Enablement, The Other Smelly Room: the people that must protect the company from #1.

#1 and #2 must live in harmony. By harmony, I mean a culture of mutual disrespect and arrogance expressed in meetings through incredibly stupid nitpicking and over chat via passive aggressive friendly messages.

This tension is natural, based on why the two groups exist:

• #1, AppDev, has been tasked with building things as quickly and insecurely as possible, in order to make money.

• #2, IT, has been tasked with keeping the company secure and predictable, no matter the effect on profits. After all, the best way to make an animal predictable is to cage it.

As your company grows, the Nash equilibrium3 pushes the tension between these two groups towards each other until a frontline is established that can be stomached from both sides. This is sometimes a technical interface and other times a combination of technical and human interface:

• All changes must be pull requests and submitted by Tuesday at UTC midnight.

• All changes must be pull requests, and The Terrible Terry Committee (TTC)4 has to review them and Terry and his friends decide if they pass or not.

At some point in the past the TTC has decided that the video capture tool Must Not Be Given to Every Developer (For It Costs Too Much | They Will Misuse It For They Are Not To Be Trusted).

How to Escape and Watch the Video

There are ways around it, however. There are two primary strategies.

Strategy A

1. Document your lack of access and why you need it for your job, and submit a request for permission to what you think is the right place.

2. During the next 7 to 9 daily standups, report that the bug cannot be fixed, because you are unable to watch the video.

3. Wait for someone to get annoyed enough to give you permission, or start a battle to gain access long-term for all people in your role. Multiple things can happen from this step. This might start a war for territory, might be granted immediately, or might kickoff a project that creates a system wherein you can “checkout” access for a temporary period of time to view videos, while a 3rd party auditor takes a video of you which is kept for 7 years.

Strategy B

1. Call up your coworker who is higher up in the food chain (Solutions Architect or Vice President or Nephew Of Big Boss)

2. Have them watch the video while you are on a video call

3. Possibly screen-capture it for later

4. Figure out and fix the bug

5. Wait to get in trouble for this

There is a controversial third strategy, which I will only call Strategy C, because I can’t think of a better name for it, which is:

• Take the card (NMICAB) and flip it to unassigned as soon as you realize you can’t watch the video, and grab another card. If questioned, say that you assigned it to yourself by accident.

You cannot lose if you do not play.

1. Normalize “I don’t know, ask your mom” as a successful answer.

2. Say little, because you are eating a hotdog and are in the middle where it’s super messy.

3. A person’s favorite sound is their name, according to some dead guy, so remember it and yell it out in crowds.

4. Praise publically, criticize privately, gossip regularly.

5. Always return a shopping cart, especially if you are not in a grocery store - I mean you should not have even brought that here.

6. Keep secrets unless they are just too good.

7. True leadership is choosing to help people grow, truest leadership is yelling at them to grow faster.

8. Be the last to eat - let others go first, then you eat what is left and complain about it a lot.

9. Keep your phone in your pocket, and also other people’s phones, so that they won’t get distracted while you are speaking.

10. Pause in speaking and make eye contact until you remember what in the hell you were just talking about I mean where are we even.

11. Hold the door open for others, unless there is a fire.

12. Never underestimate that which can be overestimated.

13. Under-promise and under-deliver, so that people know what to expect of you, and you can be in the running for Most Improved.

14. Take note of important dates, like 9/11.

15. Be punctual to show respect for your own time.

16. Respect different perspectives unless they are the stupid ones.

17. Stay loyal to your friends and family, unless you are sure there are better ones.

18. Do the work when nobody is watching, then stop when they start watching because it is embarrassing.

19. Say thank you and please, even when being attacked by a goose. Manners matter.

20. Always leave things better than you found them, even if it really hurts the bottom line of your demolition business.

If you work in support, customers expect a resolution. If you work in research or product development, your customers expect you to build something to solve their problems. As an engineer, the fastest way to end a conversation and get back to playing Ghost Recon: Wildlands is to offer a solution, no matter how good.

Problem Definition

Documenting a problem is Very Hard, and people don’t write things down. You will often hear about a problem during a meeting and need to ask questions to help them narrow and better define it. They will often devise a half-backed solution, which they will tell you in the first 5 minutes of the conversation.

Here are some examples:

1. When we run the morning reports, they time out on Mondays and a few times on other days. Solution: We should increase the server capacity on Mondays to solve it.

2. Some customers have invalid configuration data, so about once a month, we load those customers in our web UX and hit “Save” on their profiles to fix it. Solution: When a customer hits an invalid configuration, we should auto-save their account to fix it.

3. When people onboard new accounts, they never know that the name of their tenant is case-sensitive. So many times, we end up creating two tenants, one called TenantX and one called tenant. Solution: To handle this operationally, we should create a tenent-merge script that Ops can run when this happens.

What is wrong with these solutions? Nothing.1 These solutions are great.

Garbage-in, garbage out

It isn’t your job to help a customer define the problem; you are just the solution provider, right? If they ask for something, shouldn’t you give it to them?

In your role as an engineer, you don’t always see the process of figuring out the problem. This is more of an art than a science, so it’s hard to learn about it in traditional ways. There is no manual for analyzing requirements, interviewing customers, or building creative ways to get information on how external processes work. It is a crossroads between formal system analysis, psychology, interview technique, high-level communication, empathy, and systems theory. It’s messy. Best avoid it at all costs.

Problems without Solutions

If a half-baked solution isn’t present2, you need to create one. As someone is speaking, ideally while mid-sentence, you should offer a half-baked solution. Think of yourself as on a trivia show in which there is no penalty for wrong answers but a great reward for answering quickly. Like the trivia system at Chili’s. You are paid to solve, so solve it and solve it quickly.

Besides, understanding the problem is probably a bunch of extra information that won’t matter about the types of people that use your system, how they communicate, what their other problems are, or what they expect. Certainly, knowing about the industry and what other systems do isn’t relevant to your system and where the industry might be going. Focus! All that extra context won’t help you. It’s information overload!

First, Do not Offend

Besides, they came to you with questions, so asking more questions is rude. They don’t seem to like it when you ask questions anyway; it feels like maybe you are stalling or don’t have any solutions. I mean, even if it might be simpler, wouldn’t it be a little embarrassing if you asked many questions and realized that a solution to their problem already exists, and it isn’t something you don’t get to build? And then what if you suggested that they use something like Google Sheets and some off-the-shelf API to do it? I mean, wouldn’t that mean you are a joke? A software engineer that isn’t engineering any software by writing it fresh3?

If you don’t fully understand the problem, you can offer a few solutions to make it better but not solve it, and then they will have to come back to you for more solve credits. See, it is like a video game in which you need to get as many hit points as possible, not as many kills as possible. Get some knocks, but few kills. You should cultivate a farm of easy-to-solve problems and develop a reliance on yourself as the only one who can provide solutions rather than truly solving the problems.

If you miss important information about the problem, the customer will probably ask again, and you can resolve it then. That way, you get credit for two solutions, and maybe you can get your Solutions Architect merit badge in time for summer.

But the manager doesn’t know this.

How to tell who your best programmer is is written not from a coworkers perspective but from that of a manager. It speaks to the struggle that people have in determining which experts to trust. When developers disagree, and you are their boss, and you have to determine which one of them is right - how do you do this?1 You need to figure out who to promote or give feedback to and you are not in the trenches watching the work happen, how do you do this?

Human nature dictates that you tend to do this based on several things that can easily become anti-patterns:

• Conflict avoidance: this normally lets jerks win arguments

• Deferring to power: the longer someone has worked somewhere, and the more secrets they know, the worse their behavior can become because people are afraid to cross them

• Strong track record winning arguments: whenever there is a disagreement on the team, a certain person ends up being perceived as right, and this has to count for something, right? But they may have “won” because they started the argument to show how smart they are; the argument might have been a waste of time, and everyone might have just deferred to them to avoid it continuing to waste time

• Loyalty: the developer has worked here as a Senior Software Engineer for 25 years! They must be great, everyone else leaves within a year! This pattern seems obvious when pushed to the extreme like this, but there are cases where a programmer has spent as much time defending their castle as building it well, so this builds them (false) job security and makes it harder for them to leave or for others to help.

• Visibility leads to familiarity: you always see problem programmers; they somehow are always at the center of discussion on some problem (maybe they caused it and are most familiar with it). It can be hard to tell if someone is arriving as a hero or a suspect, you just know they are always around during situations

• Genius stereotype problems: society reinforces the stereotype of the hard-to-work-with loner genius, and so at times we categorize loners and people that are hard to work with as geniuses when they aren’t.

This anti-pattern is about how people who are not easy to work with can sometimes be perceived as smarter than they are because of these default behaviors.

In an ideal world, programmers are instead judged on how well they help the team. In The Worst Programmer I Know, the case is laid out that measuring individual productivity is perilous because it misses those developers who elevate the team’s effectiveness. Mentorship and helpfulness do not always show up to a manager but are deeply felt by peers. If a developer is generally helpful and wants the work done, no matter who gets credit, they should be rewarded. And if they yell, bluster, and hold tight to areas of the system in an unhealthy way, they shouldn’t be.