How to handle working software

Undocumented software that has been working is philosophically dangerous

You came into work today pretty happy. You have just passed the two year mark working on this CMS for cat blogs, and you feel like you have a good handle on the current system. You answer more questions than you ask, and you feel that your job is secure. Boy have you learned a lot about cats, and about efficient distributed storage of photos of cats, and about how passionate cat blog owners are. Plus, your cat name suggest tool - where you upload a photo of your cat and it suggests a name based on a deep learning model that the company spent $600k training - has given you some reputation with upper management.

Life is good. You have a domain, you are in control of it, and you know its boundaries.

So then its a bit of surprise when you open up JIRA and get your next support assignment: Fix BI cat content export. Odd, you haven't heard of that before in the last two years. You feel like you have heard of every single component of the system, as when you joined it seemed like the system had trouble with everything - slow blogs, inconsistently slow image loads, timeouts when saving long cat blog entries, cat blog reporting engines, insane cat data audits¹ etc. This cat content export sounds new. Everyone from the team before is either in jail or on another team and actively denying ever working on the Cat Owner Content Office Administrator (COCOA) application.

You search the wiki anyway and find nothing.

You do have the contact information for one of the older engineers that has moved into management and so you send her a nice little direct message asking what the content exporter is, and get an immediate reply:

• she isn't really sure

• oh yes she sort of remembers building this

• she thinks its a scheduled task on one of the servers

• she sends you a search string to find in the code repository: cc --help

You do as she instructs you, pay via Venmo for the trouble, and begin your search. The code does in fact exist, and it has a README.md file that explains that it is a simple command line program cc that exports all cat content to a third party API. It has a backoff retry policy and when it fails it writes to a logfile and pages someone, otherwise it logs that it ran to the Windows application event log. It runs twice a day, and has been running without error everyday for as far as the log goes back.

Until today.

You start to draw some conclusions:

• This program has stopped working and must be fixed.

• By you, today.

• The urgency on the JIRA ticket is VISIT TO VET, the highest level of emergency. This is a critical component to the business; they are asking for every-30-minutes updates.

• You have never heard of it, which means it hasn't broken in two years, and has been running successfully for even longer.

• Actually, the person who trained you never heard of it, because it isn't in the old wiki, and they never mentioned it.

• This means that it hasn't broken or needed changes in 5 years.

• This means that it has survived:

  • The company move from on-prem to the cloud

  • The company move from Azure to AWS

  • All the python, .NET, npm and Linux kernel upgrades

  • One full US presidential term

  • 17 seasons of Survivor, 5 seasons of The Bachelor, the creation of The Golden Bachelor, and the first two US seasons of Love is Blind.

Oh god no. You have discovered an unkillable mythlogical creature, and this creature has come into your hospital and needs life-saving measures.

You have stumbled onto the dark matter of our industry - our ultimate goal and worst nightmare: working software.

Working software is terrifying, like when you hire goth kids for your harvest and all they do it take photos after killing your crops.

Working Software :: Our ultimate goal

• We want to work on new things, not polishing the same thing

• The only way to truly be done with something is to create a system that works without us

• The only way to get back to playing Ghost Recons: Wildlands is for the software you work on to be boringly stable

Working Software :: Our ultimate nightmare

• When it breaks, it likely means that something it relies on has broken, which can be tricky

• If software is running unattended for too long, all the tribal knowledge disappears and you are left with only the documentation which means you are likely left with nothing

• If software can be this drama free, why can’t we seem to produce it consistently?

What to do

First, deal with your fear. Being a software developer is about holding multiple things in your head, and two of them are I can build really strong reliable software while also thinking they really need me because all the software is so fragile. Software that works even as the ground shakes beneath it is frightening because it directly contradicts your worldview, which is that all software needs help and that you are a great helper of software.

Seeing software that is this stable is unnerving to programmers, typically all we work on are things that don't exist or are wrong and we are meant to build them or fix them. To fix something that is some type of perfect already fills programmers with existential dread, like when an expert chess or Go master is suddenly beaten by a computer program and realizes his life has been spent perfecting something that is not meant for humans to perfect, or like hearing a talking dog.

We do not study working software, and its features confuses and amaze us, although we do recognize it. Imagine a cat sitting at a window and seeing a tiger walk by.

For example, try not to think too deeply about the following questions:

• What if the current system you are working on reaches this level of strength? Will you lose your job?

• Could all software work this way, impervious to the ecosystem around it? Why doesn’t it?

• What happened to the man or woman who created such software? Are they immortal, or was the problem easier or they worked on it for a long time?

Think only of the following:

• How can I rescue this beast, and leave it better than I found it, before the truth gets out of its mythical power?

There is only one answer to this question: to get it working again - you better hope the source code is still around - and to tell its story. Leave documentation, patch its wounds, and let it roam back into the forest.

Say a little prayer as it leaves.

1. Testifying before Congress is not as fun as it looks in Succession.