How to do a full rewrite
Fix the system, not your understanding
When you work on a software system that:
has existed for a long time and is making money
is stable but has an architecture that you didn’t come up with
is using older technologies that aren’t as attractive
AND
you are looking to work on one project for the next year or decade
You must rewrite this system.
Rewriting is when you take a working system that is really hard to change, and instead of figuring out how to safely change it (or if it needs change), you rewrite all of its code, starting at the beginning again. You stop all functional changes to the software (I guess you could stop making so many changes after all!) and freeze your understanding of the problem domain and the world in which it lives. Hopefully, after the rewrite, the world still needs your software.
People who don’t work on software have a hard time understanding what a full rewrite is. This is because they are stupid, so try to use metaphors and simple words. Tell them the house was built in the wrong place, and now you must rebuild it elsewhere while they still live there. Tell them that the concrete used is decaying and that new, different concrete needs to be poured. When they ask if the original builders should be consulted, say things like how you heard they were dead. When they ask how we can be sure that the concrete we are using this time will not also decay, tell them that life is a winding road and nobody knows the future. When they ask why the buildings in Rome are still standing, tell them it is time for lunch and that you have to go now.
When you start the full rewrite, you can use the full source code as the documentation. After all, it is the only thing that you can be 100% sure you can trust because it is what is currently running. You can trust the current system because it is the truth, and the truth is what you are trying to replace; the system is correct, so correct that it needs to be corrected. There is no need to find or write new requirements documentation; the source code is a live implementation of all the requirements. Could you imagine how long it would take to rewrite all that documentation? (there are hundreds of pages!) Best to rewrite the code instead.
When you find large chunks of code that seem to serve no purpose - get rid of those immediately! When you find complexity in branches and cases you don’t understand - YAGNI - remove that dead code; now it really is dead. When you aren’t sure how to test the software, let users test it, and let internal folks dogfood the system in addition to their normal jobs - they will love this.
Because your users are used to a complete system, it will be hard to give them an early version of the new system and not have them complain about missing things. You already know what is missing: you made a spreadsheet with all the features. It has nice headings and is well-formatted. Don’t waste time talking to your users, they are going to try to convince you that you should build a different system than the one in the spreadsheet. Just keep going. Once you are done, you can release the software to them so that they can be amazed at how it works like the old system does, but now it is faster, and the code, which they cannot see, is so beautiful.
If you are reading this and thinking, “well it depends,” or “this is a gross oversimplification,” or “what an idiot,” then you might be interested in Additional Commentary for this post.