A production server is a terrible place for surprises.
It should not contain a clever shell script only one person understands, a manually compiled library nobody documented, or a configuration tweak applied during an outage three years ago and never written down.
Production should be boring.
Rebuildability is a feature
If a server disappeared, could we recreate it? Not necessarily instantly, but predictably.
The packages, the configuration, the application release, the certificates, the scheduled jobs and the dependencies should all be known. The answer should not depend on remembering what somebody typed into a terminal in 2022.
Automation helps, but the objective is reproducibility. Automation is one way of getting there, not the point in itself.
Snowflakes become archaeological sites
A snowflake server is unique because it has accumulated manual changes over time. It may work perfectly well.
The problem arrives when it has to be upgraded, replaced or investigated. Nobody knows which of its differences are load-bearing, so every change becomes a risk and the organisation slowly becomes afraid of touching it.
That is not stability. It is deferred failure, accruing interest.
Cleverness increases the inheritance cost
An elegant trick can save ten lines of configuration today and cost the next engineer an afternoon later.
Production systems are maintained by people who were not present for the design decisions. Readable configuration, conventional directory layouts and ordinary tools are operational features, not a lack of ambition.
The next person should not have to admire the architecture before they can restart the service.
Documentation should describe the unusual parts
Nobody needs a document explaining that configuration lives in /etc.
Documentation earns its keep exactly where the system departs from expectations: unusual ports, external dependencies, recovery procedures, deployment order, special storage layouts, and the scheduled job the business quietly depends on.
Document the surprises. Better still, remove the unnecessary ones first.
Save interesting for the problem
The software may solve a genuinely complicated business problem, and that is where complexity sometimes belongs. The environment underneath it should make that complexity easier to operate, not add a second puzzle beneath the first.
A good production server is predictable, replaceable and mostly forgettable. We consider that a compliment.