Skip to content

What actually happens when a server dies?

2 min read

Servers fail. Good infrastructure is not designed to prevent that sentence from ever becoming true — it is designed around deciding what happens next.

First, define "dies"

A server can disappear because of power, a motherboard, memory faults, storage, the network, a crashed operating system or somebody's mistake. Those have very different recovery paths.

So monitoring needs to say more than "host unreachable". Hardware telemetry, network state and logs are what narrow it down to a machine, a component, or the software running on it — and that distinction decides everything that follows.

Virtualisation changes the unit of recovery

When workloads are virtual machines, the application is not tied to the physical host that failed. If the storage and cluster design support it, workloads restart elsewhere.

That does not make the failure free. Capacity has to exist on the remaining hosts, the storage has to still be available, and the restart itself takes time.

High availability is engineering, not a checkbox.

Dedicated machines need a different plan

If a workload occupies an entire physical server, recovery may mean replacing hardware or restoring onto a different machine. Which of those is appropriate depends entirely on how quickly the service has to return.

Keeping a hot spare for every dedicated server is expensive. Keeping tested backups and access to replacement capacity is perfectly adequate for a system with a longer recovery objective — provided somebody has decided which situation they are in.

Data usually matters more than compute

A dead CPU is annoying. A lost database is a disaster.

Compute can generally be recreated from configuration and deployment automation. Persistent data needs redundancy and backup proportionate to its importance, which is why recovery design starts with the data rather than the chassis.

Practice changes the answer

Theoretical recovery times are optimistic. Actual recovery includes detecting the failure, working out what happened, obtaining capacity, restoring data, changing networking if necessary, validating the application, and telling people it is safe to come back.

Testing turns that sequence from a set of assumptions into a number.

Servers die. The useful question is whether the system around them knows what to do about it.

Tell us what you are trying to run.

Describe the workload or the system you have in mind and we will tell you what it needs, what it costs, and whether we are the right people for it.