Every failed system we have been called in to rescue had backups. Without exception. The backups ran nightly, they reported success, and somebody had been watching the green ticks for years.
A backup exists for one reason: to restore data after something has gone wrong. Until that path has been exercised, what you have is a collection of files and a hopeful theory about them.
Backup success and recovery success are different things
Backup software can truthfully report success and still leave you with a recovery problem. The ways it happens are depressingly consistent.
- The database dump is mid-transaction. It restores, it looks fine, and the referential integrity is quietly wrong in a way you discover three weeks later.
- The files are there and the database is not, or the two are from different nights, which for most applications is the same as having neither.
- Nobody has the credentials for the storage the backups go to, because the person who set it up has left.
- The encryption keys were never included, so the archive is intact and unreadable.
- Retention quietly discarded the version you actually need.
- It works, and it takes eleven hours, which nobody knew until the day it mattered.
None of those necessarily makes the backup job fail.
Test the restore you actually expect to perform
A restore test should resemble reality. For a virtual machine, that means restoring the whole machine into an isolated network and booting it. For a database, restoring to a clean instance and running application-level checks. For a file server, recovering both recent and old files and verifying permissions as well as contents.
For a complete environment, test the dependencies too. A perfectly restored application server is not much use if its database, certificates, DNS or identity provider are missing.
RPO and RTO are not paperwork
Two numbers make the whole discussion concrete.
Recovery Point Objective asks how much data you can afford to lose. If the answer is one hour, a nightly backup is obviously insufficient.
Recovery Time Objective asks how long the service can be unavailable. If the business needs it back in two hours, a process that begins with ordering replacement hardware is not a two-hour process.
These do not have to become a consultancy exercise. Even rough answers expose the mismatches immediately.
Keep failure domains separate
A backup stored beside the system it protects is convenient, and genuinely useful for fast restores. It is also exposed to most of the same events.
A storage controller can fail. Credentials can be compromised. Ransomware encrypts whatever it can reach. A fire is extremely democratic about which rack a disk belongs to.
So: local copies for speed, remote copies for site failure, and preferably one copy that normal production credentials cannot rewrite or delete.
What we do instead
We restore. Not the whole estate every night, but on a schedule, into a scratch environment, with the result actually checked — does the application boot, does a known record exist, does a login work, and how long did the whole thing take.
It is unglamorous, and it is the only part of a backup strategy that produces evidence. Everything before it produces confidence, which is not the same thing.
Two questions to ask your current provider
If you are hosted elsewhere, these two will tell you most of what you need to know.
When did you last restore one of my backups, and how long did it take? If the answer is a date and a duration, you are in good hands. If it is a description of the backup schedule, they have answered a different question.
If my account with you were compromised tomorrow, would the attacker be able to delete the backups? The correct answer is no, and it requires having thought about it beforehand.
A green "backup completed" icon tells you that software copied something. A successful restore tells you that you have a backup.