Nearly every organization has a Recovery Time Objective. Few have a number that was ever actually measured. The gap between the two usually surfaces at the worst possible moment.
Every business continuity plan (BCP), the document that defines how an organization keeps operating during an outage or disaster, includes an RTO: the Recovery Time Objective, or how long a system is allowed to stay down before it must be back in operation. Management approves four hours for core systems, a full day for email, two days for the file server, and those numbers make their way into the document, the presentation, and the annual audit. The simple question almost nobody asks is: what is that number actually based on?
In most cases, the answer is a calculation. Data volume divided by network bandwidth, or the read speed listed on the storage vendor’s spec sheet, or a professional estimate from whoever built the environment. These are all reasonable numbers, but none of them is a measurement. And between a calculation and a measurement, the gap can be an order of magnitude.
A case from the field
In a business continuity project we recently supported for a multi-site organization, the entire server environment ran in a provider’s cloud and was backed up to a dedicated storage appliance at the organization’s offices. The planning estimate for recovery speed was 120 to 180 gigabytes per hour, a figure derived from bandwidth and vendor specifications, and it formed the basis for a four-hour target on core systems.
As part of the routine test restores we run, we took the opportunity to measure it directly. A full restore of a single server, over the actual path that would be used in a real incident, delivered roughly eight gigabytes per hour. The network was almost entirely free, and writing to the target was not the bottleneck. The bottleneck was reading from the backup storage: a long chain of incremental backups on mechanical disks, which the software had to assemble into a complete image, an operation that is inherently slow on that type of storage. At that rate, the four-hour target became a target measured in days.
It’s worth stressing: this is just one example. The configuration was entirely reasonable given the information available at planning time, and the measurement did not expose a product flaw or a misconfiguration, but a characteristic of that storage type. The point isn’t where the gap turned out to be. The point is that without measuring, nobody would have known it was there at all.
The gap can hide anywhere
By the same logic, the measurement could just as easily have exposed a gap somewhere else entirely. Bandwidth is the usual first suspect, and sometimes it really is the culprit: a connection that looks wide on paper turns out to be shared between backup traffic, ongoing operations, and the restore itself. Parallel restores are another common source: the assumption that running several restores at once shortens the total time only holds if the underlying infrastructure can actually serve multiple streams, and in many cases each additional stream slows the others down. Even after the data has already been transferred, the application layer can still surprise you: a database that needs hours of consistency checking before it’s ready to operate, an application that reads slowly from newly provisioned storage, or a server that boots successfully but doesn’t actually function because of a silent file system defect that the backup faithfully copied from the source.
Every one of these gaps has a fix, and most of them are relatively inexpensive. But there is no way to know which one applies to your organization without actually running the restore and measuring it.
How to measure correctly
A good measurement of recovery speed isn’t a theoretical exercise, it’s a routine operation. Here’s what we’ve found works.
Measure the real path: from backup storage, across the network, to the destination where the server will actually come up on the day of an incident. A restore within the same local network will give you a nice number that has nothing to do with reality.
Measure several scenarios, not one: restoring a single file folder, a full restore of one server, and two servers restored in parallel. Each scenario exposes a different bottleneck, and the relationship between them tells you more than any single figure.
Record not just the time but the reason: most modern backup systems show you where the restore time was actually spent, reading from the source, transferring, encrypting, or writing to the destination. That detail is the difference between “it’s slow” and “we know what to fix.”
Confirm the server actually works, not just that the data arrived: a restored server that won’t boot, or an application that can’t connect to its own database, is a failed restore, even if the system reported success.
And repeat it: quarterly, for every critical server, as part of routine maintenance. Environments change, backup chains grow longer, and a number measured a year ago is no longer a number you can rely on.
What to do with the result
Once the measurement is done, there are three options, and all of them are legitimate. You can shorten the recovery time: shorten the backup chain, add faster storage, keep a second copy near the target, or maintain pre-provisioned standby machines. You can update the target so it reflects reality, and build manual procedures around it that let the business keep functioning during the wait. Or you can combine both approaches, deciding system by system based on cost and impact.
And under any of these options, there is one recommendation that isn’t optional: plan the restore sequence in advance so core operations come back first. Not all servers are equal, and restoring in whatever order is most convenient for the technician is the surest way to bring back first the servers nobody is actually waiting for. The restore order should come from a business impact analysis: first the systems without which there is no sales, delivery, or production, along with the infrastructure they depend on, such as authentication servers, and only then everything else. Write that order into the procedure, test it in an exercise, and measure it separately: the recovery time of the core is the number management actually cares about, even while the full restore is still far from complete.
What isn’t legitimate is leaving a target in the document that nobody ever measured. A target like that doesn’t protect the business, it only reassures whoever reads the document. Management deserves to know the real recovery time, and to make the decision about cost versus time with eyes open.
A good business continuity plan isn’t judged by the quality of the document but by what happens when you actually run it. The only way to know what will happen is to run it first, quietly, on an ordinary day, with a stopwatch in hand.
If this is an open question for your organization, we’d be glad to hear how you’re handling it. Feel free to reach out to us.
