Resilience on AWS: From Fire-Drilled Runbooks to Immutable, Portable Recovery
How three overlapping efforts — live disaster-recovery drills, a ransomware-resistant backup design, and a deadline-driven AWS-native build — became one program, and ended with bootable images that can restore anywhere.
Executive summary
Most disaster-recovery plans are written once, reviewed occasionally, and never exercised end to end until they are needed for real. This paper follows one AWS-hosted enterprise application through the full arc of making recovery real: a three-scenario runbook hardened by live fire drills; a drill-exposed dependency on unmanaged legacy backup servers; a four-option architecture decision that produced a ransomware-resistant hybrid design; a deadline-driven, AWS-native build that proved the first backup inside 24 hours; and a final layer — bootable VHDX images created with Disk2vhd (installed on the Application, EDI, and SQL servers so it rides along in every snapshot), hosted in S3 and synced to a Synology NAS — so recovery does not depend on AWS being reachable at all.
The throughline: the architecturally right answer is not always the immediately shippable one. The skill is knowing which a given moment calls for, documenting the gap honestly, and building the bridge back to the better answer.
1. Three scenarios, one runbook
Recovery planning was organized around three distinct failures, each with its own procedure and owner.
| Scenario | What has happened | Recovery approach |
|---|---|---|
| A — Data corruption | Servers are healthy, but a database is corrupted or overwritten. | Restore from the latest full backup plus transaction logs, to a point in time. |
| B — Total server loss | An application, database, or integration server is unrecoverable. | Convert the latest snapshot to a bootable image, launch a replacement, reattach storage, cut traffic over. |
| C — Regional outage | The entire primary region is unavailable. | Same as B, from the cross-region snapshot copy. |
For a multi-tier application the pattern repeats in sequence — database first, then application tier, then integration components — with each tier's connection strings updated before traffic is cut over.
2. What the fire drills caught that design review missed
The runbook went through five documented revisions. Each substantive addition came from something a live drill exposed:
- An undocumented backup dependency. Snapshots protected every boot volume, but the SQL backup files lived on legacy file servers outside that policy. The database server restored cleanly; the transaction-log chain did not, because the backup servers were unavailable in the recovered network context. This finding drove everything that follows.
- Configuration baked into images. Legacy mapped drives from a pre-cloud architecture appeared as broken mounts in the recovered environment. The runbook now documents how to tell a stale mapping from a genuinely failed storage mount, so no recovery time is lost chasing the wrong one.
- Storage policy defaults. Freshly attached recovery volumes came back "offline" under a default OS policy — a one-click fix once known, a hard stop mid-incident if not.
- Startup-order dependencies. The application threw a misleading file-path error if started before its volumes were mounted; the runbook now carries an explicit order-of-operations callout.
- Drill-only network edge cases. A missing HTTPS binding on the temporary hostname and default-closed diagnostic-ping rules appear in every drill but not in a real cutover, so they earned their own checklist.
Every finding was captured as a dated, versioned runbook addition tied to the test that surfaced it — turning the document into something that gets more trustworthy each time it is exercised.
3. Closing the backup gap: the architecture decision
The existing design had nightly full SQL backups and 30-minute log backups on legacy file servers reached by mapped drives, with snapshot coverage everywhere else. The exposures were concrete: no verified coverage of the backup servers, no air-gap or immutability (ransomware or a compromised credential could delete the database and both backup copies in one blast radius), and an unbounded worst-case RPO. A separate, earlier DR document also stated in writing that full AWS-account compromise was out of scope. Four options were weighed:
| Option | Strength | Trade-off |
|---|---|---|
| A — Block-storage snapshots only | Fastest recovery; no new tooling. | Same account/region blast radius — no true air-gap. |
| B — Object storage only | High durability, object-lock immutability, cheap cross-region copies. | Slower restore (download first); new IAM/CLI tooling. |
| C — Managed file service | Preserves drive-letter conventions. | Meaningfully higher cost; a new service to learn. |
| D — Managed backup service | Policy-driven; native cross-account support. | New to the team; cost similar to C; added IAM complexity. |
The recommendation: a hybrid
No single option covered the risk surface at reasonable cost, so the design combined A and B:
- Primary path — fast recovery. A dedicated backup volume on the database server, added to the snapshot policy on a four-hour cadence with cross-region copies, restored with the same documented workflow as every other volume. Target: minutes, not hours.
- Secondary path — ransomware-resistant object storage. A bucket with versioning and compliance-mode object lock, populated by a sync job every fifteen minutes. Even an attacker with full account credentials cannot destroy the copy within the lock period.
- Least privilege and monitoring. The sync identity can write and list on one bucket — no delete, no ability to shorten the lock — and an alarm fires if no new file arrives within 24 hours, catching silent failures.
The block-storage path handles the common failures in minutes. The object-storage path exists for the ones it cannot — account compromise, ransomware, a deleted snapshot policy. Together they cost less than either managed alternative and use only services the team already ran daily.
4. Shipping under deadline: three paths, one decision trail
A hard deadline arrived mid-engagement: one manual backup proven within 24 hours and every target covered by the next business day, plus a plain-S3 copy that could be mounted as a drive letter elsewhere. The requirements also called for GUI-taught backup and restore any team member could run, automation that could grow from the manual process, and a capacity alert. Three paths were documented in sequence:
| Path | What it is | Outcome |
|---|---|---|
| A — Cross-account vault lock | A second AWS account holding a compliance-mode WORM vault, fed by a copy action. | The textbook answer, but a half-day to full day of one-time account, KMS, and trust setup — setup time, not the concept, was the blocker. |
| B — Third-party vendor | Seven vendors scored on the same requirements; a free-tier pilot of one. | Set aside after two findings: the free tier lacked granular point-in-time SQL restore, and the backup server itself was a new machine to patch and secure. |
| C — AWS-native | AWS Backup recovery points plus a plain S3 copy synced from each live server. | Shipped using only existing infrastructure and permissions. |
The original ask for backups "air-gapped… outside of AWS" was genuinely ambiguous — logically separated from an account compromise, or literally non-AWS. Rather than guess, the proposal recorded the ambiguity, recommended the first reading with reasoning, and scoped the second as a full fallback.
Vault type: ship the pragmatic choice, document the better one
| Dimension | Standard vault | Logically air-gapped vault |
|---|---|---|
| Survives account compromise? | No. | Yes — compliance-mode lock, recovery via a separate account. |
| Test points deletable? | Yes. | No — locked for the full retention period. |
| Cost | Baseline (~$0.05/GB-month). | ~15% premium; EBS free tier does not apply. |
| Same-account restore clarity | Simple and well documented. | Documentation centers on the cross-account flow — a risk on deadline day. |
The standard vault shipped for the deadline — full freedom to delete and retry while proving the mechanism — and the air-gapped vault was flagged as the leading candidate for the automation phase.
The pivot: browser upload to direct CLI
The first plain-S3 design restored a snapshot to a temporary instance and uploaded through the S3 console. It was rebuilt mid-engagement when its limit surfaced: the browser uploader tops out near 160 GB per session, and every real target was larger. The AWS CLI's aws s3 sync uses multipart upload with no such ceiling, runs directly on each live server (preferably under an instance IAM role so no key sits on disk), and is idempotent — the commands that proved the concept are the seed of the automation.
aws s3 ls s3://<company>-backups-<account-id>-<region>/
aws s3 sync "C:\" s3://<company>-backups-<account-id>-<region>/<Server>/OS-C-Drive/ `
--exclude "pagefile.sys" --exclude "hiberfil.sys" --exclude "swapfile.sys" `
--exclude "System Volume Information\*" --storage-class STANDARD{ "Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::<bucket>" },
{ "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": "arn:aws:s3:::<bucket>/*" } ] }Rollout was checklist-driven across six targets, starting with the one that had zero existing coverage. Two volumes already snapshotted were deliberately copied to S3 as well — protection against AWS itself being unreachable, not redundant effort. Nested mount points were synced as explicit jobs with matching excludes so nothing uploaded twice.
5. Portable, bootable recovery: Disk2vhd images on S3, synced to a NAS
File-level copies and AWS-resident recovery points protect against most failures, but a file-level sync is not a bootable system, and every AWS-hosted copy still assumes AWS is reachable. The final layer closes that: bootable VHDX images of the server backups, created with Sysinternals Disk2vhd, hosted in a dedicated S3 bucket and pulled down to a Synology NAS, so a server can be brought up on ordinary Hyper-V hardware without AWS.
- Install Disk2vhd on the three servers — Application, EDI, and SQL. Extract the tool to the same fixed folder on each server's OS volume (for example
C:\Tools\Disk2vhd), at the same version everywhere, and record the version in the runbook. Because every server's boot volume is already covered by the daily snapshot policy with cross-region copies, the tool travels inside every snapshot: any instance recovered from a snapshot already has it, with no download or internet access needed mid-incident. A tool installed in advance, in the same place on every server, is one fewer thing to find during a recovery. - Pick the capture source. Either capture live on the server — Disk2vhd works from a Volume Shadow Copy, so the server stays online — or launch an instance from the air-gapped recovery point and capture there, where the tool is already present. In both cases production volumes are never modified and the locked original stays intact.
- Prepare a local landing volume. Write the image to a separate volume with enough free space for the output — never to a volume being captured. Keep that landing volume out of the daily snapshot policy, or clear it after upload, so multi-gigabyte images don't inflate every snapshot.
- Capture with Disk2vhd. Select every volume needed to boot — the system, boot, and recovery or EFI partitions that Disk2vhd lists, plus the data volumes the server needs — choose the VHDX format, and leave Volume Shadow Copy enabled. Run it off-hours; capture adds read load. Be deliberate on the SQL server about which volumes belong in the image and which are already covered by the log-backup and sync jobs. The command-line form is equivalent to the GUI (check the options for the installed version):
C:\Tools\Disk2vhd\disk2vhd.exe C: D: F:\VHDX\<Server>-<yyyymmdd>.vhdx - Boot-test the image in Hyper-V. Match the VM generation to the source firmware (BIOS → Generation 1, UEFI → Generation 2), boot with networking disconnected, and confirm services start. Record any first-boot adjustments — for instance, reconfiguring the network adapter or ignoring stale drive mappings (see the fire-drill findings above) — in the runbook.
- Upload to a dedicated S3 bucket with multipart upload, one prefix per server and date, with versioning and encryption on and a lifecycle rule keeping only the most recent images. Then clear the local copy from the landing volume:
aws s3 cp F:\VHDX\<Server>-<yyyymmdd>.vhdx ` s3://<company>-vhdx-<account-id>-<region>/<Server>/ --storage-class STANDARD - Sync the bucket to the Synology NAS. Configure a Cloud Sync task for the bucket and prefix, set to download remote changes only, on a schedule. Give the NAS its own read-only identity — list and get, nothing else:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::<vhdx-bucket>" }, { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": "arn:aws:s3:::<vhdx-bucket>/*" } ] } - Verify on the NAS side. Compare file sizes and checksums against the S3 object, and periodically boot-test the NAS copy itself, not just the original capture.
- Tear down. If a recovered instance was used for capture, delete it and its volumes, and deactivate any one-time access keys — enumerated as a numbered step so nothing quietly accrues cost.
A VHDX is a point-in-time image, so it complements — not replaces — the 15-minute object-storage path and the transaction-log chain for point-in-time database recovery. Pulling images to a NAS incurs S3 data-transfer-out charges, and large images take time to sync, so cadence should follow how much change the business can tolerate in a bare-metal recovery. Drivers and first-boot behavior differ on non-AWS hardware, which is exactly why the boot test and the periodic NAS boot test are part of the procedure.
6. Cost model
For a 50 GiB daily delta and 500 GiB steady-state footprint, the hybrid design's incremental cost was estimated at $60–$80 per month, more than offset by retiring two legacy backup servers.
| Component | Estimated monthly cost |
|---|---|
| Block-storage backup volume + snapshots | $45–$50 |
| Object storage (active + lifecycle tiers) | $5–$12 |
| Cross-region replication | $10–$15 |
| Monitoring & alerting | < $1 |
| Total new spend | $60–$80 |
| Offset — retire legacy backup servers | −$120 to −$320 |
| Net monthly change | −$60 to −$260 (savings) |
The deadline-phase AWS-native build used no new subscription: AWS Backup storage (~$0.05/GB-month) plus plain S3 (~$0.023/GB-month) came to roughly $22–$58 per month across all six targets. VHDX images add S3 storage for the retained images and data-transfer-out for NAS syncs; both scale with image size and sync cadence.
7. Risk management
| Risk | Mitigation |
|---|---|
| Silent sync failures | 24-hour stale-backup alarm. |
| Partial uploads on large files | Resumable, idempotent sync with retry logic. |
| Wrong immutability period | Confirm with the compliance owner first — it can be extended, never shortened. |
| Cost growth | Billing alarm at 150% of estimate; lifecycle rules on images. |
| Over-broad permissions | Per-credential scoping — single bucket, minimal actions, peer-reviewed. |
| Image that will not boot off-AWS | Boot-test at conversion and periodically from the NAS copy. |
8. Outcomes
- RPO improved from undefined to ≤15 minutes (object storage) and ≤4 hours (snapshots); RTO target ≤30 minutes, verified by drill.
- Deadline met: first backup proven in 24 hours, all six targets by day two, zero new subscriptions or vendors.
- Zero remaining dependency on unmonitored legacy backup infrastructure; every backup file protected by tested immutability.
- A recovery path that no longer requires AWS to be reachable.
- A handed-off roadmap rather than a closed ticket: automate the backup schedule, automate the S3 copy, evaluate the air-gapped vault properly, and set the VHDX cadence.
- Two audiences, two documents: the full runbook for decision-makers, and a one-page-per-server cheat sheet of pure commands for whoever executes at 6 a.m.
9. Lessons learned
- "We have snapshots" and "we have tested recovery" are different claims. Only a live drill proves the second.
- Map every dependency recovery assumes, especially infrastructure that predates the current architecture.
- State ambiguity instead of guessing, and version every runbook change against the test that prompted it.
- Interactive CLI tools and PowerShell ISE don't mix. ISE's pseudo-console is unreliable for interactive native executables; use plain PowerShell or Windows Terminal.
- An "access denied" can be a wrong-principal problem. The inline policy had been attached to the operator's console user rather than the service account the servers actually used.
- Separate scenarios explicitly — data-level, single-server, regional, and AWS-unreachable need different responses and different owners.