Case study — illustrative reference architecture
Resilient Data & Application Platform
Keeping a customer-facing application available and its data recoverable when infrastructure fails.
Context
Scenario & business problem
Scenario
A hypothetical customer-facing application must remain available during infrastructure failures while protecting data integrity and supporting predictable recovery.
Business problem
The platform has redundancy on paper but no tested failover, unclear recovery objectives, a shared cache treated as durable state, and dependencies that fail hard rather than degrade. The architecture must define availability and recovery objectives, make failure detection explicit, and prove recovery through drills.
Business drivers
- Protect revenue during infrastructure failure
- Define and evidence recovery objectives
- Protect data integrity during failover
- Degrade gracefully instead of failing entirely
- Integrate detection with incident response
Current-state challenges
- Failover has never been executed outside a design document
- Backups are taken but restores are untimed and unverified
- A single shared cache is a hidden single point of failure and is treated as durable state
- Alerts fire on symptoms rather than customer impact
Who is affected
Stakeholders
Product Owner
Customer-visible availability and honest status communication.
Platform Engineering
Failover automation and clear dependency ownership.
Cloud Operations
Actionable alerts and rehearsed recovery runbooks.
Security Engineering
Key availability in the recovery region and backup immutability.
Application Team
Degradation modes that keep core journeys usable.
Analysis
Requirements, assumptions & constraints
Functional behaviour is separated from measurable quality attributes so that tradeoffs can be argued against numbers rather than opinions.
Functional requirements
- Serve core customer journeys from at least two availability zones
- Queue non-critical work when downstream dependencies are unavailable
- Expose a health endpoint per service consumed by the load balancer
- Support read-only mode during data-tier recovery
- Provide restore of any dataset to a chosen point within retention
Non-functional requirements
- Illustrative availability objective of 99.95% monthly for core journeys (design target, not a measured result)
- Illustrative RTO targets (target time to restore an acceptable level of service): 30 minutes in-region, 4 hours cross-region
- Illustrative RPO targets (target maximum data-loss window): near-zero in-region for committed protected transactions, up to 1 hour cross-region
- Illustrative detection objective: impact-based alert within 60 seconds of a failure signal
- Illustrative sizing assumption: 40% capacity headroom to absorb the loss of one zone
- Cache tier is non-authoritative: core journeys must remain functional, in reduced mode if necessary, when it is unavailable
Assumptions
- Two availability zones remain reachable during a single-zone event
- Cross-region replication lag is assumed to stay within the illustrative RPO target
- Non-critical features can be disabled by configuration flag
Constraints
- Cross-region active-active is out of budget for this phase
- One third-party payment dependency has no fallback provider
- Restore testing must run outside business hours
Guardrails
Architecture principles
Design
Target architecture
Components, trust zones, and the flows that cross their boundaries.
Normal operating architecture
Traffic is distributed across redundant application instances in two zones. The primary database replicates synchronously to a standby and asynchronously to a secondary region. Health monitoring and central logging feed alerting continuously.
Traffic entry
Health-aware routingGlobal DNS / traffic policy
Health-check driven regional routing
Load balancer
Removes unhealthy targets within seconds
Application tier — primary region
Redundant across two zonesService instances — Zone A
Stateless, autoscaled, illustrative 40% headroom assumption
Service instances — Zone B
Identical footprint, independent failure domain
Durable queue
Absorbs load and buffers non-critical work
Non-authoritative cache tier
Redundant across failure domains; timeouts and circuit-breaking; bypassable and rebuilt from authoritative data
Degradation flags
Disable non-essential features under stress
Data tier
Replication and backupPrimary database
Synchronous standby in second zone
Zone B standby
Automatic promotion candidate
Cross-region replica
Asynchronous; illustrative RPO target up to 1 hour
Backup vault
Immutable copies, tested restores
Operations plane
Detection and responseHealth monitoring
Synthetic journeys and per-service probes
Central logging & metrics
Correlated traces across services
Alerting & on-call
Customer-impact alerts page a human
Secrets & key management
Key service, policies, and access prepared in the recovery region
Dependency map
Criticality and fallback per dependency
Flows & boundaries
- Global DNS Load balancerRoute to healthy regionRuntime data path
- Load balancer Zone A and Zone B instancesEven distribution, unhealthy targets removedRuntime data path
- Service instances Primary databaseReads and writesRuntime data path
- Primary database Zone B standbySynchronous replication; illustrative near-zero RPO target for committed transactionsRuntime data path
- Primary database Cross-region replicaAsynchronous replicationRuntime data path
- Service instances Non-authoritative cache tierRead-through with timeout and circuit-breaker; bypass to database on failureRuntime data path
- Backup vault Restore test environmentIllustrative monthly timed restore exerciseControl & telemetry
- Health monitoring Alerting & on-callImpact-based alert; illustrative 60-second detection objectiveControl & telemetry
Failure detection and disaster recovery flow
Detection drives a decision: contained failures are absorbed by in-region redundancy and degradation flags; a region-level failure escalates to a declared disaster-recovery event with cross-region promotion and a documented data-loss window.
1 — Detect
Illustrative 60-second detection objectiveFailed health probes
Instance, zone, or dependency level
Synthetic journey failure
Customer-impact confirmation
Error-rate / latency alert
Golden signals breach threshold
2 — Classify
Scope determines the responseInstance failure
Load balancer removes the target; autoscale replaces it
Zone failure
Standby promotion, capacity headroom absorbs load
Dependency or cache failure
Cache is bypassed and rebuilt from authoritative data; degradation flags disable affected non-critical features where the journey allows
Region failure
Disaster-recovery event declared by on-call lead
3 — Recover
Automated where safe, human-approved where notAutomatic in-region failover
Standby promoted, target 30-minute recovery time
Graceful degradation
Read-only mode; queued work replays on restore
Cross-region promotion
Approved runbook, 4-hour recovery time
Point-in-time restore
Used for data corruption rather than outage
4 — Confirm & learn
Integrated with incident responseIntegrity verification
Reconcile replayed queue work and data checks
Incident response integration
Status communication and severity handling
Post-incident review
Feeds the risk register and roadmap
Flows & boundaries
- Health probe failure ClassificationAlert with impact scopeFailure / recovery
- Instance failure Load balancerTarget removed, replacement launchedFailure / recovery
- Zone failure Automatic in-region failoverStandby promoted; illustrative 30-minute in-region RTO targetFailure / recovery
- Dependency failure Graceful degradationFeature flag off; queue buffers workFailure / recovery
- Region failure Cross-region promotionDeclared event, approved runbookFailure / recovery
- Recovery Integrity verificationReconcile data before full serviceControl & telemetry
- Post-incident review Risk registerMitigations and owners updatedControl & telemetry
Controls
Security, resilience & operational controls
Availability design
- Load balancing
- Health-checked distribution across two zones.
- Redundant services
- Stateless instances with an illustrative 40% capacity-headroom assumption.
- Resilient cache pattern
- Cache is non-authoritative and never the system of record: redundant across failure domains where warranted, fronted by timeouts and circuit-breaking, bypassable so core journeys degrade rather than fail, and rebuilt from authoritative stores rather than treated as durable state.
- Graceful degradation
- Feature flags preserve core journeys under stress.
Data protection
- Database replication
- Synchronous in-region standby supports in-region availability; asynchronous cross-region replication supports regional disaster recovery. These are distinct mechanisms from backup.
- Backup strategy
- Immutable, versioned, retained to policy.
- Restore testing
- Illustrative monthly timed restore exercise producing measured evidence to compare against the stated RTO target. Backups and point-in-time restore address recoverability and data-corruption scenarios, not availability.
Detection & response
- Health monitoring
- Per-service probes plus synthetic customer journeys.
- Centralized logging
- Correlated traces to shorten diagnosis time.
- Alerting
- Customer-impact alerts page; symptom noise is suppressed.
- Incident response integration
- Severity, comms, and review are part of the design.
Foundational services
- Secrets & key management
- Recovery readiness requires more than copying key material: the key-management service, authorization policies, application access paths, rotation state, and rehearsed recovery procedures must all be available in the recovery region.
- Dependency mapping
- Each dependency has criticality, fallback, and owner.
- Capacity planning
- Headroom sized under an illustrative assumption of surviving the loss of one zone.
- Monitoring vs recovery
- Monitoring detects conditions but does not itself provide recovery; incident response governs declaration, communication, escalation, and review, and post-incident review feeds the risk register and roadmap.
Honesty
Tradeoff analysis
Every architectural gain is paid for somewhere. These are the bills.
| Choice | What it gains | What it costs |
|---|---|---|
| Synchronous in-region replication | Illustrative near-zero RPO target for committed protected transactions on zone failure, subject to platform and failure semantics. | Added write latency, tighter coupling between zones, and no protection against logical corruption — backups remain necessary. |
| Warm cross-region standby instead of active-active | Far lower cost and no multi-region write-conflict handling. | Longer illustrative RTO target (4 hours) and a larger illustrative RPO target (up to 1 hour) in a regional event, plus partly idle standby spend. |
| Human-approved cross-region promotion | Avoids split-brain and unnecessary failovers. | Recovery depends on on-call availability, declaration speed, and drill quality. |
ADRs
Architecture decision records
Warm cross-region standby vs active-active multi-region
Accepted- Context
- The platform must survive a regional failure, but active-active operation would require data-conflict handling and roughly double the run-rate.
- Decision
- Run multi-zone active in the primary region with an asynchronously replicated warm standby region promoted through an approved runbook.
- Alternatives considered
- Active-active across two regions
- Backup-and-restore only recovery
- Pilot-light region rebuilt on demand
- Benefits
- Designed around an illustrative 4-hour cross-region RTO target at moderate cost
- Avoids multi-region write-conflict complexity
- Keeps the everyday operating model simple
- Tradeoffs
- The illustrative asynchronous-replication design accepts an RPO target of up to one hour for a regional disaster scenario
- Recovery depends on human declaration and runbook quality
- Standby capacity is partly idle spend
- Operational impact
- An illustrative quarterly promotion-drill cadence is modeled to exercise the recovery runbook; the accepted data-loss window must be communicated to the business in advance.
Graceful degradation via feature flags vs hard dependency failure
Accepted- Context
- A third-party payment dependency has no alternative provider, and today its failure takes down unrelated customer journeys.
- Decision
- Introduce degradation flags and durable queueing so non-critical features disable independently while core journeys continue in a reduced mode.
- Alternatives considered
- Fail the whole request when any dependency is unavailable
- Add a second payment provider now
- Synchronous retries with longer timeouts
- Benefits
- Core journeys survive selected dependency outages, including loss of the non-authoritative cache tier, which is bypassed or served in reduced mode
- Queued work replays after recovery rather than being lost
- Clearer customer communication during partial outages
- Tradeoffs
- More application state and flag configuration to maintain
- Reduced-mode behaviour must be tested like a real feature
- Risk of flags being left in the wrong position
- Degradation is only valid where the business journey and data-integrity requirements allow it; some dependency failures must still fail closed
- Operational impact
- Flag state must be visible in dashboards and reviewed after every incident; degraded modes need their own test coverage.
Illustrative monthly timed restore testing vs backup-success monitoring only
Accepted- Context
- Backups complete successfully but no restore has been timed, so the recovery time objective is an assumption rather than a measurement.
- Decision
- Run an illustrative monthly automated restore into an isolated environment and record the restore duration.
- Alternatives considered
- Rely on backup job success metrics
- Annual manual restore exercise
- Continuous restore validation on every backup
- Benefits
- Timed restore exercises provide measured evidence that can be compared with the stated recovery objective
- Restore exercises can surface backup-integrity problems, restore failures, or procedural gaps that backup-success monitoring alone may not reveal
- Provides illustrative evidence for stakeholders reviewing recovery readiness
- Tradeoffs
- Ongoing compute cost for the restore environment
- Engineering time to maintain the automation
- Test window constrained to out-of-hours
- A successful restore test validates the tested restore path under the exercise conditions; it does not by itself prove full application disaster-recovery readiness
- Operational impact
- Restore duration is tracked as an operational metric; regressions raise a risk-register update.
Governance
Risks & mitigations
| ID | Scenario | Likelihood | Impact | Component | Mitigation | Residual | Owner |
|---|---|---|---|---|---|---|---|
| RSK-09 | Zone failure exceeds remaining capacity and the surviving zone degrades under load. | Medium | High | Application tier capacity | Maintain the illustrative 40% headroom sizing assumption, autoscale ahead of saturation, and validate in zone-failure game days. | Medium | Platform Engineering |
| RSK-10 | Cross-region replication lag exceeds the stated recovery point before a regional failure. | Medium | High | Cross-region replica | Alert on replication lag thresholds and treat sustained lag as an incident, not a warning. | Medium | Cloud Operations |
| RSK-11 | Backups complete but restores fail due to undetected corruption or missing keys. | Low | High | Backup vault / key management | Monthly timed restore tests including key availability in the recovery region. | Low | Cloud Operations |
| RSK-12 | Degradation flags left enabled after an incident silently disable customer features. | Medium | Medium | Feature flag service | Flag state on the operations dashboard, automatic expiry, and a post-incident checklist item. | Low | Application Team |
| RSK-13 | Alert noise causes on-call responders to miss a genuine customer-impacting failure. | Medium | High | Alerting & observability | Page only on customer-impact signals; route symptom alerts to dashboards and weekly review. | Medium | Cloud Operations |
Sequencing
Reference implementation roadmap
These phases describe a reference implementation approach. They are not a record of a real production deployment.
- Phase 1
Discovery & Requirements
- Define availability, recovery time, and recovery point objectives
- Map dependencies and criticality
- Identify single points of failure
Exit criteria Agreed objectives with named owners.
- Phase 2
Foundation
- Establish multi-zone footprint
- Set up replication and immutable backups
- Instrument health and golden-signal metrics
Exit criteria Redundancy and telemetry in place.
- Phase 3
Migration / Implementation
- Make services stateless
- Introduce durable queueing
- Implement degradation flags
Exit criteria Core journeys survive a simulated instance loss.
- Phase 4
Security & Operational Hardening
- Prepare key-management service availability, authorization policies, and application access in the recovery region
- Write and rehearse runbooks
- Tune alerts to customer impact
Exit criteria On-call can execute recovery without tribal knowledge.
- Phase 5
Validation
- Zone-failure game day
- Illustrative monthly timed restore exercise
- Illustrative quarterly cross-region promotion drill
Exit criteria Exercise results are compared against the stated illustrative objectives.
- Phase 6
Optimization
- Reduce recovery time through automation
- Right-size headroom
- Retire remaining single points of failure
Exit criteria Lower recovery time at stable cost.
Evidence
Validation approach
- Zone-failure drill keeps core journeys available within the illustrative RTO target
- Timed restore exercises produce measured evidence compared against the illustrative RTO and RPO targets; a successful restore test validates the tested restore path under exercise conditions, not full disaster-recovery readiness
- Cache-outage exercise confirms core journeys bypass or degrade around the cache
- Dependency outage triggers degradation rather than total failure, where the business journey and data-integrity requirements allow it
- Every alert that paged mapped to real customer impact