Test Script: Disaster Recovery / Failover

This is a ready-to-use disaster recovery and failover test script. A documented DR plan you have never exercised is a hope, not a control. This script proves three things you can show an inspector: that failover to your recovery site or standby actually works when invoked, that the recovered system functions correctly and its GxP data is complete and unaltered, and that the time to recover and the data you would lose stay inside the recovery time and recovery point objectives you committed to. It also proves you can fail back cleanly. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and execute it under an approved DR test plan or strategy. A worked filled specimen follows the template so you can see how a completed version reads. A test script is contemporaneous GxP evidence: write each actual result as you observe it, in ink or in the validated tool, with the real clock time, never reconstructed afterward. Verify each cited regulation against the current source before you rely on it.

What this script is, and where it sits

A backup restoration test proves you can read data back from a backup. A DR failover test proves something larger: that the whole service, the application, its data, its dependencies, its access paths, can be brought up at the recovery site or on the standby and used to do GxP work, inside the time you promised, with no more data lost than you promised, and then returned to the primary site. This script is the execution-level evidence for that. It sits below the DR plan, which names the systems, the recovery objectives, and the recovery sequence, and below the DR test strategy, which schedules and scopes the exercise. It produces the timestamped, step-by-step record that the DR test report rolls up.

Scope this script to one DR scenario and one system or tightly grouped service set per execution, so each result is traceable to a single, clear recovery objective. Common scenarios are a planned failover (graceful), an unplanned failover (simulated loss of primary with no graceful shutdown), and a tabletop or partial test for systems too disruptive to fail over live. Decide the scenario and the recovery sequence first, in the plan, then write the script. The rigor of verification should match the criticality of the system: a system that holds release-decision or patient-safety data earns the full functional and integrity verification below, while a lower-criticality system may justify a lighter functional check, documented as such.

Document control header

1. Objective and requirement traced

State what passing this script demonstrates and the plan requirement it traces to, so the DR test report can link the recovery objective to the evidence in both directions.

Example objective: this script verifies that, on a simulated loss of the primary, the system can be failed over to the recovery site, that an authorized user can log in and perform a representative GxP transaction, that the recovered data is complete and unaltered up to the replication point, that the measured recovery time and data loss window meet the committed RTO and RPO, and that the system can be returned to the primary site without data loss.

<<FILL: objective>>

2. Definitions

• Failover: switching live processing from the primary system or site to the standby or recovery site.
• Failback: returning processing from the recovery site to the primary after the test or after the primary is restored, including reconciling any data written while on the recovery site.
• Recovery time objective (RTO): the maximum acceptable time from the loss event (or the decision to invoke DR) to the system being usable for GxP work. This script measures actual recovery time against it.
• Recovery point objective (RPO): the maximum acceptable amount of data, measured in time, the organization is willing to lose in a recovery. This script measures the actual data loss window against it.
• Recovery site / standby: the segregated infrastructure (hot, warm, or cold site, replicated region, or vendor DR instance) that takes over on failover.
• Replication lag: the delay between a change committed on primary and that change being present on the standby; it sets the achievable RPO.
• Split brain: the failure state where both primary and recovery believe they are live and accept writes, producing two divergent data sets. Preventing and detecting it is a required check.

3. Prerequisites

List everything that must be true before execution starts. If any prerequisite is not met, do not execute; record the gap as a test deviation and stop. A DR test launched on wrong assumptions can damage live data.

4. Roles at execution

5. Execution conventions

Read these before executing, then apply them to every step.

• Record each actual result contemporaneously, with the real clock time from the synchronized source. RTO and RPO are time measurements; an estimated or backfilled timestamp invalidates them.
• “Meets expected result” is not an actual result. Write what you actually saw: the value, the message, the screen state, the timestamp.
• Use one unambiguous mark for pass or fail. A blank cell is incomplete, not a pass.
• For any single-line correction, strike through once, initial, date, and add a brief reason. Never obliterate the original entry.
• Capture objective evidence (screenshots, failover logs, replication-status reports, query output) for every step whose pass depends on a system output, and record a unique evidence reference in the step.
• Honor the abort criteria. If the test risks live data or runs past its window, abort per the runbook, fail back, and record it as a deviation rather than forcing the test.
• Where the scenario is “unplanned”, simulate the loss as defined (pull the network path, power down the primary node, or use the platform’s force-failover) rather than a graceful shutdown, because graceful and ungraceful failover behave differently and only the ungraceful path proves the real case.

6. Pre-failover baseline (capture on the primary)

Capture and record the known-good state of the primary before you invoke failover. Without this, completeness and RPO cannot be measured after recovery.

7. Test steps

Each step gives the action, the precise expected result, the actual result as observed with its clock time, pass or fail, and the evidence reference. Number steps continuously. Steps 7.1 to 7.4 invoke failover and start the RTO clock; 7.5 to 7.10 verify recovered function and data integrity; 7.11 to 7.12 measure RPO; 7.13 to 7.16 fail back and reconcile.

Add or remove rows so the recovery sequence in your plan is covered. For a tabletop or partial test, mark the steps that were walked through rather than physically executed, and state that limitation clearly in section 9. Do not reuse a step number after a revision.

8. RTO and RPO measurement

Record the measured outcomes once, from the steps above, so the report and the plan can be compared directly.

If the measured recovery time exceeds the RTO, or the data loss window exceeds the RPO, that is a deviation and a real gap in the continuity control, not a footnote. It usually means the recovery sequence must be streamlined or replication frequency tightened, and the plan’s committed objectives revisited.

9. Acceptance criteria

This script passes only when all of the following are true:

• Failover was invoked per the scenario and the system reached a usable state at the recovery site, with no split-brain.
• An authorized user could log in with correct access controls and perform the representative GxP read and write at the recovery site.
• Recovered data was complete (counts and coverage matched the baseline to the replication point) and unaltered (hashes matched, audit trail continuous, signatures intact).
• Measured recovery time was at or within the committed RTO.
• Computed data loss window was at or within the committed RPO.
• Failback completed, recovery-site writes were reconciled onto the primary with no divergence, and the audit trail spans the whole window.
• Every deviation raised during execution is resolved with a documented disposition, and any retest is recorded.
• The completed script is reviewed and approved per section 12, by a reviewer who did not execute it.

A script with one or more open Fail steps, an RTO or RPO breach not dispositioned, or unresolved deviations does not pass and cannot support the conclusion in the DR test report. For a tabletop or partial test, the result is qualified explicitly to the steps actually executed.

10. Deviations raised during execution

Any step that does not meet its expected result, any RTO or RPO breach, any departure from the runbook, any aborted test, and any prerequisite not met is recorded here and managed per the parent plan and <<FILL: deviation / test-incident SOP-ID>>. Cross-reference the deviation number on the failing step.

A failed failover or a missed RTO is data, not a finding against the team. An inspector looks for the failure to be captured honestly, classified, dispositioned, and any retest traceable to the original failure. Where the failure shows the architecture or the committed objective is wrong, the plan and the RTO/RPO are corrected under change control and the deviation references that change.

11. Execution summary

12. Sign-off

The test lead and operator execute and sign; an independent reviewer who did not execute the test confirms the record is complete, the evidence supports each pass, the timings are real, and any deviation is resolved.

13. References

21 CFR Part 11 (electronic records and electronic signatures), 11.10(c) protection of records and 11.10(b) accurate and complete retrieval throughout the retention period. 21 CFR 211.68 (automatic, mechanical, and electronic equipment, including backup of data) for drug GMP. EU GMP Annex 11 (Computerised Systems), section 16 (business continuity) and section 7 (data storage and backup). EU GMP Annex 15 (Qualification and Validation), 2015 revision. MHRA GxP Data Integrity Guidance and Definitions (backup, restore, archiving, and business continuity). PIC/S PI 041, Good Practices for Data Management and Integrity. ICH Q9, Quality Risk Management (for the risk basis of test scope and depth). ISO 22301 (Business Continuity Management Systems) and ISO/IEC 27031 (ICT readiness for business continuity), as design references. GAMP 5 (Second Edition), business continuity and disaster recovery for computerized systems. For medical device software: 21 CFR Part 820 (QMSR, effective 2 February 2026, incorporating ISO 13485:2016 by reference) and IEC 62304.

Confirm the current version and clause numbers of each reference before issue.

14. Revision history

Filled specimen

The following shows the script executed for an example laboratory information management system on an active-passive standby with cross-region replication, running an unplanned (simulated loss) failover and a failback. The company, system, accounts, and numbers are illustrative; replace them with your own.

Header (extract)

Pre-failover baseline (captured on primary)

Test steps (executed, extract)

RTO and RPO measurement

Execution summary

Deviation (one raised)

Sign-off

In this example the team simulated a real loss rather than a graceful shutdown, fenced the primary to avoid split-brain, measured an actual recovery time of 2h34m against an 8 hour RTO and a 32 second data loss window against a 1 hour RPO, and proved the recovered data matched the primary by counts, hashes, audit trail, and signatures. They caught a quiet failover config flaw (an interface reconnecting to live production), raised it, fixed it, re-executed the step, and then failed back and reconciled the one record written during the test so nothing was lost or duplicated. That sequence, declare to failover to verify to RTO/RPO to deviation to fix to failback to reconcile, is exactly what a reviewer is expected to demonstrate.

Common inspection findings this script prevents

• A DR plan exists on paper but no failover has ever been executed, so recoverability is unproven.
• Only a graceful, planned failover was ever tested, never an ungraceful loss, so the real disaster case is unverified.
• Failover “tested” by confirming the standby came up, with no check that the recovered data is complete and unaltered and no representative GxP transaction performed.
• RTO and RPO stated in the plan but never measured against an actual failover, so the committed objectives are unsubstantiated.
• Recovery time measured but data loss never computed, or counts checked without hashes, so silent corruption or a too-large loss window passes unnoticed.
• The recovered audit trail or electronic signatures were not verified to survive failover, leaving a Part 11 gap after recovery.
• Split-brain not prevented or detected, with both sites accepting writes, or failback performed without reconciling records written on the recovery site, so data is lost or duplicated.
• An interface at the recovery site silently connected back to live production during the test, putting real records at risk.
• The same person executed and reviewed the test, with no independent check that the timings and evidence support the conclusion.
• A breached RTO or RPO recorded with no deviation and no plan revision, so a known continuity gap is left unmanaged.

How to adapt this script

1. Set your test script ID, parent DR plan or test strategy, recovery architecture, and the committed RTO and RPO in the header, and confirm a verified pre-test backup exists so a real failure during the test is itself recoverable.
2. Choose one DR scenario per execution (planned, unplanned, partial, or tabletop) and keep the verification depth matched to the system’s criticality; reserve the full functional and integrity verification for higher-criticality systems.
3. Capture your real pre-failover baseline in section 6: the counts, last-committed timestamp, and hashes you will measure recovery against.
4. Rewrite the steps in section 7 to follow your actual recovery sequence and your runbook, including the dependencies and interfaces specific to your system, and confirm you simulate an ungraceful loss where the scenario calls for it.
5. Record RTO and RPO with real clock times from a synchronized source, and if either is breached, raise a deviation and feed the result back into the plan’s committed objectives.
6. Point the cross-references in sections 3, 10, and 12 to your real DR plan, deviation or test-incident procedure, and review and approval process.
7. Confirm every regulation in section 13 against the current published version before issue.