Disaster Recovery Plan (DR Setup and Runbook)

This is a ready-to-use disaster recovery plan. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, attach your architecture diagrams and contact roster, and route it through your normal document control, review, and approval. A worked filled specimen follows the template so you can see how a completed version reads. Verify each cited regulation against the current source before you rely on it.

A disaster recovery plan answers two numbers and one question. The numbers are how much data you can afford to lose (recovery point objective, RPO) and how long you can be down (recovery time objective, RTO). The question is: when the primary site or system is gone, what does each person do, in what order, to bring the GxP records back to a trustworthy state. This plan is the document an inspector asks for after a ransomware event, a data center loss, or a failed upgrade, and the document your own team follows at 2 a.m. when the production batch system will not come up.

Document control header

1. Purpose

This plan defines how <<FILL: COMPANY NAME>> recovers the GxP computerized systems in scope after an event that makes a primary system or site unavailable, so that GxP records are restored complete, accurate, and attributable within the agreed recovery objectives. The objective is to return regulated operations to a validated state with no silent data loss and a documented trail of what was recovered and how it was verified.

2. Scope

This plan applies to the systems listed in the inventory in section 5 at the sites named in the header. It covers loss of a server, a database, a storage array, a network segment, a whole site or data center, a cloud region or service, and logical loss from corruption, malware, or a failed change. It covers the technical recovery of the system and its GxP data, and the verification that the recovered system is fit to use again.

This plan does not cover routine backup and restore of single files, which is governed by <<FILL: SOP-ID for backup and restore>>, nor the wider business continuity plan for people, facilities, and supply, which is governed by <<FILL: BCP-ID>>. DR is the technology recovery subset of business continuity; keep the two linked but distinct.

3. Responsibilities

4. Definitions

• RTO (recovery time objective): the maximum acceptable elapsed time from disaster declaration to the system being usable again. It drives the architecture you must buy.
• RPO (recovery point objective): the maximum acceptable amount of data loss, expressed as time. An RPO of 15 minutes means you may lose up to the last 15 minutes of transactions. It drives backup and replication frequency.
• MTD (maximum tolerable downtime): the absolute outer limit beyond which the outage causes unacceptable harm to patients, product, or the business. RTO must sit inside MTD with margin.
• Failover: switching production from the failed primary to a standby or recovered environment.
• Failback: returning production to the primary (or a rebuilt primary) once it is stable, in a controlled way that does not lose data created during failover.
• Warm / hot / cold standby: a standby kept running and current (hot), kept built but lagging or off (warm), or assembled only when needed (cold). Colder is cheaper and slower.
• Recovery point achieved / recovery time achieved: the measured RPO and RTO actually delivered in a real event or a test, compared against the objective.

5. System inventory with RTO and RPO

List every in-scope system, its GxP criticality, the recovery objectives it must meet, and the recovery method that delivers them. Derive criticality and objectives from a documented business impact analysis (BIA), referenced in Attachment 1. RTO and RPO are commitments you must be able to demonstrate, not aspirations; set them where your architecture can actually meet them, then prove it in the test in section 10.

Tier guidance, to set defaults you then justify per system:

6. DR architecture

Describe how the in-scope systems are protected and where they recover to. State the technology, the replication mechanism, and the geographic separation, and attach the topology diagram (Attachment 2). The architecture must be able to meet the RTO and RPO in section 5; if it cannot, either fix the architecture or change the objectives, do not leave a gap.

Two architecture facts that inspectors and ransomware both test. First, asynchronous replication means your real RPO equals the replication lag at the moment of failure, not the schedule on paper; measure the lag and record it. Second, a backup that the same credentials can reach and delete is not a backup against malware; keep at least one immutable or air-gapped copy following the 3-2-1 pattern (three copies, two media, one off-site), and prove you can restore from it.

7. Invocation criteria

A disaster is declared, and this plan invoked, when one or more of the following is true and a routine fix inside the normal incident process cannot restore service inside the RTO:

1. A primary system, database, or storage component is lost or corrupted and cannot be restored in place within its RTO.
2. A site, data center, network segment, or cloud region is unavailable and expected to stay unavailable beyond the RTO.
3. GxP data is confirmed corrupted, encrypted by malware, or otherwise untrustworthy, so the system cannot be used safely even if it is technically running.
4. A change or upgrade has failed and cannot be rolled back inside the change window, leaving a GxP system unusable.

Who may declare: <<FILL: roles authorized to declare a disaster, e.g. DR coordinator or on-call IT manager>>. The declaration, with timestamp and reason, is the first entry in the decision log (Attachment 3). Declaring is not failure; declaring late is. If in doubt and the RTO clock is running, declare.

8. Failover runbook (step by step)

Execute in order. Record the actor, timestamp, and result for every step in the decision log. Do not skip the pre-checks; a failover onto an unverified or stale standby creates a second disaster.

8.1 Declare and mobilize

1. The authorized person declares the disaster and starts the decision log with the timestamp and reason.
2. The DR coordinator opens the bridge / war room and assigns the roles in section 3 from the on-call roster (Attachment 4).
3. Start the RTO clock and record it. Note the last known good replication point as the expected RPO.

8.2 Assess and stop the bleeding

1. Confirm the scope: which systems, which data stores, which sites are affected.
2. Confirm the primary cannot be restored in place inside the RTO (otherwise this is incident handling, not DR).
3. If malware is suspected, isolate affected networks and do not connect the clean recovery environment to anything that could re-infect it. Recover only from an immutable or air-gapped copy.

8.3 Recover infrastructure

1. Bring up the recovery environment: compute, storage, network, and the authoritative time source.
2. Confirm the recovery environment matches the qualified configuration baseline (Attachment 5); record any difference as a deviation.
3. Confirm network routing, firewall rules, and certificates are in place for the recovery environment.

8.4 Recover data to the recovery point

1. Restore or activate the database from the latest valid replica or backup.
2. The DBA confirms the actual recovery point reached and records the achieved RPO (gap between last committed transaction and the disaster time).
3. Restore associated file shares, object stores, and interfaces to the same point in time so the data set is internally consistent.
4. Run database and file integrity checks (checksums, row counts, referential integrity) and record the result.

8.5 Bring up the application and cut over

1. Start the application services in the recovery environment and confirm they connect to the recovered data.
2. Confirm time synchronization so new audit trail entries carry correct, attributable timestamps.
3. Redirect clients via DNS or load balancer to the recovery environment and confirm users can authenticate with correct roles and privileges.

8.6 Verify before release

1. Execute the recovery verification checks in section 11 and obtain the system owner’s functional confirmation.
2. QA confirms the recovered system is in a validated, fit-for-use state and records the disposition.
3. The DR coordinator records recovery time achieved against RTO and the recovery point achieved against RPO.
4. Authorize return to GxP use only after the section 11 checks pass. Until then, the system is not to be used for GxP decisions.

9. Communication

Run the cascade below as soon as a disaster is declared, then on a fixed interval until all-clear. Keep messages factual: what is down, expected restoration, what users must do meanwhile.

Manual workaround instruction: while a system is down, operations follow <<FILL: SOP-ID for the documented manual / contingency procedure>>. Records created on paper or an alternative system during the outage must be reconciled back into the recovered system per section 11, with full traceability and no loss of attribution.

10. DR testing

A DR plan that has never been tested is a hypothesis. Test on a defined schedule and after any major architecture change, and record the result against the objectives in a test report (linked in the header).

At least once across the test cycle, perform a full timed restore that actually brings the system up and verifies the data, not only a tabletop walk-through. A restore you have only ever discussed is the one that fails in the real event.

11. Validation of recovered systems

A recovered GxP system must be proven fit for use before it carries GxP decisions again. The depth is risk-based: a like-for-like restore onto qualified infrastructure needs verification, not a full revalidation, but it needs documented verification.

If the recovery rebuilt the system differently (new infrastructure, changed configuration, a vendor-side change, a different version), assess what revalidation is required per <<FILL: CSV / change control SOP-ID>> before release. Treat that assessment as a change, not as part of the firefight.

12. Dependencies

A system rarely recovers alone. Map what each in-scope system needs to function so you recover in the right order and do not bring up an application whose authentication, database, or interface partner is still down.

Cross-dependencies to confirm explicitly: identity and single sign-on, DNS, the authoritative time source, certificate authority, the database tier, shared storage, network and VPN, license servers, and the cloud provider control plane. Recover foundational services first; an application brought up before its identity provider cannot authenticate anyone.

13. Acceptance criteria

Recovery is complete and the plan closed only when all of the following are true:

• The system is restored and authorized for GxP use by the system owner and QA.
• Recovery time achieved is at or below the RTO, or the overrun is documented as a deviation with cause.
• Recovery point achieved is at or below the RPO, or the data gap is documented, assessed for GxP impact, and dispositioned.
• The section 11 verification checks pass, with audit trail intact and data reconciled.
• The decision log is complete and signed, and any CAPA from the event or test is opened.

14. Failback

Do not run on the recovery environment indefinitely; return to a stable primary in a controlled way that loses none of the data created during failover.

1. Confirm the primary (rebuilt or repaired) is stable and matches the qualified baseline.
2. Reverse-replicate or synchronize all data created on the recovery environment during the outage back to the primary, and verify completeness.
3. Schedule the cutover in a planned window under change control per <<FILL: change control SOP-ID>>; do not fail back under pressure.
4. Cut clients back to the primary, run the section 11 checks again on the primary, and obtain QA disposition.
5. Stand the recovery environment back down to standby and restore normal replication and backups.

15. References

21 CFR Part 11 (electronic records and signatures), including controls for protection of records to enable accurate and ready retrieval. 21 CFR 211.68 (automatic, mechanical, and electronic equipment), including backup of data. EU GMP Annex 11, sections 7.2 (data, regular backups), 16 (business continuity), and 17 (archiving). EU GMP Annex 15 (Qualification and Validation), 2015 revision, for the qualified state and change control of infrastructure. PIC/S PI 041, Good Practices for Data Management and Integrity, for backup, restoration, and data protection. GAMP 5 (Second Edition), including IT infrastructure control and business continuity / disaster recovery expectations. ICH Q9, Quality Risk Management, for the risk-based RTO / RPO and revalidation decisions. ISO 22301 (Business continuity management systems) and ISO/IEC 27031 (ICT readiness for business continuity), as recognized frameworks.

Confirm the current version and clause numbers of each reference before issue.

16. Attachments

17. Revision history

18. Approvals

Filled specimen

The following shows the inventory, an architecture summary, and a recovery log completed for an example manufacturing execution system, so you can see the level of detail an inspector expects. The company, system, and numbers are illustrative; replace them with your own.

System inventory (extract)

Architecture summary

Recovery log extract (real event, primary region storage failure)

In this example the team declared early rather than chasing an in-place fix past the RTO, restored from a measured replica and recorded a 6-minute real RPO against a 15-minute objective, verified data integrity and the audit trail before release, reconciled the records made on paper during the outage, and closed inside the 4-hour RTO with a signed QA disposition. That sequence, declare to recover to verify to dispose, with achieved RTO and RPO recorded against objectives, is exactly what a reviewer is expected to demonstrate.

Common inspection findings this plan prevents

• A disaster recovery plan exists but RTO and RPO are not defined per system, or are aspirations the architecture cannot meet.
• Backups or replicas have never been test-restored, so there is no evidence recovery actually works.
• The only backup copy is reachable and deletable by the same credentials, so ransomware encrypts the backup too; no immutable or air-gapped copy.
• Asynchronous replication is in place but the real RPO (replication lag at failure) is never measured or recorded, so the claimed RPO is unproven.
• A recovered system is returned to GxP use with no data-completeness, audit-trail, or QA disposition check.
• Records created on paper during the outage are entered into the recovered system later with no traceability, breaking attribution.
• Recovery happens out of dependency order, so an application is brought up before its identity provider, database, or time source.
• Failback is improvised under pressure and loses the data created on the recovery environment during the outage.
• The plan was never updated after a major architecture or cloud change, so the runbook no longer matches reality.

How to adapt this plan

1. Set your document number, owners, effective date, and linked DR test report number in the header.
2. Run or reference a business impact analysis, then fill the inventory in section 5 with your real systems and the RTO and RPO each must meet; set objectives where your architecture can actually deliver them.
3. Describe your real architecture in section 6 and attach the topology diagram, including your immutable or air-gapped backup copy.
4. Replace the dependency map in section 12 with your real identity, DNS, time, database, storage, and interface dependencies, and set the recovery order.
5. Point the cross-references in sections 2, 9, 11, and 14 to your real backup, manual-workaround, audit trail review, CAPA, and change control procedures.
6. Schedule and run a full timed restore test per section 10 before you rely on this plan, and feed every gap back into the document.
7. Confirm every regulation in section 15 against the current published version before issue.