SOP: Operational Incident Management for GxP Computerized Systems

This is a ready-to-use SOP for handling incidents on a GxP computerized system that is already in production, live-system events such as an outage, a data-entry rejection, an audit trail gap, or an unexpected calculation result. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and route it through your normal document control, review, and approval. A worked filled specimen follows the template. Verify each cited regulation against the current source before you rely on it, and treat this as an educational reference to adapt rather than as legal or regulatory advice.

This procedure covers a different moment than two related documents that are easy to confuse with it. The SOP: Validation Test Incident Management governs failures found while a protocol is being executed, before the system is released to production; this SOP starts only after that release. The general SOP: Deviation Management governs the investigation and disposition once a GxP impact is confirmed; this SOP governs everything before that point, the detection, classification, immediate response, and the impact assessment that decides whether a deviation is opened at all.

Document control header

1. Purpose

This procedure defines how <<FILL: COMPANY NAME>> detects, classifies, responds to, investigates the GxP impact of, and closes incidents affecting GxP computerized systems that are in operational, production use. The objective is that every incident, regardless of how it was first noticed, is captured promptly, assessed for whether it touched a GxP record, escalated to a deviation where warranted, and closed with evidence that the system and its data can still be relied on.

2. Scope

This procedure applies to unplanned events affecting any computerized system recorded as GxP in the system inventory maintained under <<FILL: SOP-ID or register reference>>, at the sites named in the header, from the point the system enters production use until it is formally retired. It covers system unavailability, functional malfunctions, data-quality anomalies, audit trail failures, and security events that present operationally (unexpected lockouts, suspicious activity) rather than as a planned security investigation.

It does not cover:

• Test discrepancies found during protocol execution before production release, governed by <<FILL: SOP-ID for validation test incident management>>.
• Deliberate, planned modifications to a system, governed by <<FILL: SOP-ID for change control>>. Where an incident is caused by a change gone wrong, the change record and this incident record are cross-referenced, not merged into one.
• The investigation and disposition activity that follows once a deviation is opened; that work is governed by <<FILL: SOP-ID for deviation management>>. This procedure hands off to it at the point in section 5.5 where a deviation is confirmed.

3. Responsibilities

4. Definitions

• Incident: an unplanned event affecting a GxP computerized system’s availability, functionality, or data quality. It is distinct from a deviation: an incident is what happened; a deviation is the quality record opened once GxP impact is confirmed.
• Severity: the operational classification (critical, major, minor) assigned at detection and reassessed as facts emerge, described in section 5.2.
• GxP impact: whether the incident affected the creation, integrity, availability, or reliability of any GxP record, as distinct from mere inconvenience or downtime with no data consequence.
• Workaround: a pre-validated fallback procedure, typically paper-based, that keeps a GxP activity running while the system is unavailable, invoked and reconciled per section 5.7.
• Restoration: the point the system returns to normal operation, which is not the same point as the incident being closed; restoration ends the outage, closure ends the record.
• Reconciliation: the documented, independently verified transfer of any manual or workaround data into the system once it is restored.

5. Procedure

5.1 Detect and report

1. Any user, administrator, or monitoring alert may be the first detection of an incident. The discoverer reports it immediately through <<FILL: channel, e.g. the IT service desk, an incident hotline, the monitoring platform's alert queue>>.
2. The discoverer does not attempt an undocumented workaround or repeated retries before reporting. A brief factual note of what was observed (what was tried, what happened, the time) is captured at the point of detection, not reconstructed afterward.
3. An incident record is opened with a unique identifier within <<FILL: e.g. 1 hour for critical, same business day for major and minor>> of detection.

5.2 Classify severity

Assign one of three severities at intake, and re-assess as facts emerge. A severity may be raised or lowered as the picture becomes clearer, with the reason recorded.

5.3 Respond and restore

1. The system administrator or IT on-call performs first response: contain the issue, prevent further data loss or corruption, and work toward restoration.
2. For a critical or major incident, the process owner decides, with the system owner, whether a documented workaround is invoked per section 5.7 while restoration is in progress.
3. Restoration is confirmed and timestamped. Restoration ends the outage; it does not close the incident record, which still requires the impact assessment, any deviation, and reconciliation to be complete.

5.4 Escalate and notify

1. Follow the notification chain defined for the system’s criticality tier: at minimum, the system owner and QA are notified of every critical incident on detection, and of every major incident within the shift.
2. For a critical incident during a release window, batch-critical activity, or another time-sensitive GxP process, notification also reaches whoever is authorized to approve a workaround, so that decision does not wait on the standard escalation chain.
3. Provide status updates at a defined cadence while the incident remains open, <<FILL: e.g. every 30 to 60 minutes for critical>>, so the record shows the response was active rather than reconstructed after the fact.

5.5 Assess GxP impact

1. Every incident on a GxP system gets a documented impact assessment, regardless of severity, answering: did this affect any GxP record, existing or in progress, and if so, which records and how.
2. Where the answer is no, meaning no data already in the system was altered, lost, or made unreadable, and no record created during the event is in question, document that conclusion with its basis, restore service, and proceed to closure without opening a deviation.
3. Where the answer is yes, or where an audit trail, timestamp, or access control gap means the normal evidence needed to answer the question is itself missing, open a deviation under <<FILL: SOP-ID for deviation management>> immediately. Do not wait for the incident record to be finished. Cross-reference the deviation number on the incident record and the incident number on the deviation record.
4. The impact assessment is written at the time, not reconstructed weeks later. A conclusion drafted from memory after the fact is treated as unreliable evidence at periodic review.

5.6 Assess scope

For any incident that opens a deviation, assess whether the same cause could affect other systems, other functions of the same system, or data already produced before the incident was noticed. An audit trail gap discovered in one module often applies to a shared logging service used elsewhere; check before assuming the finding is contained.

5.7 Invoke and reconcile a workaround

1. A workaround is invoked only where a validated fallback procedure exists for the activity; an improvised paper substitute invented during the outage is not a workaround, it is a new, undocumented process and is itself a finding.
2. The workaround record (paper form, manual log) meets the same ALCOA+ expectations as the system record it temporarily replaces: attributable, legible, contemporaneous, original, and accurate, with the person and time captured at the point of entry.
3. Once the system is restored, the workaround data is transcribed or entered into the system and independently verified by someone other than the person who captured it, with the original workaround record retained as source data, not discarded once transcribed.
4. Reconciliation is itself documented: what was transferred, by whom, verified by whom, and any discrepancy found and how it was resolved.

5.8 Investigate root cause

1. For a critical or major incident, or any incident that opened a deviation, determine root cause proportionate to risk. A vague cause such as “system was slow” is not acceptable; the cause must be specific enough that a corrective action can act on it.
2. Where the incident is a vendor product defect, the vendor’s own root cause analysis is reviewed and assessed by the system owner or SME rather than accepted without review.
3. Raise a CAPA under <<FILL: SOP-ID for CAPA management>> where the cause could recur or affects more than the single instance.

5.9 Close the incident

An incident closes only when all of the following are complete and recorded:

1. Restoration is confirmed and timestamped.
2. The GxP impact assessment is documented with its conclusion and basis.
3. Where a deviation was opened, it is cross-referenced and its status noted (open incidents do not wait for the deviation to fully close, but the link must exist).
4. Any workaround used is reconciled and independently verified.
5. Root cause and CAPA references are recorded where applicable.
6. The system owner signs the closure; QA approves closure for every critical and major incident.

5.10 Feed the operational record

1. The incident is logged in the system’s operational incident register so it is visible at the next periodic review, not just in a ticketing system that periodic review does not look at.
2. Trend the incident log at the cadence used for operational monitoring: a repeated incident on the same function is a pattern, not a series of unrelated coincidences, and is escalated as such even where each individual occurrence was minor.

6. Acceptance criteria

• Every incident is logged with a timestamp at or near detection, not reconstructed afterward.
• Every incident carries a severity classification with a stated basis, re-assessed if the facts changed.
• Every incident has a documented GxP impact assessment with an explicit conclusion, written at the time.
• Every incident where the answer to the impact assessment is yes, or where the evidence needed to answer it is itself missing, has a cross-referenced deviation.
• Any workaround used is a pre-validated procedure, not an improvisation, and its data is reconciled and independently verified before the incident closes.
• Root cause and CAPA are recorded for every critical or major incident and for any recurring minor incident.
• QA has independently reviewed and approved closure of every critical and major incident.
• The incident register is a visible input to the next periodic review.

7. Records generated

8. References

21 CFR 211.68(b) (equipment and records shall be maintained and protected), for the underlying expectation that a system failure affecting records is addressed and documented. 21 CFR 211.192 (production record review; investigation of unexplained discrepancies), applied by analogy to a computerized system incident that touches a GxP record. EU GMP Annex 11, Computerised Systems, clause 13 (incident management: incidents, not only system failures, should be reported and assessed, with root cause identified for critical incidents and corrective and preventive action taken). A substantially expanded draft revision was published for consultation in July 2025 and had not been finalized as of this writing; confirm the current in-force version before citing clause numbers. ICH Q9(R1), Quality Risk Management, for the risk basis of severity classification and investigation depth. ISPE GAMP 5, Second Edition (2022), for the operational-phase treatment of incident management within the system lifecycle. Reference only; do not reproduce its text.

Confirm the current version and clause numbers of each reference before issue.

9. Revision history

10. Approvals

Filled specimen

The following shows an incident on a clinical Electronic Data Capture platform. The company, system, names, and numbers are illustrative; replace them with your own.

What made this defensible: the coordinator reported instead of quietly re-entering data, the system was not treated as merely “glitchy,” the impact assessment reached a clear yes and a deviation opened the same day, the scope check went looking for the same pattern elsewhere instead of assuming one site was unique, and the interim compensating control covered the gap until the permanent fix shipped.

Common inspection findings this SOP prevents

• An outage or malfunction is logged only as an IT help-desk ticket, with no GxP impact assessment anywhere in the record.
• An incident’s severity is downgraded after the fact to avoid the escalation and QA review a higher severity would require.
• A workaround was used but was never a validated procedure, so the fallback data itself cannot be relied on.
• Workaround data was transcribed into the system but never independently verified, so a transcription error propagated silently.
• The GxP impact assessment is written weeks after the event, from memory, once an audit or inspection prompts someone to produce one.
• An incident is closed on restoration alone, with the impact assessment, deviation cross-reference, or reconciliation still outstanding.
• The same incident recurs three or four times, each logged and closed independently, with no one connecting the pattern until an inspector does it for them.
• The operational incident register and the deviation log carry different counts for the same period, with no reconciliation between the two.

How to adapt this SOP

1. Set your document number, owner, effective date, and the related-procedure references in the header and in section 2.
2. Point every cross-reference to your real change control, deviation, CAPA, and business continuity procedures.
3. Set your severity definitions and response targets in section 5.2 to match your actual operating capability; a target you cannot meet is worse than a longer one you can.
4. Name your actual notification channels and escalation roles in sections 5.1 and 5.4.
5. Confirm your workaround procedures are validated and referenced by SOP number before relying on section 5.7; a workaround with no prior validation is not usable under this procedure as written.
6. Confirm every regulation in section 8 against the current published version before issue, including the in-force status of the Annex 11 revision.