SOP: Validation Test Incident Management

This is a ready-to-use SOP. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and route it through your normal document control, review, and approval. A worked filled specimen follows the template so you can see how a completed version reads. Verify each cited regulation against the current published source before you rely on it, and treat this as an educational reference to adapt rather than as legal or regulatory advice.

This procedure governs the process. The individual deviation record it produces is covered separately in Validation Deviation / Discrepancy Record, and the two documents are designed to be used together.

Document control header

1. Purpose

This procedure defines how <<FILL: COMPANY NAME>> records, logs, triages, classifies, escalates, investigates, corrects, re-executes, and closes test incidents arising during the execution of validation and qualification protocols.

The objective is that every result which did not meet its acceptance criterion, and every unplanned event during execution, is captured as it happened, assessed for what it means, and resolved through a traceable path that a reviewer can follow from the first observation to the final disposition. A validation package asserts that a system or process is fit for its intended use. That assertion only holds if the failures behind it are visible and explained.

2. Scope

This procedure applies to execution-phase incidents in all validation and qualification activities at the sites named in the header, including:

• Computerized system validation (IQ, OQ, PQ, UAT) for GxP systems.
• Equipment and utility qualification (DQ, IQ, OQ, PQ).
• Analytical method validation and method transfer execution.
• Process validation and process performance qualification runs.
• Cleaning validation execution.

It applies from the moment protocol execution begins until the validation summary report is approved. It does not cover:

• Deviations occurring in routine GxP operation after release, which are governed by <<FILL: SOP-ID for deviation management>>.
• Changes to an already validated system, governed by <<FILL: SOP-ID for change control>>.
• Defects found during supplier-side development and testing before protocol execution starts, which are handled under the supplier’s own defect process and assessed under <<FILL: SOP-ID for supplier assessment>>.

3. Responsibilities

4. Definitions

• Test incident: the neutral, first-level factual record that an actual result did not match the expected result, or that something unplanned occurred during execution. It is an observation, not yet a judgment about fault.
• Discrepancy: used in this procedure as a synonym for test incident. Where local terminology differs, map the terms in the validation plan.
• Deviation: a formal quality management system record opened when a test incident meets the escalation criteria in section 5.4, requiring investigation and QA disposition.
• Test artifact correction: a documented correction to the test instrument itself, meaning the protocol step, the expected result, the test data, or the evidence capture, where the system under test behaved correctly. It is retained in the incident log rather than escalated, only where the validation plan permits and QA has agreed the approach in advance.
• Defect: a confirmed fault in the software, configuration, equipment, method, or design that caused the failure.
• Re-execution (retest): a fresh, separately dated and signed execution of a test step or test case after a documented correction. It is a new record. It never overwrites the original.
• Regression scope: the set of previously executed tests re-run because a correction could plausibly have affected them.
• Open item: an incident that is not closed-resolved at the time the summary report is prepared, carried forward with a risk position, an owner, and a target date.
• Testing into compliance: re-running a test until it passes without an understood and corrected cause. Prohibited under this procedure.

5. Procedure

5.1 Set escalation criteria before execution starts

Escalation criteria are defined during validation planning and written into the approved validation plan or protocol, before the first test step is executed. They are not decided while a team is looking at its first failure, because at that moment the decision is contaminated by schedule pressure.

The validation plan states, at minimum:

QA reviews and approves the escalation criteria as part of plan approval. If execution begins without approved escalation criteria, execution is suspended until they are approved, and that suspension is itself recorded as an incident.

Where the risk-based rationale for test depth is documented under a computer software assurance approach, keep the escalation criteria consistent with the assurance rationale. FDA’s guidance “Computer Software Assurance for Production and Quality Management System Software”, issued 3 February 2026, describes a risk-based approach to assurance effort. Effort scaled by risk is not a licence to record less when something fails. See Computer software assurance (FDA).

5.2 Record the observation contemporaneously

The instant an actual result does not match the expected result, the executor records it. Before any retry. Before any adjustment. Before asking anyone whether it “really” failed.

Record the following on the execution record at the point of observation:

1. Protocol number, test case, and step identifier.
2. The expected result exactly as written in the approved protocol.
3. The actual result, stated as fact with values. “Field accepted 105 with configured range 0 to 100” is usable. “Did not work” is not.
4. Objective evidence reference: screenshot, printout, system log extract, chart, or photograph, with the capture time and the identity of who captured it.
5. Date, time, and the executor’s signature or unique user identity.
6. The environment in effect: build or firmware version, instance or environment name, configuration set, test data set, user role used.
7. The incident number assigned under section 5.3.

Rule 5.2.1 (contemporaneous recording, no retry first). A test step may not be re-run, refreshed, reloaded, restarted, or re-entered before the failing observation is recorded and an incident number is assigned. A second attempt made before the first is recorded destroys the original observation and is treated as a data integrity event under <<FILL: SOP-ID for data integrity investigation>>.

Rule 5.2.2 (the failed step stays FAIL). The failing step is marked FAIL on the original execution record, annotated with the incident number, and left in place. It is not erased, blanked, obliterated, overwritten, or corrected to PASS. Where a single-line correction is unavoidable on paper, apply the good documentation practice convention: single line through, initial, date, reason, with the original entry still legible.

Rule 5.2.3 (never edit the expectation to match the actual). The expected result recorded in an approved protocol is never changed to match observed system behaviour as a way of closing a failure. Where an expected result was genuinely wrong, it is corrected under section 5.7.2 through an approved, justified change, and the step is then re-executed against the corrected expectation.

5.3 Open the incident in the test incident log

Assign a unique sequential incident number from the log for the protocol or the validation project. Enter the incident the same working day it is observed.

Test incident log fields

The log is a controlled record. It is reconciled against the protocol execution pages and against the QMS deviation register before the summary report is issued.

5.4 Triage

Triage is performed by the validation lead, with the SME where a technical judgment is needed, within <<FILL: target, e.g. 1 working day>> of the incident being logged. Work the questions in order and record the answer to each.

Question 6 exists because premature classification is a common failure. An incident is allowed to sit as “under investigation” while cause is established. It is not allowed to sit as “minor, tester error” by default.

5.5 Classify and route

Assign one classification. The classification determines the route, the required investigation depth, and the approval level. QA confirms or overrides every classification; a classification proposed by the executor alone is not final.

Rule 5.5.1 (no default to C2). A pattern of C2 closures with no analysis of why the error was possible is treated as an inadequate investigation. The validation lead reports the C1 to C6 distribution at each execution status review so a drift toward the low-effort categories is visible while it can still be corrected. See Human error in deviations.

Rule 5.5.2 (parallel execution). Testing of unrelated test cases may continue while an incident is open, provided the validation lead records that the open incident does not affect those cases. Testing of any test case that depends on the affected function is suspended until the incident is dispositioned.

5.6 Assess scope and impact

For every incident classified C3, C4, C5, or C6, complete a scope and impact assessment before defining the correction. Use Test Failure Scope and Impact Assessment.

The assessment answers four questions and records the evidence for each:

1. Same defect elsewhere in the system? Other functions sharing the same code path, calculation, configuration parameter, interface, template, or user role.
2. Other test cases affected? Including test cases already executed and recorded as PASS whose result depended on the defective function.
3. Other configurations or instances affected? Other methods, recipes, products, lines, instruments, sites, or tenants built the same way or in the same configuration session.
4. Data already created affected? Any GxP data generated in this environment or in production before the defect was found, and its disposition.

An incident where the scope assessment concludes “this step only” states the basis for that conclusion. An unexamined assumption that a failure is local is one of the most commonly cited weaknesses in validation packages.

5.7 Investigate root cause and define the correction

5.7.1 Root cause proportionate to risk

Investigation depth is proportionate to classification and risk, consistent with quality risk management principles described in ICH Q9.

Use a structured technique appropriate to the complexity. See Root cause analysis techniques and Root Cause Analysis Toolkit. Record the technique used and the evidence that confirms the cause, not just the conclusion.

A root cause statement is acceptable only if it names a specific, verifiable cause. “The system rounded incorrectly” restates the symptom. “The reporting precision parameter was left at the vendor default of three decimal places because the configuration specification did not state the required value” is a cause a corrective action can act on.

5.7.2 Define the correction by class

Where a defect will not be fixed, an accept-with-justification disposition is permitted only with a documented technical rationale, a residual risk assessment, the system or process owner’s acceptance, and QA approval, at the authority level in section 5.9. Record it as an open item under section 5.10.

5.8 Authorize and perform re-execution

Re-execution requires an approved authorization. Use Validation Retest / Re-execution Authorization.

Rule 5.8.1 (preconditions). A re-execution may be authorized only when all of the following are true:

1. The root cause of the original failure is documented and verified.
2. The corrective action is complete and its completion has been verified by someone other than the person who performed it.
3. The scope and impact assessment is complete for classes C3 to C6.
4. The re-test scope, including regression scope, is defined with a written justification.
5. The environment and build the re-test will run on are stated and confirmed.
6. The authorization carries the “not testing into compliance” attestation, signed.

Rule 5.8.2 (a re-execution is a new record). The re-execution is documented on a fresh, separately dated, timed, and signed execution record referencing the incident number. It does not overwrite, annotate over, or replace the original failed record. Both records remain in the package. The narrative reads: failed here, investigated, corrected, re-executed and passed there.

Rule 5.8.3 (regression is the default). Where the correction touched shared logic, a shared configuration object, an interface, a common calculation, a template, or a security role, regression testing is performed unless the validation lead and QA record a specific justification for not doing so. The cost of an unnecessary regression run is small. The cost of an undetected side effect found later is not.

Rule 5.8.4 (a failed re-execution). If the re-execution fails, it is recorded as a new incident referencing the original. It does not reuse the original incident number as a second attempt, because that hides the number of attempts made.

Rule 5.8.5 (attempt limit). Where <<FILL: number, e.g. 2>> re-executions of the same step have failed, execution of that test case is suspended and the incident is escalated to <<FILL: role, e.g. Head of Quality>> for a decision on whether the underlying design, requirement, or acceptance criterion is sound. Repeated re-execution without escalation is the shape testing into compliance takes.

5.9 Authority matrix

No person approves their own work. The executor of a step does not approve its re-execution authorization, and the person who implemented a correction does not verify it.

Replace the role names with your own titles and confirm the matrix does not conflict with your delegation of authority record.

5.10 Closure, and open items carried into the summary report

Every incident closes with a documented disposition. Record on the incident:

1. Final classification and the basis for it.
2. Verified root cause and the evidence supporting it.
3. Corrective action taken, with change control and CAPA references.
4. Scope and impact assessment conclusion.
5. Re-execution authorization reference, re-execution record reference, and result.
6. A statement that no other test result, no other data, and no other aspect of fitness for intended use is compromised, or a clear statement of what is affected and how it was handled.
7. Disposition: closed-resolved, closed with a justified open item, or closed as accept-with-justification with documented residual risk.

Open items. Any incident not closed-resolved when the validation summary report is prepared is carried into the report as an open item with:

Rule 5.10.1. No system, equipment, method, or process is released to GxP use with an incident that has not been assessed and dispositioned. An open item is permitted. An unassessed open item is not. The complete open-item list, with risk positions, is reproduced in the validation summary report. See Validation summary report and release and Validation Summary Report.

Rule 5.10.2 (reconciliation before report issue). Before the summary report is issued, the validation lead reconciles the test incident log against the protocol execution pages and the QMS deviation register, and records that every FAIL annotation on an execution page maps to a logged incident, and every logged incident maps to a disposition. QA verifies the reconciliation.

5.11 Audit trail verification of the execution timeline

For computerized systems, before closing execution, review the system audit trail for the execution period against the documented execution timeline. Confirm that:

• Record creation and modification times sit inside the documented execution window.
• Re-execution timestamps follow, and do not precede, the corrective action completion date.
• No test records were created or modified by an identity other than the documented executor.
• There are no audit trail gaps or disabled periods across the execution window.

Discrepancies found here are recorded as incidents in their own right, classified C6. See Audit trail design and review and Time stamps and system clock control.

6. Acceptance criteria

Execution-phase incident handling for a validation project is acceptable when all of the following are met:

7. Records generated

Retain all records for not less than <<FILL: retention period>> per the records retention schedule, and for at least the operational life of the system or process plus <<FILL: number>> years.

8. References

21 CFR 211.100(a) and 211.192 (production and control procedures; investigation of unexplained discrepancies). Part 211.192 requires that any unexplained discrepancy be thoroughly investigated and the investigation extended to other batches that may have been associated with the failure, with the conclusion and follow-up recorded. 21 CFR Part 11 (electronic records and electronic signatures), for the integrity of electronically captured test evidence. EU GMP Annex 15, Qualification and Validation (2015), which requires deviations from validation protocols to be documented with appropriate justification and quality risk management to be applied throughout. EU GMP Annex 11, Computerised Systems (2011, the version currently in force), for the requirement that deviations arising during validation are reported and assessed. A draft revision was published on 7 July 2025 with consultation closing 7 October 2025; it is not final and should not be cited as an in-force requirement. ICH Q9(R1), Quality Risk Management, for the risk basis of classification and investigation depth. ICH Q10, Pharmaceutical Quality System, for the CAPA and management review linkage. FDA guidance “Computer Software Assurance for Production and Quality Management System Software”, issued 3 February 2026, for the risk-based assurance context. FDA guidance on data integrity and compliance with drug CGMP (2018), MHRA GxP Data Integrity Guidance and Definitions (2018), and PIC/S PI 041, for the ALCOA+ expectations applied to the execution record itself. ISPE GAMP 5 Second Edition (2022), for the computerized system validation lifecycle and its treatment of test discrepancies. Reference only; do not reproduce its text.

Confirm the current version and clause numbers of each reference before issue.

9. Revision history

10. Approvals

Filled specimen

The following shows the incident log and disposition for one incident during the OQ of a manufacturing execution system supporting an autologous cell therapy process. The company, system, numbers, and names are illustrative.

Test incident log entry

How it ran

Recording. The analyst did not rescan. She captured the warning banner, captured the screen showing the next unit operation reached, pulled the MES event log extract, and wrote the observation on the execution page with the time. Step 5 was marked FAIL with “see TI-OQ-MES-018”. The step was left as written.

Triage. Question 1: the system behaved incorrectly, since the configured rule was a hard block, not a warning. Question 2: chain of identity in an autologous process is directly patient-safety relevant. Question 3: no record alteration, so not a data integrity event. Question 5: the same identity-check rule is used at four other unit operations. Classification C5, escalated to VAL-DEV-2026-044 the same day.

Scope. SIA-018 found the identity-check rule was implemented as a shared verification object referenced by five unit operations. Four of the five had been configured with severity “warning” instead of “block”. The fifth, at final fill, was configured correctly. Two OQ test cases already recorded PASS (TC-028 and TC-029) had exercised the same shared object at other unit operations with matching identifiers only, so they never challenged the block behaviour. Both were flagged for re-execution with a mismatch case. No GxP material had been processed on the QUAL instance, so no product data was affected.

Root cause. Five-whys against the configuration record and the design specification. The design specification stated “the system shall prevent progression on identity mismatch” but the configuration workbook offered severity values of “info”, “warning”, and “block” with a default of “warning”, and the configuration specification did not state the required value. The builder accepted the default. Verified by comparing the configuration export against the design specification.

Correction. CR-MES-2026-077 set severity to “block” on all five instances of the shared verification object. Change implemented by the MES administrator, verified by a second engineer against the configuration export. CAPA-2026-052 added an explicit severity column to the configuration specification template and a verification step in the build checklist so a default value cannot be silently accepted.

Re-execution. RTA-018 authorized re-execution of TC-031 step 5, plus regression re-execution of TC-028 and TC-029 with mismatch cases added, plus a new negative test at each of the other three unit operations. The validation lead recommended, the SME and system owner reviewed, QA approved, and because the classification was C5 the Quality Head countersigned. The “not testing into compliance” attestation was signed by the validation lead. All re-executions were recorded on OQ-MES-006 addendum 2, separately dated and signed on 16 July 2026, each referencing TI-OQ-MES-018. All passed.

Closure. VAL-DEV-2026-044 closed-resolved on 20 July 2026. The audit trail review under 5.11 confirmed the re-execution records were created on 16 July, after CR-MES-2026-077 was verified complete on 15 July, with no records created outside the documented execution window. No open item carried forward.

What made this defensible: the analyst did not rescan before recording, the scope question turned one unit operation into five, the root cause named a specific configuration default the specification never overrode, the corrective action fixed the specification template rather than only the one setting, the regression scope included two tests that had already passed, and every re-execution was its own dated record.

Common inspection findings this SOP prevents

• Test failures visible in the audit trail or system logs with no corresponding entry in the test incident log.
• Expected results changed on the protocol to match what the system actually did, with the step then marked PASS and no incident raised.
• Failed steps erased, blanked, or overwritten so the failure is not visible on the execution page.
• Re-execution records dated before, or on the same day as, a corrective action that demonstrably took longer to complete.
• Repeated re-execution of a failing step until it passed, with only the passing attempt recorded.
• Escalation criteria written after the first failure occurred, or never written at all, so what became a deviation was decided case by case.
• A pattern of incidents closed as “tester error” or “human error” with no analysis of why the error was possible.
• Scope assessed only on the failing step, with tests that already passed against the same defective function never revisited.
• Genuine GxP defects managed entirely in a project spreadsheet with no QMS deviation and no QA disposition.
• Classification assigned and closed by the same person who executed the test, with no independent QA review.
• A validation summary report issued while incidents remain in the log with no disposition, or open items listed with no risk position, owner, or target date.
• The test incident log and the QMS deviation register carrying different totals for the same project, with no reconciliation record.

How to adapt this SOP

1. Set your document number, owner, effective date, and the related-procedure references in the header and in sections 2, 5.7.2, and 5.10.
2. Map your own terminology onto the definitions in section 4. If your QMS calls the first-level record a “discrepancy” or an “observation”, change the term throughout and say so once in section 4 rather than mixing terms.
3. Tune the C1 to C6 classification table in section 5.5 to your severity language. Keep the link between class, investigation depth, and approval level, and keep the data integrity trigger independent of functional severity.
4. Rewrite the authority matrix in section 5.9 with your real job titles and check it against your delegation of authority record. Verify that no row lets one person both perform and approve.
5. Set the re-execution attempt limit in rule 5.8.5 and name the escalation role. Two is a common starting point; pick a number and hold to it.
6. Decide whether your validation plan permits C1 and C2 incidents to be retained as documented test-artifact corrections, and state that decision in the plan template so it is agreed before execution rather than argued during it.
7. Add the test incident log fields from section 5.3 to your execution protocol templates so a tester has fields to complete rather than a blank page.
8. For paper execution, define your correction convention explicitly under rule 5.2.2 and train executors on it before execution starts.
9. Set retention periods in section 7 against your records retention schedule.
10. Confirm every regulation in section 8 against the current published version before issue, in particular the status of the EU GMP Annex 11 revision, which was still in draft at the time of writing.