AI Validation in Pharmacovigilance: Pre-Go-Live Testing

AI Validation in Pharmacovigilance: How to Safely Deploy Automation

The demonstration usually goes well.

The AI reads an adverse event report, extracts relevant fields, proposes coding, identifies possible seriousness and produces an output in seconds.

Then someone asks, "When can we go live?"

The correct answer depends on a different question: what has the organisation proven?

A successful demonstration shows that the technology can perform a task. It does not prove that it performs reliably across real pharmacovigilance data, protects regulated records, handles unusual inputs or supports effective human review.

Automation does not transfer accountability. When an AI-enabled tool supports case intake, triage, literature screening or signal activities, the MAH remains responsible for the process and its regulatory outcomes.

AI validation in pharmacovigilance is the evidence required to place automation into a live safety process with appropriate control.

Key Takeaway

Before AI automation goes live, validate the computerised system and evaluate the model for its defined use. Testing must cover data integrity, representative datasets, critical errors, human review, privacy, audit trails, failure handling, vendor controls and ongoing monitoring.

Why Computer System Validation Is Not Enough

Computer System Validation (CSV) remains essential. It establishes whether software operates consistently according to approved requirements and its intended use.

CSV may cover configuration, access controls, interfaces, electronic records, audit trails, backup, recovery and security. AI introduces an additional question: are the model's outputs sufficiently reliable for the defined PV activity?

Validation layer	Question to answer
System validation	Does the application operate as specified?
Model evaluation	Are outputs reliable for the defined PV task?
Workflow validation	Can users detect and correct unreliable outputs?
Lifecycle assurance	Will performance remain controlled after changes?

AI models may perform differently across data types, languages and patient populations. Validation must therefore extend beyond confirming that the software functions.

The EMA's AI framework places AI use within existing medicines, data-protection and regulatory requirements. The joint EMA-FDA principles for good AI practice also call for risk management across the medicine lifecycle.

Define the Intended Use

The organisation cannot validate "the AI" in general. It must validate a defined use within a controlled process.

A tool may identify safety information, extract case data, recommend MedDRA coding, classify seriousness, detect duplicates, screen literature or generate draft narratives.

For each use, define the users, input sources, expected output, downstream decision, degree of automation, required human intervention and consequences of an incorrect result.

An inaccurate narrative suggestion that receives complete medical review presents a different risk from a false-negative intake decision that prevents safety information from entering the system.

Apply Risk-Based Validation

Validation rigour should reflect the potential effect of failure on patient safety, data integrity and regulatory compliance.

Assess whether an incorrect output could affect reportability, seriousness classification, reporting timelines, signal evaluation or benefit-risk decisions. Consider whether the error is detectable, reversible and limited to one record or capable of affecting the whole system.

This analysis determines the required test depth, acceptance criteria and level of human oversight.

Test the Data and Model

Testing must use representative data rather than a convenient collection of clean examples.

The test set should include incomplete reports, follow-up information, ambiguous language, multilingual content, poor-quality documents and rare but important scenarios.

Document the data source, selection method, expected answers and separation from model development.

Overall accuracy is rarely sufficient:

PV use case	Relevant measures
Safety-information detection	Sensitivity and false-negative rate
Seriousness classification	Class sensitivity and disagreement analysis
Duplicate detection	Recall, precision and unresolved risk
Literature screening	Relevant-record recall and exclusion errors
Coding support	Agreement, correction and significant errors
Narrative generation	Factual accuracy and source traceability

Acceptance criteria must reflect the intended use. A false negative may carry greater regulatory and patient-safety consequences than a false positive that receives review.

Validate Human Oversight

Human-in-the-loop pharmacovigilance is not automatically an effective control. Reviewers may accept recommendations without sufficient challenge, particularly when volumes are high.

Testing must confirm that reviewers can access source information, understand the tool's limitations, reject recommendations, escalate high-risk cases and document decisions.

The organisation also needs enough trained capacity to perform the stated oversight. A procedure requiring full human review provides limited assurance if workloads make that review unrealistic.

Protect Privacy and Data Integrity

ICSRs may contain identifiable patient, reporter and health information. Before using a third-party AI tool, establish how that information will be processed and protected.

Assess data minimisation, access controls, encryption, retention, deletion, cross-border transfers, subprocessors and whether submitted information may train the vendor's model.

Contracts must define ownership, confidentiality, incident notification, record availability and system-change responsibilities. These controls are particularly important when service providers operate computerised pharmacovigilance systems.

Test Failure, Recovery and Change

Test AI-enabled software within the complete PV workflow.

Scenarios should include interface interruption, incomplete input, duplicate transmission, delayed processing, unexpected output and service unavailability. Manual fallback, backlog management, reconciliation and recovery must also be demonstrated.

Validation does not end at release. Model retraining, vendor updates, configuration changes and new data sources may affect the original evidence.

Define revalidation triggers, version control, release assessment, performance monitoring and rollback arrangements. Monitor error trends, human overrides, false negatives, deviations and performance across relevant data categories.

Pre-Go-Live Decision Checklist

Before release, confirm that:

• The intended and prohibited uses are approved
• The system and model have been tested
• Critical error risks have been assessed
• Human oversight has been demonstrated
• Privacy and data-integrity controls are approved
• Interfaces, fallback and recovery are verified
• Vendor responsibilities are documented
• Procedures and training are effective
• Monitoring and revalidation plans are active
• The release decision is formally authorised

Go-Live Begins the Next Stage of Validation

AI adoption should improve safety operations without weakening accountability, traceability or regulatory control.

PVCON Consulting supports organisations through PV QMS services, pharmacovigilance consulting and independent PV audits.

Support can include risk assessment, validation governance, vendor oversight, procedural integration and inspection readiness for AI-enabled PV processes.

Go-live is not the end of validation. It is the point at which controlled evidence becomes live regulatory performance.

PVCON Consulting supports pharmaceutical, biotechnology, CRO, and medical device organizations through specialized services including GxP Audits, PV Audits, GCP Audits, Other GxP Audits, Pharmacovigilance Consulting, PV Quality Management System support, PvOIC services, Regulatory Intelligence, Medical Writing, Aggregate Report Writing, Clinical Safety Documents, RMP and REMS Writing, PSMF Management, and Training & Upskilling initiatives such as Training Matrix, Regulatory Compliance Training, PV Boot Camp, and Customized Learnings.

Our expertise helps organizations strengthen drug safety operations, improve inspection and audit readiness, and keep PSMF documentation compliant, accurate, and aligned with real-world PV system practices and regulatory expectations.

To discuss validation requirements for an AI-enabled PV process, you can contact our team or learn more about us.