Data Cleansing for Healthcare Organisations: NHS Records, Patient Data and Compliance

How NHS trusts and private healthcare providers can improve patient record quality, deduplicate across EMIS and SystmOne, and meet DSP Toolkit requirements.

Why Patient Data Quality Is a Clinical and Compliance Issue

Poor data quality in healthcare is not merely an administrative inconvenience — it carries genuine clinical risk. Duplicate patient records, outdated contact details, and mismatched identifiers across clinical systems can lead to missed appointments, delayed diagnoses, incorrect medication records, and in the worst cases, patient safety incidents. For NHS trusts, GP federations, and private healthcare providers operating across England, maintaining accurate, deduplicated patient data is both a regulatory obligation and an ethical imperative.

The NHS generates and manages one of the largest repositories of personal data in the world. Yet despite significant investment in electronic patient record (EPR) systems over the past two decades, data quality problems remain stubbornly persistent. A 2023 audit by NHS England found that duplicate patient records were present in virtually every acute trust's Patient Administration System (PAS), with some organisations carrying duplication rates of 3–5% — meaning tens of thousands of affected records.

How Duplicate Records Arise in NHS Systems

Patient record duplication typically originates from a handful of predictable sources:

• Emergency registrations: When a patient arrives at A&E without their NHS number, a new record is often created rather than matched to an existing one. These "emergency" records frequently persist long after the patient's permanent record could have been linked.
• Name and date of birth variations: Records created for "William Smith" and "Bill Smith" may refer to the same person. Hyphenated surnames, name changes following marriage or divorce, and simple data entry errors all contribute.
• System migrations: When a trust moves from one EPR platform to another — or merges with another organisation — patient records from both systems are loaded into the new environment. Without rigorous matching logic, this process reliably creates duplicates.
• Cross-organisational referrals: A patient referred from a GP practice on EMIS to a hospital using SystmOne or a bespoke PAS may end up with independent records in each system, with no automated link between them.
• Temporary residents and non-registered attendees: Walk-in centres, sexual health clinics, and urgent treatment centres often create standalone records for patients who are already registered elsewhere in the system.

EMIS, SystmOne and the Cross-System Challenge

The majority of GP practices in England operate on either EMIS Web or SystmOne, and both systems maintain their own patient demographics data. When patients move between practices, change their name, or are referred to secondary care, their records do not automatically synchronise. The NHS number — the unique patient identifier introduced in the 1990s — should in theory act as the anchor for record matching across systems, but it is frequently absent or incorrectly recorded in legacy data.

For organisations managing data across both EMIS and SystmOne environments, a structured cleansing exercise typically involves:

• Extracting demographic data from each system in a standardised format
• Running probabilistic matching against the NHS Personal Demographics Service (PDS) to confirm NHS numbers and retrieve authoritative name, date of birth, and address data
• Identifying pairs or clusters of records that are likely to refer to the same patient, using matching algorithms that account for spelling variations, address changes, and partial data
• Presenting matched pairs to clinical or administrative staff for review and confirmation before any record merge is committed

The merge process itself must be handled carefully. Merging the wrong records creates a new clinical risk — a patient's medication history, allergy records, or test results could be attributed to the wrong person. Human review at the final stage is non-negotiable.

Address and Contact Validation in Patient Records

Outdated or incorrect address data undermines appointment letters, prescription delivery, and population health outreach. NHS organisations are required to make reasonable efforts to maintain current contact details, and the Royal Mail's Postcode Address File (PAF) provides the authoritative reference point for validating and standardising UK addresses.

A structured address cleansing exercise for patient records should include:

• Matching existing address strings against PAF to confirm validity and apply standardised formatting
• Checking against the NHS Change of Address notification service where available
• Flagging records where address data is clearly incomplete, outdated, or implausible (e.g., postcodes that do not exist, or flat numbers without a building name)
• Validating telephone numbers against Ofcom's numbering plan to identify disconnected or incorrectly formatted numbers

Data Security and Protection Toolkit Requirements

All organisations that access NHS patient data — including NHS trusts, GP practices, independent providers, and third-party suppliers — are required to submit an annual self-assessment against the Data Security and Protection (DSP) Toolkit. Formerly known as the IG Toolkit, the DSP Toolkit is published by NHS England and sets out mandatory standards for data handling, security, and governance.

Several DSP Toolkit standards directly relate to data quality:

• Standard 4 (Data Quality): Organisations must demonstrate that they have processes in place to identify, report, and address data quality issues. This includes maintaining accurate patient demographics and acting on data quality alerts from the NHS Spine.
• Standard 7 (Continuity of Care): Accurate and complete records are a prerequisite for safe continuity of care, particularly where patients are transferred between care settings.
• Standard 10 (Accountability and Audit): Any data cleansing activity must be documented, with evidence of the processes used, decisions made, and outcomes achieved.

When engaging an external data cleansing provider, healthcare organisations must ensure that a Data Processing Agreement (DPA) is in place and that the supplier has demonstrated compliance with relevant NHS data security standards. Processing identifiable patient data outside the NHS network requires careful consideration of data transfer mechanisms and must be approved by the organisation's Caldicott Guardian or Senior Information Risk Owner (SIRO).

Practical Steps for a Healthcare Data Cleansing Project

For NHS and private healthcare organisations embarking on a patient data cleansing programme, the following approach provides a sound starting framework:

1. Scope and baseline: Define which systems and record sets are in scope. Run an initial data profiling exercise to quantify the scale of duplication, incompleteness, and formatting issues before committing to a full project.
2. Governance sign-off: Obtain approval from your Caldicott Guardian, SIRO, and DPO. Confirm that your Data Processing Agreement with any external supplier is current and covers the specific activities planned.
3. Pseudonymisation for initial matching: Where possible, use pseudonymised or tokenised data for the matching and deduplication stage, only reverting to identifiable data at the final review and merge stage.
4. Staged rollout: Process records in manageable batches, beginning with the highest-risk or highest-volume areas. Review outcomes before proceeding to the next batch.
5. Prevention at point of entry: Implement NHS number verification at registration, real-time address validation at data entry, and training for administrative staff on data quality standards. Cleansing is only valuable if new poor-quality data is not continuously entering the system.

The Business Case for Cleaner Patient Data

Beyond compliance, there is a compelling operational case for investing in patient data quality. NHS trusts that have undertaken systematic deduplication programmes have reported measurable reductions in Did Not Attend (DNA) rates for outpatient appointments, attributable in part to appointment letters reaching patients at their correct address. Accurate contact data also underpins effective patient recall programmes for chronic disease management, cancer screening, and vaccination campaigns — activities where incomplete data translates directly into worse health outcomes.

For private healthcare providers, data quality directly affects revenue: billing errors, insurance claim rejections, and failed payment communications are frequently rooted in inaccurate patient records. Investing in a structured cleansing programme typically pays for itself within a single billing cycle.