UK Web Scraping Compliance Checklist: 15 Steps to Stay Legal

Quick answer: before a UK web scraping project starts, run these 15 checks. They cover the same four legal areas as our full compliance guide — data protection, access rights, intellectual property, and governance — reduced to a list you can work through project by project.

1. Establish a lawful basis before scraping personal data

Before scraping personal data, establish a lawful basis under Article 6 of GDPR. Legitimate interests is the most commonly used basis for web scraping — it requires balancing your interests against data subjects' rights, and suits market research and competitive analysis. Consent requires explicit, informed opt-in. Contractual necessity covers processing needed for a contract you already have with the person. See the full GDPR compliance section for detail on each.

2. Apply the seven UK GDPR data protection principles

All scraping activity must comply with: lawfulness, fairness and transparency; purpose limitation; data minimisation; accuracy; storage limitation; integrity and confidentiality; and accountability. These aren't optional extras — they're the baseline any processing of personal data has to clear.

3. Review the target site's Terms of Service

UK courts have shown increasing willingness to uphold ToS restrictions on automated access. Before deploying a scraper, locate the target site's Terms of Service, Privacy Policy, and any Acceptable Use Policy, and search for "automated", "scraping", "crawling", "robots" and "commercial use". Many platforms explicitly prohibit data extraction for commercial purposes.

4. Check and respect robots.txt

robots.txt isn't legally binding in itself, but courts and regulators treat compliance with it as strong evidence of good faith. A site that explicitly disallows crawling is communicating a clear intention to restrict automated access — ignoring that significantly increases legal exposure. Honour Crawl-delay directives too, and re-check on ongoing projects as policies change.

5. Never bypass CAPTCHAs or authentication

Programmatically solving or circumventing CAPTCHAs is a strong indicator of intent to exceed authorisation and may constitute an offence under section 1 of the Computer Misuse Act 1990. The same applies to credential stuffing or any other circumvention of an access control.

6. Stay off login-gated content without permission

Scraping behind a login wall without explicit permission carries significant Computer Misuse Act risk, regardless of whether a technical barrier was formally "broken" — if the ToS prohibits automated access, a court may find that access unauthorised on that basis alone.

7. Rate-limit requests and identify your bot

Configure a descriptive User-Agent string naming your bot, organisation and a contact address — masquerading as a standard browser undermines your good-faith defence. As a general rule, stay below one request per second for sensitive or consumer-facing sites, and stop immediately if you see a cease-and-desist or HTTP 429 responses at scale.

8. Confine scraping to factual data, not expressive content

Original literary, artistic or editorial content — article text, product descriptions, photographs — is automatically protected by copyright under the CDPA 1988 from the moment of creation. The UK's originality threshold is low: if a human exercised skill and judgement in creating it, it's likely protected. Stick to factual data where you can, or rely on the Text and Data Mining exception (section 29A CDPA) for non-commercial research.

9. Check database-right exposure on database-heavy sites

The UK retained the sui generis database right post-Brexit. It protects databases built with substantial investment in obtaining, verifying or presenting their contents — systematically extracting a substantial part can infringe the right even where individual records are factual and unoriginal. Price comparison sites, property portals and job boards are typical examples.

10. Run a four-factor risk assessment before each project

Score data type, volume, website sensitivity and commercial use from 1–3 each. A total of 4–6 is low risk and can proceed with standard documentation; 7–9 is medium risk and needs a written legal-basis assessment plus senior sign-off; 10–12 is high risk and needs legal review before any data is collected. See the full risk assessment framework for the scoring matrix.

11. Apply data minimisation at the point of collection

Collect only the personal data necessary for your stated purpose. Don't harvest email addresses, names or profile data speculatively — filter at the point of collection rather than post-hoc.

12. Keep audit logs of every scraping job

Log the target URL, date and time, volume of records collected, fields extracted, and the lawful basis relied upon for every job. These records are what determines the outcome if your activities are later challenged by a site operator, a data subject, or a regulator.

13. Maintain a Records of Processing Activities entry

Under UK GDPR Article 30, any scraping activity touching personal data needs a ROPA entry: purpose of processing, categories of data subjects and data, lawful basis, retention period, security measures, and any third parties data is shared with.

14. Set a retention and deletion schedule before you collect

Define a retention period for every dataset before collection begins. Don't hold scraped data indefinitely — implement automated deletion or pseudonymisation once the purpose is fulfilled, document the decision in your ROPA entry, and review it annually.

15. Have an incident-response plan ready

If a scraper receives a cease-and-desist or formal complaint, know the procedure before it happens: immediate suspension of the relevant crawl, preservation of logs, escalation to legal counsel, and a designated point of contact. Do not delete logs or data when challenged — that may constitute destruction of evidence.