Web scraping ethics and best practices: robots.txt, rate limits, privacy and the law

Start with the alternatives

Before writing a scraper, check for:

1. An official API or data export. It is more stable, documented and clearly permitted.
2. A feed: RSS, Atom, JSON Feed, sitemaps.
3. An open dataset: Common Crawl, government open data portals, Wikidata.
4. Asking. Many site owners will provide data if you explain the purpose.

The questions that decide whether it is acceptable

• Is the content public? Content behind a login, a paywall or an access control is off limits for scraping. Bypassing such controls can be unauthorized access under computer misuse laws, not just a terms violation.
• What do the terms of service say? Terms may forbid automated access. Violating them can have contractual consequences and gets you blocked.
• Is there personal data? Names, profiles, photos and contact details are protected by privacy laws such as the GDPR in many places, even if they are publicly visible. You need a lawful basis, a purpose, minimization and retention limits.
• Is it copyrighted? Facts are generally free; substantial copying of articles, images or databases may not be. Republishing is riskier than analysis.
• Will it harm the site? Load that degrades service for real users is never acceptable.

This is not legal advice; laws differ by country and change. When the answer matters, ask a lawyer.

robots.txt

The Robots Exclusion Protocol, standardized as RFC 9309, lets sites state which paths crawlers should avoid. It is not an access control, but honoring it is the baseline of good behavior. Fetch /robots.txt, match your user agent group, respect Disallow, and treat Crawl-delay (non-standard) as a hint to slow down.

A polite fetcher

import time, urllib.robotparser, httpx
from urllib.parse import urlsplit

UA = "ExampleResearchBot/1.0 (+https://example.com/bot; bot@example.com)"
robots = {}

def allowed(url):
    root = "{0.scheme}://{0.netloc}".format(urlsplit(url))
    if root not in robots:
        rp = urllib.robotparser.RobotFileParser(root + "/robots.txt"); rp.read(); robots[root] = rp
    return robots[root].can_fetch(UA, url)

client = httpx.Client(headers={"User-Agent": UA}, timeout=20, follow_redirects=True)
last = {}

def fetch(url, min_interval=5.0):
    if not allowed(url):
        return None
    host = urlsplit(url).netloc
    wait = min_interval - (time.monotonic() - last.get(host, 0))
    if wait > 0: time.sleep(wait)
    r = client.get(url)
    last[host] = time.monotonic()
    if r.status_code in (429, 503):
        time.sleep(int(r.headers.get("Retry-After", 60)))
    return r

Principles in that code: an honest user agent with contact information, robots.txt checks, one request at a time per host with a delay, and backing off when the server signals overload.

More good habits

• Cache responses and use conditional requests (If-None-Match, If-Modified-Since).
• Crawl off-peak and stop on repeated errors.
• Collect only the fields you need, and delete raw pages when you have extracted them.
• Do not rotate IPs or user agents to evade blocks. If a site blocks you, that is the answer.
• Never solve or bypass CAPTCHAs or bot challenges with automation services.
• Use a headless browser only when the content truly requires JavaScript; it costs the site more than a plain request.

AI crawlers and training data

If you collect data to train or ground AI models, be transparent: publish your crawler's user agent and purpose, honor robots.txt rules aimed at AI crawlers, and respect opt-out signals. Site owners increasingly block unknown crawlers; a documented, well-behaved one is much less likely to be blocked.

When you are the site owner

Bots will visit your own site too. Publish a clear robots.txt, offer an API or feed for legitimate uses, rate limit by IP and token, and use bot management for abusive traffic rather than blocking every automated client.