Data collection and web scraping

In short

Data collection is the step that turns a real-world question into rows a program can read; web scraping is the automated extraction of data from web pages when no file or API is offered. Every AI/DS pipeline starts here - a model trained on badly collected data cannot be rescued by better algorithms. University syllabi examine collection methods, sampling, HTTP, HTML parsing and the ethics/legal frame; placement tests ask for requests + BeautifulSoup code, API pagination logic and rate-limit handling. Indian employers who hire for this include Flipkart, Zomato, Swiggy, Paytm, PhonePe, Jio, Dream11, Zepto, Tracxn, Mu Sigma and almost every pricing-intelligence or lending-analytics firm; it is also the core of government open-data work (data.gov.in).

Explained

Sources and classification of data

• Primary data - collected first-hand for your question: surveys, forms, sensors, IoT devices, experiments, clickstream logs. Costly, slow, but exactly fitted.
• Secondary data - already collected by someone else: NSSO/NSO and MoSPI releases, Census, RBI DBIE, data.gov.in, Kaggle, World Bank, company filings on BSE/NSE. Cheap and fast; you inherit its definitions and errors.
• By structure: structured (RDBMS tables, CSV), semi-structured (JSON, XML, HTML, logs), unstructured (text, images, audio, PDFs).
• By frequency: batch (nightly dumps) vs streaming (Kafka topics, webhooks, ticker feeds).
• Acquisition ladder - always try in this order: (1) bulk file/dataset download, (2) official API, (3) partner/licensed feed, (4) scraping, (5) manual entry. Scraping is a fallback, not a first choice, because it is brittle and legally sensitive.
• Sampling designs examined: simple random, systematic (every k-th), stratified (proportional within strata), cluster, multistage (used by NSO), and non-probability ones - convenience, quota, snowball.
• Bias types: selection/coverage bias, survivorship bias, non-response bias, measurement bias, and *temporal* bias (prices scraped only at 11 a.m. on weekdays).

HTTP, the protocol everything rides on

• A scrape is just an HTTP request-response. Methods: GET (fetch, idempotent, parameters in URL), POST (submit, body carries data), PUT/PATCH/DELETE for APIs.
• URL anatomy: scheme://host:port/path?query=value&page=2#fragment. The fragment is never sent to the server - a clue that the content is rendered by JavaScript.
• Status codes: 200 OK; 301/302 redirect; 304 not modified; 400 bad request; 401 unauthenticated; 403 forbidden (usually bot protection, *not* a dead page); 404 not found; 429 Too Many Requests (back off); 500/502/503 server-side - retry later.
• Request headers that matter: User-Agent (identify your bot honestly), Accept, Accept-Language, Referer, Authorization: Bearer <token>, Cookie.
• Response headers: Content-Type (text/html vs application/json), Set-Cookie, Retry-After, ETag/Last-Modified (use for conditional, polite re-fetching), Link (pagination).
• Sessions: cookies maintain login state; requests.Session() reuses the TCP connection and cookie jar, which is both faster and necessary behind a login.
• Robots and politeness: read https://site/robots.txt *before* fetching; honour Disallow, Crawl-delay and the sitemap it advertises. Add your own delay, cap concurrency per host, and set a timeout on every request.

Parsing HTML: the DOM, selectors and libraries

• HTML is a tree (DOM) of elements, attributes and text. Extraction = navigate the tree and pull text or attribute values.
• CSS selectors: div.price, #main, ul > li, a[href^="/job/"], td:nth-of-type(2). Compact, the common choice.
• XPath: //table[@id="t1"]//tr[position()>1]/td[2]/text(), //a[contains(@href,"page=")]. More powerful - supports axes (parent::, following-sibling::) and text predicates, which CSS cannot do.
• Library map:
  • requests - fetch (HTTP client, no parsing).
  • BeautifulSoup (bs4) - forgiving parser API: find, find_all, select, get_text(); use the lxml parser for speed, html.parser for zero dependencies.
  • lxml - fast C-based parser, full XPath.
  • pandas.read_html - pulls every <table> on a page into a list of DataFrames; one line for tabular pages.
  • Scrapy - full asynchronous framework: spiders, items, pipelines, middlewares, autothrottle, built-in retry and robots.txt obedience. Right choice beyond a few thousand pages.
  • Selenium / Playwright - drive a real browser; needed only when content is JavaScript-rendered or requires interaction.
  • httpx - async HTTP with HTTP/2 for concurrent fetching.
• Static vs dynamic pages: if "View Page Source" contains the data, requests suffices. If the data appears only in the rendered DOM, either (a) find the underlying JSON/XHR endpoint in the browser Network tab - fastest and cleanest - or (b) automate a browser (10-50× slower, heavy on RAM).

APIs - the preferred route

• REST: resource URLs + HTTP verbs, JSON responses. GraphQL: one endpoint, you specify the fields you want. SOAP/XML: older enterprise and some government services.
• Authentication: API key in header or query, OAuth 2.0 bearer tokens, HMAC-signed requests (common in Indian payment and brokerage APIs - Razorpay, Kite Connect).
• Pagination patterns: offset/limit (?offset=100&limit=50), page number (?page=3), cursor/token (next_cursor - safest, no duplicates when rows are inserted mid-crawl), and Link: rel="next" headers.
• Rate limits: quoted as requests per second/minute/day. Handle 429 with exponential backoff plus jitter: wait 1, 2, 4, 8 s (± random) and respect Retry-After when present.
• Parsing JSON: r.json() → dict/list; flatten nested structures with pandas.json_normalize(data, record_path=..., meta=[...]).
• Idempotency and resumability: store a checkpoint (last page/cursor) so an interrupted job restarts without re-fetching; write raw responses to disk before parsing so a parser bug does not force a re-crawl.

Building a reliable scraper - the pipeline

1. Scope: define the exact fields, the universe (how many pages/records *in total*) and the refresh frequency.
2. Reconnaissance: robots.txt, sitemap.xml, terms of use, URL patterns, hidden JSON endpoints.
3. Fetch layer: session, honest User-Agent, timeout, retries with backoff, per-host delay, proxy pool only if contractually allowed.
4. Store raw: save HTML/JSON with the URL and a UTC timestamp (raw → parsed → clean is the standard three-zone layout, i.e. bronze/silver/gold).
5. Parse: selectors into a flat record; coerce types; keep a source_url and fetched_at on every row - this is your provenance.
6. Validate: required fields non-null, numeric ranges, row-count within expected bounds, duplicate check on a natural key. Fail loudly.
7. Deduplicate: exact key match, then normalised match (lowercase, strip punctuation), then fuzzy (token-set ratio) for names.
8. Incremental loads: use ETag/Last-Modified, "new since last id", or hash the content to detect change.
9. Monitor: alert on a drop in rows scraped, a surge in 403/429, or a selector returning None - the usual symptom of a site redesign.
10. Document: a data dictionary per field with its source, unit and collection date.
11. Anti-bot measures you will meet: CAPTCHAs, Cloudflare/Akamai JS challenges, TLS fingerprinting, honeypot links, dynamic class names (class="css-1x9y3z" - never key on these; anchor on stable text, id, or data-* attributes).
12. File formats for storage: CSV (universal, untyped), Parquet (columnar, compressed, typed - the default for large scrapes), JSONL (one record per line, append-friendly), SQLite/Postgres for querying.

Legal, ethical and Indian regulatory frame

• Digital Personal Data Protection Act, 2023 (DPDP Act) - India's data-protection law. It governs personal data processed digitally; core duties are a lawful basis (consent or a listed legitimate use), notice, purpose limitation, data minimisation, accuracy, security safeguards, breach notification to the Data Protection Board, and erasure when the purpose ends. The Act is enacted; its rules and the Board's operation have been phasing in with transition timelines, so check the current status of specific obligations before relying on a date. The Act carves out personal data that the data principal has made *publicly available* - but that carve-out is narrow and does not cover re-identification or scraping behind a login.
• Information Technology Act, 2000 - s.43 (unauthorised access / damage to a computer resource) and s.66 can be invoked where access is unauthorised; the SPDI Rules, 2011 governed sensitive personal data and are being superseded by the DPDP regime.
• Copyright Act, 1957 - the *content* of pages (text, photographs, databases by way of originality in selection/arrangement) can be protected; facts and raw numbers are not. Indian law has no broad US-style "fair use"; s.52 lists specific fair-dealing exceptions, and a general text-and-data-mining exception is still under policy discussion.
• Contract: a site's Terms of Use can prohibit automated access; breaching them is a contractual matter and has grounded trespass/breach claims.
• Sector rules: SEBI and RBI restrict redistribution of market and credit data; TRAI/DoT rules and the telecom framework govern subscriber data; CERT-In directions require logging and prompt incident reporting by service providers.
• Practical ethics checklist: scrape only public pages; never bypass authentication, paywalls or CAPTCHAs; take no personal data you do not need; aggregate and anonymise; never resell someone's content as your own; identify your bot and publish a contact; respect robots.txt and takedown requests; prefer a licensed feed when one exists.

Key points

• Primary vs secondary data - collected by you for your purpose vs reused from an existing source with its definitions baked in.
• Acquisition ladder - bulk file, then API, then licensed feed, then scraping, then manual; never start at scraping.
• robots.txt - a site's crawl policy; read it first and honour Disallow and Crawl-delay.
• 403 vs 404 - 403 is bot protection or permission denial (page may exist); 404 means no such resource.
• 429 Too Many Requests - you are too fast; back off exponentially and obey Retry-After.
• requests fetches, BeautifulSoup/lxml parse, Scrapy orchestrates at scale, Selenium/Playwright render JavaScript.
• Static vs dynamic - if the data is absent from page source it is JS-rendered; hunt the XHR/JSON endpoint before reaching for a browser.
• CSS selector vs XPath - CSS is terser; XPath adds axes and text-content predicates.
• Cursor pagination beats offset pagination because inserts mid-crawl cause no duplicates or skips.
• Exponential backoff with jitter - 1, 2, 4, 8 s ± random, to avoid synchronised retry storms.
• Provenance columns - keep source_url and fetched_at (UTC) on every scraped row; without them nothing is auditable.
• Store raw first - raw → parsed → clean, so a parser bug never forces a re-crawl.
• Stable anchors - key on id, data-* or visible text, never on generated class names.
• Parquet for large scrapes: columnar, compressed, typed; CSV loses types; JSONL is append-friendly.
• DPDP Act, 2023 governs digital personal data in India - lawful basis, notice, purpose limitation, minimisation, breach reporting to the Data Protection Board.
• Publicly available ≠ free to use - copyright, terms of use and sectoral rules (SEBI, RBI, TRAI) still apply.
• State the denominator - "scraped 12 cities" is meaningless without "of 451"; coverage must always be reported as a fraction.
• Sampling bias - selection, survivorship, non-response and temporal bias all enter at collection and cannot be fixed downstream.

Formulas and examples

Example 1 - crawl budget and time. A price monitor must cover 2,40,000 product pages. Average server response 0.8 s, politeness delay 0.5 s per worker, 8 concurrent workers allowed.

• Per-worker cycle = 0.8 + 0.5 = 1.3 s → 1/1.3 = 0.769 pages/s.
• R = 8 × 0.769 = 6.15 pages/s.
• T = 2,40,000 / 6.15 = 39,024 s ≈ 10.8 hours.
• At 180 KB per page, V = 2,40,000 × 180 KB = 43,200,000 KB ≈ 40.2 GiB of raw HTML (43.2 GB decimal) - so store compressed, or parse and keep Parquet.

Example 2 - pagination and request count. A job board shows 7,480 listings, 25 per page, plus one detail page per listing.

• List pages P = ceil(7480/25) = 300.
• Total requests = 300 + 7,480 = 7,780.
• At a rate limit of 60 requests/minute (1 per second, single worker): 7,780 s = 2 h 9.7 min.
• If 4% of detail requests fail and are retried once: extra = 0.04 × 7,480 ≈ 299 requests → 8,079 total ≈ 2 h 14.7 min.

Example 3 - exponential backoff schedule. Base b = 1 s, factor 2, jitter uniform 0-0.5 s, 5 attempts, cap 30 s.

• Total deterministic wait = 1+2+4+8+16 = 31 s for 5 attempts; expected jitter adds 5 × 0.25 = 1.25 s → ≈ 32.25 s.

Example 4 - coverage, success rate and sample size. A crawl of 451 Indian cities issues 12,300 requests: 11,205 return 200, 640 return 403, 455 return 429.

• Success rate = 11,205/12,300 = 91.1%.
• If those 11,205 pages yielded data for 418 cities, coverage = 418/451 = 92.7% - the remaining 33 cities must be reported, not hidden.
• Sample survey instead: for p = 0.5, e = 0.03, Z = 1.96 → n₀ = 1.96² × 0.25/0.03² = 3.8416 × 0.25/0.0009 = 1,067. With a finite population N = 20,000: n = 1,067/(1 + 1,066/20,000) = 1,067/1.0533 = 1,013.

Example 5 - working code, static page.

import time, requests, pandas as pd
from bs4 import BeautifulSoup
from urllib.robotparser import RobotFileParser

BASE = "https://example.com/jobs"
UA = "campus-research-bot/1.0 (+contact: team@college.ac.in)"

rp = RobotFileParser(); rp.set_url("https://example.com/robots.txt"); rp.read()
assert rp.can_fetch(UA, BASE), "disallowed by robots.txt"

s = requests.Session(); s.headers["User-Agent"] = UA
rows, page = [], 1
while True:
    r = s.get(BASE, params={"page": page}, timeout=10)
    if r.status_code == 429:                       # rate limited
        time.sleep(int(r.headers.get("Retry-After", 2 ** page)))
        continue
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "lxml")
    cards = soup.select("div.job-card")
    if not cards:
        break                                      # no more pages
    for c in cards:
        rows.append({
            "title":    c.select_one("h2.title").get_text(strip=True),
            "employer": c.select_one("span.employer").get_text(strip=True),
            "city":     (c.select_one("span.city") or {}).get("title"),
            "url":      requests.compat.urljoin(BASE, c.select_one("a")["href"]),
            "fetched_at": pd.Timestamp.utcnow(),
        })
    page += 1
    time.sleep(1.0)                                 # politeness delay

df = pd.DataFrame(rows).drop_duplicates(subset="url")
df.to_parquet("jobs_raw.parquet", index=False)
print(len(df), "rows from", page - 1, "pages")

Comparison table - choosing a tool.

Where it is used

• E-commerce and quick commerce - price and assortment intelligence at Flipkart, Amazon India, Zepto, Blinkit; dynamic repricing and stock-out detection.
• Fintech and lending - alternative data for credit models, GSTN/MCA/EPFO public filings, Account Aggregator feeds (consent-based, the compliant alternative to scraping bank data).
• Travel, food and real estate aggregators - MakeMyTrip, Zomato, Swiggy, 99acres, NoBroker build inventory and review datasets; fare and menu monitoring.
• Market and research firms - Tracxn, Mu Sigma, Nielsen, CMIE; media monitoring and ESG/news scraping for brokerages.
• Government and policy analytics - data.gov.in, MoSPI/NSO, RBI DBIE, eNAM mandi prices, IMD weather feeds for agri-analytics and smart-city dashboards.
• Job roles - Data Engineer, Data Collection/Crawl Engineer, Data Analyst, ML Engineer (training-corpus building), QA automation; LLM pretraining-corpus teams rely on Common Crawl-style pipelines.

Interview questions

• Q: When should you scrape instead of using an API? A: Only when no bulk download, official API or licensed feed exposes the data, and the site's robots.txt and terms permit automated access - scraping is the last option because it is brittle and legally sensitive.
• Q: Difference between find_all and select in BeautifulSoup? A: find_all matches by tag name and attributes in Python arguments; select takes a CSS selector string and returns matching elements - select_one returns the first.
• Q: A page shows data in the browser but requests returns none of it. Why, and what do you do? A: The content is rendered by JavaScript after load; open the browser's Network tab, find the XHR/JSON endpoint and call that directly, and only fall back to Playwright/Selenium if there is no such endpoint.
• Q: What does a 403 mean for a crawler? A: Access was refused - usually bot detection, a missing header, or a geo/login restriction. The page may well exist, so a 403 is never evidence that the resource is dead.
• Q: How do you handle rate limiting properly? A: Respect the documented limit, keep a per-host delay and concurrency cap, and on 429 obey Retry-After or back off exponentially with jitter rather than retrying immediately.
• Q: Why is cursor-based pagination safer than offset-based? A: A cursor points at a position in the result set, so rows inserted or deleted during the crawl cannot cause the duplicates or skipped records that shifting offsets produce.
• Q: What is CSS-selector brittleness and how do you reduce it? A: Build-generated class names like css-1x9y3z change on every deploy; anchor instead on id, data-* attributes, stable text labels or structural relationships, and alert when a selector starts returning nothing.
• Q: Which columns must every scraped row carry? A: source_url and fetched_at in UTC at minimum, plus a stable natural key - without provenance the dataset cannot be audited, refreshed or deduplicated.
• Q: Does the DPDP Act, 2023 stop you from scraping? A: It does not ban scraping, but any digital personal data you collect needs a lawful basis plus notice, purpose limitation, minimisation and security (Data Fiduciary duties from 13 May 2027; until then IT Act s.43A and the SPDI Rules 2011 govern); the Act excludes personal data the individual has voluntarily made public, and that exclusion is narrow.
• Q: Is public web data free to use commercially? A: No - copyright in the content, the site's terms of use, and sectoral rules (SEBI/RBI for market and credit data, for instance) can each restrict copying or redistribution even when the page needs no login.
• Q: How do you make a long crawl resumable? A: Checkpoint the last completed page or cursor, write raw responses to disk keyed by URL, make writes idempotent on a natural key, and skip anything already stored when the job restarts.
• Q: How do you detect that a scraper has silently broken? A: Monitor row counts against expectation, null rates per field, the share of non-2xx responses, and selector hit rates - a healthy HTTP 200 with zero extracted rows is the classic silent failure.

Common mistakes

• Treating a 403 or an empty result as proof that the page or company does not exist, instead of bot protection or a wrong URL.
• Reporting a sample as the whole job - "crawled 12 cities" when the universe is 451 - and omitting the denominator.
• Hammering a site with no delay, no timeout and no retry policy, then being IP-blocked mid-crawl.
• Parsing HTML with regular expressions instead of a DOM parser, which breaks on nested tags, attributes and whitespace.
• Keying on auto-generated class names or on absolute positions (div > div > div:nth-child(3)), so the scraper dies at the site's next deploy.
• Parsing immediately and discarding the raw response, so every parser bug forces a full re-crawl.
• Forgetting provenance and timezone - storing local time or no timestamp at all, which makes change detection and auditing impossible.
• Assuming "publicly visible" means "legally reusable", and collecting names, phone numbers or emails that the use case never needed.

Related topics

Data cleaning and feature engineering · Python essentials · Python for data analysis: NumPy and pandas · SQL queries and joins · Big data and distributed storage · Data ethics and privacy law