Practical Data Scraping

• Translate a data question into fields, source URLs, and acceptance criteria
• Read HTTP methods, response codes, headers, and the browser Network panel
• Review source permissions, robots.txt, terms, and data minimisation
• Choose a permitted source and define request and collection limits

Lab 01: a source assessment and a field-level collection specification.

Checkpoint: Explain when an API, a static page, or a browser is the right starting point.

• Set up a virtual environment, dependencies, and a small project structure
• Use Requests sessions, parameters, explicit timeouts, and status checks
• Read text and JSON responses and normalise relative URLs
• Keep configuration and secrets out of source code

Lab 02: a command-line collector that saves a response and handles a failed request.

Checkpoint: Demonstrate what happens on a timeout, a missing page, and an unexpected response type.

• Inspect the DOM and write focused CSS selectors with Beautiful Soup
• Handle absent elements, nested content, whitespace, and encodings
• Parse dates, currencies, and numeric fields without losing source values
• Define required fields and separate valid records from rejected ones

Lab 03: a product extractor with saved HTML fixtures and field validation.

Project in this module: Build a product data collector.

Checkpoint: Repair the parser when a field is missing or the page contains a second layout.

• Follow page-number, next-link, and cursor pagination
• Inspect documented or authorised JSON endpoints
• Set stopping conditions, deduplicate URLs, and track visited pages
• Checkpoint progress and resume an interrupted collection

Lab 04: a paginated collector that can resume without duplicating records.

Project in this module: Build a local business review dataset.

Project in this module: Build a job listings collector.

Project in this module: Build a clean business directory dataset.

Checkpoint: Show that the last page is collected and a rerun does not silently double the dataset.

• Use Playwright locators and condition-based waits
• Handle filters, load-more buttons, and lazy-rendered content
• Reuse browser contexts for authorised sessions
• Capture traces and screenshots to diagnose a failed browser step

Lab 05: a browser collector for a permitted dynamic listing with a failure trace.

Checkpoint: Replace a fixed sleep with a condition and explain why it is more dependable.

• Separate retryable failures from permanent failures
• Use bounded retries, backoff, and Retry-After responses
• Limit concurrency and request rate to respect the source
• Persist progress, reuse connections, and stop safely on access restrictions

Lab 06: a controlled collector tested against simulated rate limits and transient failures.

Checkpoint: Prove that retries eventually stop and a denied source is not retried indefinitely.

• Normalise and deduplicate with Pandas while preserving provenance
• Design stable record IDs and SQL tables
• Use idempotent inserts or upserts and track collected-at timestamps
• Export CSV and JSON with explicit schemas and quality summaries

Lab 07: a reproducible cleaning and storage pipeline with a data-quality report.

Project in this module: Build a retail catalogue pipeline.

Checkpoint: Run the same input twice and show stable IDs, row counts, and update behaviour.

• Write parser tests using local fixtures and mocked network responses
• Cover changed selectors, empty pages, malformed records, and partial runs
• Log collection counts, rejection counts, and request failures
• Detect schema drift and separate a genuine empty result from an extraction failure

Lab 08: a regression suite and a run summary that makes collection failures visible.

Project in this module: Build a travel review quality report.

Checkpoint: Break a selector deliberately and show that the tests or quality checks catch it.

• Select a permitted product catalogue and define a bounded scope
• Combine collection, parsing, validation, and storage
• Record product prices with timestamps and source URLs
• Compare snapshots and produce a meaningful change report

Capstone draft: a working pipeline, sample dataset, and price-change report.

Project in this module: Build a price-monitoring pipeline.

Checkpoint: Trace one reported price change back to its source record and collection run.

• Run the capstone on a schedule with explicit execution limits
• Handle interrupted runs, partial outputs, and safe restarts
• Write setup instructions, source assumptions, and maintenance notes
• Revisit earlier failure cases and check the capstone against the completion rubric

Final capstone: a documented, scheduled pipeline with a recovery demonstration.

Project in this module: Build a price-monitoring pipeline.

Checkpoint: Have another developer follow the README and reproduce a successful local run.