Data scraping and AI app development courses

• Translate a data question into fields, source URLs, and acceptance criteria
• Read HTTP methods, response codes, headers, and the browser Network panel
• Review source permissions, robots.txt, terms, and data minimisation
• Choose a permitted source and define request and collection limits

Lab 01: a source assessment and a field-level collection specification.

Checkpoint: Explain when an API, a static page, or a browser is the right starting point.

• Set up a virtual environment, dependencies, and a small project structure
• Use Requests sessions, parameters, explicit timeouts, and status checks
• Read text and JSON responses and normalise relative URLs
• Keep configuration and secrets out of source code

Lab 02: a command-line collector that saves a response and handles a failed request.

Checkpoint: Demonstrate what happens on a timeout, a missing page, and an unexpected response type.

• Inspect the DOM and write focused CSS selectors with Beautiful Soup
• Handle absent elements, nested content, whitespace, and encodings
• Parse dates, currencies, and numeric fields without losing source values
• Define required fields and separate valid records from rejected ones

Lab 03: a product extractor with saved HTML fixtures and field validation.

Project in this module: Build a product data collector.

Checkpoint: Repair the parser when a field is missing or the page contains a second layout.

• Follow page-number, next-link, and cursor pagination
• Inspect documented or authorised JSON endpoints
• Set stopping conditions, deduplicate URLs, and track visited pages
• Checkpoint progress and resume an interrupted collection

Lab 04: a paginated collector that can resume without duplicating records.

Project in this module: Build a local business review dataset.

Project in this module: Build a job listings collector.

Project in this module: Build a clean business directory dataset.

Checkpoint: Show that the last page is collected and a rerun does not silently double the dataset.

• Use Playwright locators and condition-based waits
• Handle filters, load-more buttons, and lazy-rendered content
• Reuse browser contexts for authorised sessions
• Capture traces and screenshots to diagnose a failed browser step

Lab 05: a browser collector for a permitted dynamic listing with a failure trace.

Checkpoint: Replace a fixed sleep with a condition and explain why it is more dependable.

• Separate retryable failures from permanent failures
• Use bounded retries, backoff, and Retry-After responses
• Limit concurrency and request rate to respect the source
• Persist progress, reuse connections, and stop safely on access restrictions

Lab 06: a controlled collector tested against simulated rate limits and transient failures.

Checkpoint: Prove that retries eventually stop and a denied source is not retried indefinitely.

• Normalise and deduplicate with Pandas while preserving provenance
• Design stable record IDs and SQL tables
• Use idempotent inserts or upserts and track collected-at timestamps
• Export CSV and JSON with explicit schemas and quality summaries

Lab 07: a reproducible cleaning and storage pipeline with a data-quality report.

Project in this module: Build a retail catalogue pipeline.

Checkpoint: Run the same input twice and show stable IDs, row counts, and update behaviour.

• Write parser tests using local fixtures and mocked network responses
• Cover changed selectors, empty pages, malformed records, and partial runs
• Log collection counts, rejection counts, and request failures
• Detect schema drift and separate a genuine empty result from an extraction failure

Lab 08: a regression suite and a run summary that makes collection failures visible.

Project in this module: Build a travel review quality report.

Checkpoint: Break a selector deliberately and show that the tests or quality checks catch it.

• Select a permitted product catalogue and define a bounded scope
• Combine collection, parsing, validation, and storage
• Record product prices with timestamps and source URLs
• Compare snapshots and produce a meaningful change report

Capstone draft: a working pipeline, sample dataset, and price-change report.

Project in this module: Build a price-monitoring pipeline.

Checkpoint: Trace one reported price change back to its source record and collection run.

• Run the capstone on a schedule with explicit execution limits
• Handle interrupted runs, partial outputs, and safe restarts
• Write setup instructions, source assumptions, and maintenance notes
• Revisit earlier failure cases and check the capstone against the completion rubric

Final capstone: a documented, scheduled pipeline with a recovery demonstration.

Project in this module: Build a price-monitoring pipeline.

Checkpoint: Have another developer follow the README and reproduce a successful local run.

• Identify a narrow user task and compare an AI approach with a simpler baseline
• Understand tokens, context windows, probabilistic outputs, and model limits
• Define acceptable answers, unsupported questions, and failure behaviour
• Create an initial set of representative and difficult evaluation cases

Lab 01: a product brief, baseline, and a 20-case evaluation set.

Checkpoint: Explain what would count as an improvement and what evidence would disprove it.

• Call a model from Python with server-side credentials
• Work with messages, streaming, timeouts, and provider errors
• Validate structured outputs with JSON schemas and Pydantic
• Record tokens, latency, failures, and estimated request cost

Lab 02: a typed AI endpoint with validation, usage logging, and error states.

Checkpoint: Handle malformed output and a provider timeout without exposing secrets or returning a false success.

• Write explicit instructions, context boundaries, and worked examples
• Version prompts and compare them against a baseline
• Use structured extraction and classification for language understanding
• Separate instructions from untrusted user or document content

Lab 03: a versioned extraction workflow with an evaluation comparison.

Checkpoint: Improve a failing case without degrading the held-out cases or adding phrase-specific rules.

• Load permitted documents and preserve source metadata
• Clean text, split it into chunks, and create embeddings
• Compare lexical and semantic retrieval and inspect retrieved results
• Version documents, prevent duplicate ingestion, and update changed content

Lab 04: a searchable document index with traceable sources and repeatable ingestion.

Checkpoint: Find a retrieval failure and explain whether the problem is ingestion, chunking, or ranking.

• Assemble retrieved context within a token budget
• Generate answers with source references and verifiable quotations
• Handle missing, conflicting, or stale evidence
• Measure retrieval coverage separately from answer groundedness

Lab 05: a document assistant that cites evidence and abstains when it cannot support an answer.

Project in this module: Build an evidence-based review assistant.

Checkpoint: Ask an unanswerable question and demonstrate a useful, honest fallback.

• Define tools with narrow input schemas and validated results
• Choose a deterministic workflow before adding autonomous loops
• Set tool-call limits, time budgets, and explicit stopping conditions
• Require confirmation before actions that change external state

Lab 06: a read-only tool workflow with bounded execution and a visible activity trace.

Checkpoint: Demonstrate that invalid arguments, repeated tool calls, and a tool failure end safely.

• Serve the workflow through FastAPI and a simple user interface
• Show loading, streaming, citations, empty results, and retry states
• Store conversation state with clear session boundaries
• Add authentication, access-scoped retrieval, and usage limits

Lab 07: a usable application with source links, isolated sessions, and clear errors.

Checkpoint: Show that one user's session cannot retrieve another user's private documents.

• Extend the evaluation set with unsupported, adversarial, and regression cases
• Use deterministic checks and a clear rubric alongside model-assisted judging
• Test prompt injection, data leakage, tool misuse, and ungrounded answers
• Compare quality, latency, and cost before changing a model or prompt

Lab 08: an evaluation report with failure analysis and a measured improvement.

Project in this module: Build an evidence-based review assistant.

Checkpoint: Explain why an average score can hide a serious failure and which cases block release.

• Choose a bounded document set and a concrete target user
• Combine ingestion, retrieval, cited answers, and a read-only tool
• Build an end-to-end application with evaluation and usage tracking
• Document supported tasks, out-of-scope questions, and known limitations

Capstone draft: a working assistant, evaluation dataset, and a quality/cost report.

Project in this module: Build a cited research assistant.

Checkpoint: Demonstrate a supported answer, an unsupported question, and an attempted instruction injection.

• Package and deploy the app with environment-based configuration
• Add health checks, structured logs, request tracing, and budget alerts
• Version the prompt, model configuration, and retrieval index
• Re-run evaluations, practise rollback, and revisit earlier failure cases

Final capstone: a deployed or locally reproducible release with a runbook and release checklist.

Project in this module: Build a cited research assistant.

Checkpoint: Reproduce the release, inspect a failed request, and roll back a configuration change.