AI App Development
Build a useful AI application around your own data. Learn model APIs, retrieval, tool use, evaluation, and the engineering needed beyond a demo.
Before you start
Comfortable Python, JSON, HTTP APIs, Git, and basic testing. You should be able to build or modify a small application. No machine-learning mathematics or model-training background required.
Suggested workload: 65–80 hours. Suggested pace: 10–12 weeks.
Suggested pace at around 6–7 hours a week. Estimates include examples, eight labs, evaluation work, revision, and the capstone; they are not recorded-video hours.
Learning outcomes
- Turn a narrow user problem into measurable AI behaviour and an evaluation set.
- Build validated model calls, retrieval with citations, and bounded tool workflows.
- Measure answer quality, failure cases, latency, and per-request cost.
- Deploy a documented application with access controls, logs, and a rollback plan.
Full curriculum
1. Define the product and the quality bar
- Identify a narrow user task and compare an AI approach with a simpler baseline
- Understand tokens, context windows, probabilistic outputs, and model limits
- Define acceptable answers, unsupported questions, and failure behaviour
- Create an initial set of representative and difficult evaluation cases
Lab 01: a product brief, baseline, and a 20-case evaluation set.
Checkpoint: Explain what would count as an improvement and what evidence would disprove it.
2. Integrate a model API properly
- Call a model from Python with server-side credentials
- Work with messages, streaming, timeouts, and provider errors
- Validate structured outputs with JSON schemas and Pydantic
- Record tokens, latency, failures, and estimated request cost
Lab 02: a typed AI endpoint with validation, usage logging, and error states.
Checkpoint: Handle malformed output and a provider timeout without exposing secrets or returning a false success.
3. Engineer and test prompts
- Write explicit instructions, context boundaries, and worked examples
- Version prompts and compare them against a baseline
- Use structured extraction and classification for language understanding
- Separate instructions from untrusted user or document content
Lab 03: a versioned extraction workflow with an evaluation comparison.
Checkpoint: Improve a failing case without degrading the held-out cases or adding phrase-specific rules.
4. Prepare data for retrieval
- Load permitted documents and preserve source metadata
- Clean text, split it into chunks, and create embeddings
- Compare lexical and semantic retrieval and inspect retrieved results
- Version documents, prevent duplicate ingestion, and update changed content
Lab 04: a searchable document index with traceable sources and repeatable ingestion.
Checkpoint: Find a retrieval failure and explain whether the problem is ingestion, chunking, or ranking.
5. Build grounded answers with RAG
- Assemble retrieved context within a token budget
- Generate answers with source references and verifiable quotations
- Handle missing, conflicting, or stale evidence
- Measure retrieval coverage separately from answer groundedness
Lab 05: a document assistant that cites evidence and abstains when it cannot support an answer.
Project in this module: Build an evidence-based review assistant.
Checkpoint: Ask an unanswerable question and demonstrate a useful, honest fallback.
6. Add tools and bounded workflows
- Define tools with narrow input schemas and validated results
- Choose a deterministic workflow before adding autonomous loops
- Set tool-call limits, time budgets, and explicit stopping conditions
- Require confirmation before actions that change external state
Lab 06: a read-only tool workflow with bounded execution and a visible activity trace.
Checkpoint: Demonstrate that invalid arguments, repeated tool calls, and a tool failure end safely.
7. Build the application experience
- Serve the workflow through FastAPI and a simple user interface
- Show loading, streaming, citations, empty results, and retry states
- Store conversation state with clear session boundaries
- Add authentication, access-scoped retrieval, and usage limits
Lab 07: a usable application with source links, isolated sessions, and clear errors.
Checkpoint: Show that one user's session cannot retrieve another user's private documents.
8. Evaluate quality, safety, and cost
- Extend the evaluation set with unsupported, adversarial, and regression cases
- Use deterministic checks and a clear rubric alongside model-assisted judging
- Test prompt injection, data leakage, tool misuse, and ungrounded answers
- Compare quality, latency, and cost before changing a model or prompt
Lab 08: an evaluation report with failure analysis and a measured improvement.
Project in this module: Build an evidence-based review assistant.
Checkpoint: Explain why an average score can hide a serious failure and which cases block release.
9. Capstone: build a knowledge assistant
- Choose a bounded document set and a concrete target user
- Combine ingestion, retrieval, cited answers, and a read-only tool
- Build an end-to-end application with evaluation and usage tracking
- Document supported tasks, out-of-scope questions, and known limitations
Capstone draft: a working assistant, evaluation dataset, and a quality/cost report.
Project in this module: Build a cited research assistant.
Checkpoint: Demonstrate a supported answer, an unsupported question, and an attempted instruction injection.
10. Deploy, observe, and iterate
- Package and deploy the app with environment-based configuration
- Add health checks, structured logs, request tracing, and budget alerts
- Version the prompt, model configuration, and retrieval index
- Re-run evaluations, practise rollback, and revisit earlier failure cases
Final capstone: a deployed or locally reproducible release with a runbook and release checklist.
Project in this module: Build a cited research assistant.
Checkpoint: Reproduce the release, inspect a failed request, and roll back a configuration change.
A source-grounded knowledge assistant
Build an assistant for a specific collection of documents, with cited answers, a read-only tool, explicit handling of unsupported questions, and an evaluation report.
- Runnable app, source repository, setup guide, and a short demo
- Repeatable document ingestion with provenance and access boundaries
- Evaluation dataset, failure analysis, and quality/latency/cost comparison
- Release configuration, monitoring notes, budget limits, and rollback runbook
Completion rubric
- Usefulness: the assistant completes the supported user task against an explicit baseline.
- Grounding: answers cite relevant evidence; unsupported questions are handled honestly.
- Control: private data stays scoped and tool execution has enforceable limits.
- Operability: evaluation, usage, error traces, and release steps are reproducible.
Model APIs and any optional hosting are billed by their providers. Use a small project budget, track usage, and confirm the expected tool costs before enrolling.