Skip to content
National AI Course

AI Advanced

Design, build, evaluate and operate AI systems you'd put in production.

Lessons
7 lessons
Length
About 1.5 hours
Access
Yours permanently

For people who write code. Everything between a working prototype and a system you would be willing to put in front of users: evaluation, retrieval, tools, agents, security and the production engineering that keeps it honest.

Who it’s for

Developers, analysts and technical product people who have used these APIs and want to ship something they can defend. You should be comfortable reading code and running things locally.

What you need first

AI at Work, or equivalent judgement about where these systems fail, plus some coding or scripting ability.

What you’ll be able to do

  • Build an evaluation set that tells you whether a change made things better
  • Design retrieval that grounds answers and never returns a document the user cannot open
  • Give a model tools, and know which actions must sit behind an approval gate
  • Recognise prompt injection and design so private data, untrusted content and external communication never coexist
  • Run a system in production: monitoring, cost control, versioning and rollback

What’s inside

Module 1 · How models actually work

Tokens, attention, context windows and sampling — enough to predict behaviour.

  • Context windows and sampling controls

    11 min

    Tokens, position effects, why long context degrades, and what temperature actually does — enough to predict behaviour rather than guess at it.

    • Explain what a context window holds and why position matters
    • Predict how a sampling parameter change affects output
    • Design around long-context degradation rather than hoping

Module 2 · Choosing and evaluating models

Why leaderboards do not predict your use case, and what to build instead.

  • Building an eval set that tells you something

    11 min

    Why public benchmarks do not predict your use case, and how to build the 20–100 examples that do.

    • Explain why leaderboard performance does not transfer to your task
    • Build a graded eval set from real examples
    • Use LLM-as-judge without inheriting its failure modes

Module 3 · Advanced prompting and context engineering

System prompts, structured output, decomposition, and treating prompts as code.

  • Structured output and decomposition

    10 min

    Getting parseable output reliably, and knowing when a chain of small calls beats one large prompt.

    • Get reliably parseable output using structural rather than prompt defences
    • Decide between one prompt and a chain
    • Apply map-reduce over documents that exceed the context window

Module 4 · Retrieval and knowledge systems

Grounding answers in your own documents, and evaluating whether it worked.

  • Retrieval that actually works

    12 min

    Why pure vector search underperforms, how to tell a retrieval failure from a generation failure, and the permission rule that is not optional.

    • Design ingestion and chunking that preserves meaning
    • Combine keyword and semantic search, and rerank
    • Separate retrieval failures from generation failures when evaluating

Module 5 · Tool use and agents

Function calling, agent architectures, failure modes and human-in-the-loop design.

  • Tool design and agent loops

    12 min

    Tool descriptions are prompts, agent failure modes are predictable, and observability is what makes a non-deterministic system debuggable.

    • Design tools a model can use correctly without guessing
    • Recognise and design against the standard agent failure modes
    • Instrument an agent so failures can be diagnosed after the fact

Module 6 · Security

Prompt injection, the lethal trifecta, exfiltration and red-teaming your own system.

  • Prompt injection and the lethal trifecta

    12 min

    The unsolved vulnerability at the centre of LLM applications, why prompt-based defences do not work, and the architectural rule that does.

    • Distinguish direct from indirect prompt injection
    • Explain why instruction and data cannot be separated
    • Apply the lethal trifecta rule when designing a system

Module 7 · Production engineering

Evals in CI, monitoring, cost, versioning, rollback and graceful degradation.

  • Running it in production

    11 min

    Evals in CI, monitoring what actually drifts, versioning everything, and degrading gracefully when the model is down.

    • Gate deployments on eval scores rather than hoping
    • Monitor the things that actually drift in an LLM system
    • Design fallbacks for a dependency that fails differently from a database

Questions about this course

How much code is there?
Enough that you should be comfortable reading it. The activities build real things — an eval harness, a retrieval pipeline, a tool server, an agent with tracing — and the code is provided to read and adapt rather than typed from scratch.
Which language and which provider?
Examples are in TypeScript and Python, and the concepts are provider-neutral. One module deals specifically with designing against lock-in, because the decisions that trap you are architectural rather than a matter of which API you called first.
Will the specifics date quickly?
Some will, and the course is structured so that they are contained. Model names, prices and interfaces live in one clearly-marked lesson per module rather than being sprinkled through; the architectural material — evaluation, retrieval, tool design, security — has been stable and is where most of the course sits.
Do I need AI at Work first?
Recommended rather than enforced. Advanced assumes you already know why these systems fail and who is accountable when they do; it teaches you to build around that rather than re-establishing it.

Start AI Advanced today.

One payment, this course, permanently. No subscription to cancel.

Compare all courses