LLM bootcamp

Every later module assumes you can point at a piece of the stack and say what it does and what it costs you. This one draws that map. The canonical end-to-end architecture behind an agentic application: tokenization, embeddings and how models generate text; the loop that turns a model into an agent — plan, call a tool, observe, decide again; vector databases and similarity search as the memory that loop reads from; prompt engineering best practices; the three paths to a custom application; and where retrieval breaks down in production.

Nearly every constraint you hit later traces back to what happens inside the model: why context runs out, why one embedding beats another, what fine-tuning can and cannot move. You cannot reason about those from the outside. Where large language models are strong and where they are not, and how generative models differ from discriminative ones. Then transformers from the ground up: tokenization, embeddings and similarity, positional encoding, self-attention with keys, queries and values, and the softmax step that turns raw scores into probabilities — followed by how these models are trained and fine-tuned, and how the same embeddings power search and retrieval. Hands-on with Sentence Transformers, semantic search, and implementing attention scoring directly.

Retrieval sets the ceiling on everything above it. An agent that reads the wrong chunk gives the wrong answer, and no orchestration further up the stack recovers it — so this is the module that raises the ceiling. How embeddings turn meaning into geometry, and what a vector database does with them: semantic search, keyword search, and the hybrid search that beats either on its own; the index structures underneath — HNSW (hierarchical navigable small world), PQ (product quantization) and LSH (locality-sensitive hashing) — and what each one trades away in speed, memory or recall; then query rewriting, re-ranking, auto-cut, embedding compression, multi-tenancy, and the scaling problems that only surface at real query volume.

A retrieval demo works on the first try. The same pipeline in production meets private data, several languages and modalities, a latency budget, and someone asking for evidence that it works — which is where most teams stall. PII-safe ingestion and compliant retrieval; multimodal and multilingual pipelines; advanced architectures (DSPy, KG-RAG, self-improving retrieval); index optimization across FAISS, Qdrant, and Pinecone; evaluation with Precision@K, LLM-as-a-Judge, TruLens, and Phoenix; and controlling latency and cost at scale.

This is where you stop reasoning about the stack and start building on it. It also fixes the vocabulary — components, chains, tools, memory — that the next two modules assume you already have. Core components, output parsers, retrieval and vector stores, chains, tool use and memory, callbacks and observability, LCEL composition, and LangGraph for orchestrating multi-agent workflows with dynamic control flow.

The chains you just built stop holding the moment an agent runs many steps, and prompt engineering stops scaling with them. This module makes state the thing you design rather than a side effect of the prompt. Graphs instead of chains, typed state with reducers, checkpointers and thread scoping for short-term memory, a store for what outlives a conversation, and the trimming, summarisation and subgraph patterns that keep a long-running agent’s context window from filling with its own history.

An agent is only as useful as the systems it can reach, and hand-wiring each one does not survive the second integration. MCP is how a tool gets exposed once and called by anything — including agents you did not write. The “USB-C for AI”: client–server handshake, resources, tools, and prompts over JSON-RPC; exposing context with governance and controlled access; connecting MCP endpoints to planning and multi-agent patterns. Labs: building an MCP client in Streamlit.

By now you can build an agent. This is the module that tells you whether it is any good — because without a number, “it feels better” is the only review you can give a change, and no one signs off on a demo. Benchmarks (MMLU, HELM, BBH), text-quality metrics (BLEU, ROUGE, BERTScore, perplexity), RAG-specific evaluation with RAGAs, open-ended scoring with G-Eval, and safety evaluation — measuring what actually matters before shipping.

Evaluation tells you whether a system was good on your test set; observability tells you what it did last night, on traffic no test set anticipated. It is the half of trustworthy AI that only starts working after launch. Traces, spans and sessions across a multi-step agent run; how to instrument one and which tools to run it through; online evaluation against live traffic; capturing user feedback and human annotation; the cost, latency and drift signals that belong on a dashboard before launch; and how to work back from a failed run to the step that actually broke it.

The most expensive wrong turn in an LLM project is fine-tuning a model when better retrieval would have fixed the problem. This module is about knowing which lever you are reaching for, and what it costs to pull. Full fine-tuning versus LoRA and QLoRA; when fine-tuning beats RAG and when it cannot help; data quality as the primary lever. In class: instruction fine-tuning and deploying a quantized LLaMA model. Take-home: fine-tuning on Azure AI Studio.

You have seen every part by now. This is how they get assembled, so you start your next project from a shape that already works in production instead of inventing one against a deadline. The handful of architectures that recur across production agent deployments — single agent with tools, supervisor and workers, hierarchical teams, human-in-the-loop review — with the serving infrastructure, cost, reliability and rollout characteristics of each. Includes running agents safely inside Docker sandboxes, with policy enforcement and a reusable take-home deployment pattern.

By now you have seen every component of an agentic application on its own. This is where they become one system. You are given boilerplate code for each piece and instructions for assembling them in increasing order of complexity: a bare LLM wrapper with no agentic behavior, then memory, a vector database, and further context through the MCP servers the job needs; then reasoning, from ReAct through reflection to deep agents; then the guardrails that make it safe to run. You push the result to your own GitHub account and deploy it to a dedicated cloud URL you can share. You leave with an end-to-end understanding of the architecture and hands-on experience of the full lifecycle.