AI Foundations

Machine learning fits a function to data. Deep learning uses neural networks with many layers so the function can be very complex. Generative AI is deep learning trained to produce outputs in the same form as its inputs (text in, text out). LLMs are generative models trained on text, built on the transformer architecture from 2017; their defining trick is predicting the next token so well that useful behaviour emerges.

Diffusion models (images), speech models, and multimodal models sit in the GenAI ring beside LLMs. Since 2024 most frontier models are multimodal: one model reads text, images, and audio.

Training is an optimisation: pick the parameters that make the model's outputs closest to the labelled answers, measured by a loss function. Module 2 shows the loop. The key consequence is that a model is only as good as the examples: if the labels are biased, late, or from a different population than production, the learned rules are wrong in ways nobody wrote down.

Tokens are sub-word pieces; roughly 3 to 4 characters of English each. Embeddings map each token to a vector of a few thousand numbers. Each transformer layer runs self-attention (each position computes how much to weigh every other position, with multiple 'heads' looking for different relationships) followed by a feed-forward network; residual connections and layer normalisation keep training stable. Frontier models stack dozens of these layers. The final layer produces a score for every token in the vocabulary, turned into probabilities with softmax.

Sampling controls (from the Google prompt-engineering whitepaper): temperature 0 always picks the top token (greedy, deterministic); higher values flatten the distribution and increase variety. Top-K limits choice to the K most likely tokens; top-P keeps the smallest set whose probabilities sum to P. Use low temperature for extraction and classification, higher for brainstorming.

The context window is the maximum number of tokens the model can attend to at once (prompt plus output). Attention cost grows with the square of that length, which is why long contexts are slower and pricier.

Pretraining is unsupervised next-token prediction on a huge corpus; it produces a base model that continues text but does not naturally answer questions. Supervised fine-tuning (SFT) trains on curated prompt-response pairs so the model behaves like an assistant. RLHF (reinforcement learning from human feedback) collects human rankings of candidate answers, trains a reward model to imitate those preferences, then optimises the assistant against the reward model. Newer variants use AI feedback (RLAIF) or verifiable rewards (RLVR) for maths and code, which is how reasoning models are trained.

Parameter-efficient fine-tuning (LoRA and adapters) lets a product team do a small stage-2 style update on top of a vendor model, training a tiny fraction of the weights.

From the Google LLM whitepaper's trade-off section: quality vs latency/cost can often be improved with marginal quality loss by using a smaller model or a quantised one; the practical question is whether the model still performs your task, not whether it lost benchmark points. Latency vs cost is really latency vs throughput: batching more requests per GPU lowers cost per token but raises per-request wait, so bulk offline jobs and interactive chat want different settings.

Quantisation stores weights in 8- or 4-bit integers instead of 32-bit floats: smaller memory, faster arithmetic, usually tiny quality change. Distillation trains a small student model to imitate a large teacher, keeping much of the quality at a fraction of the cost. Output-preserving speedups include prefix caching (reuse computation for a repeated system prompt), speculative decoding (a small model drafts, the big one verifies), and batching.

Prompting covers system prompts, examples, and structured output formats. RAG retrieves relevant passages at request time and puts them in the prompt (next section). Fine-tuning adjusts weights on your examples; parameter-efficient methods make it affordable but it still needs a clean training set and a held-out evaluation set. Agents wrap the model in a loop with tools and memory (section 8). A fifth option, training your own model, is rarely justified outside labs.

Passages are embedded with an embedding model into vectors; similarity is cosine distance. Production systems usually combine vector search with keyword search (hybrid retrieval) and add a reranker that re-scores the top candidates. Agentic RAG (ByteByteGo EP169) lets the model decide whether to retrieve, rewrite the query, or try another source when the first result is empty. Metadata filters (tenant, date, permission) are essential for enterprise deployments so a user never retrieves another user's documents.

The loop is often called ReAct (reason, act, observe). Tool calling is a structured output: the model emits a function name and arguments, the software runs it and feeds the result back as a message. Memory can be short-term (the conversation), working notes (a scratch file), or long-term (a store the agent reads and writes). Multi-agent systems split work across specialised agents; MCP handles tool access while agent-to-agent protocols (A2A) handle their communication.

A four-part agent scorecard: Effectiveness (goal achievement rate, constraint adherence, failure-mode breakdown), Efficiency (steps to success, time to completion, tokens or cost per successful task), Reliability (variance across runs, regression tracking, recovery rate on retry), Trustworthiness (traceability of reasoning and tool use, safety catches before the user, whether a human reviewer would ship the output).

Techniques from the whitepaper: zero-shot (instruction only), one- and few-shot (include examples), system / role / contextual prompting, step-back (ask a general question first), chain of thought (ask for reasoning before the answer, which improves accuracy on multi-step problems), self-consistency (sample several reasoning paths and vote), ReAct (interleave reasoning with tool calls). Best practices: provide examples, design for simplicity, be specific about output, prefer instructions over constraints, control length, use variables, and re-test when the model version changes.

Input guards: classifiers for injection and abuse, allow-lists for tools, per-user rate limits and token budgets. Output guards: PII detection, policy classifiers, schema validation, groundedness checks that compare the answer to retrieved passages. Observability: log prompts, retrieved context, tool calls, and outputs so failures can be replayed. Evaluation (section 12) is how you know the rate of each failure before and after a change. Regulatory context (EU AI Act risk tiers, sector rules) usually maps neatly onto the same high-stakes classification.

Other findings worth quoting: about half of high performers intend to redesign workflows; 64% of all respondents say AI is enabling innovation; expectations for workforce size over the next year split roughly a third decrease, four in ten no change, one in ten increase; and a third of high performers commit more than 20% of their digital budgets to AI. The report defines high performers as organisations attributing more than 5% of EBIT to AI and reporting 'significant' value, about 6% of respondents.

Scoring methods: exact match for classification and extraction; rubric grading by a stronger model (LLM-as-judge) with spot checks; human review for a sample. Track pass rate per category so a regression in one intent is visible. Keep eval data separate from few-shot examples in the prompt (otherwise you are testing on the training set). For agents, score end-to-end task completion and the four scorecard dimensions from section 8 rather than individual messages. Concepts Raviv and Khan single out as product-sense builders: context rot (quality degrades as the window fills), memory, tool selection, and context engineering.