Technical AI Literacy

Tokens, not words

LLMs don't process text the way humans read it. They operate on tokens — fragments of text that can be whole words, parts of words, punctuation, or spaces. The word tokenisation splits into roughly two tokens: 'token' and 'isation'. A typical page of prose contains 500–700 tokens. This matters because LLMs have a fixed context window — the amount of text they can process at once — and that limit is measured in tokens, not words or pages.

The standard tokenisation method is Byte-Pair Encoding (BPE): starting from individual bytes or characters, the algorithm repeatedly merges the most frequently co-occurring adjacent pairs until it reaches a target vocabulary size — typically 32,000 to 128,000 tokens. Common English words like 'the' or 'and' become single tokens; rarer words and technical terms fragment into multiple subword units. This is why LLMs perform better on common English than on unusual proper nouns, non-Latin scripts, or domain-specific jargon that appeared rarely in training data — those strings consume more tokens per unit of meaning.

Training: learning from vast text

Before an LLM can do anything useful, it undergoes pre-training: processing hundreds of billions of tokens of text — books, websites, code, scientific papers — and learning to predict what comes next. This isn't memorisation; it's pattern recognition at extreme scale. The model adjusts billions of internal numerical parameters (called weights) until it becomes highly reliable at predicting the next token given all previous ones. This process requires enormous compute and typically takes weeks to months of continuous GPU time.

After pre-training, models are usually fine-tuned — trained further on curated, higher-quality examples to follow instructions, be helpful, and avoid harmful outputs. This second stage is where much of what makes a model feel aligned with human expectations comes from.

The Transformer architecture

Almost every modern LLM is built on the Transformer architecture, introduced in a landmark 2017 Google paper ('Attention Is All You Need'). The key innovation is the self-attention mechanism: rather than processing text sequentially left-to-right, every token can attend to every other token simultaneously, regardless of distance.

Attention works by computing three representations for each token: a Query (what this token is looking for), a Key (what this token offers to others), and a Value (what this token contributes when attended to). Attention scores are computed by taking the dot product of each token's Query against every other token's Key, then normalising with softmax so scores sum to 1. Each token's output representation is then a weighted sum of all Values — effectively asking: given what I am looking for, how much should I attend to each other token? This happens in parallel across all tokens simultaneously, which is why Transformers train far faster than the sequential RNN architectures they replaced.

Modern LLMs stack many Transformer layers (32–128 for frontier models), with each layer running multiple attention patterns in parallel (multi-head attention — typically 32 to 128 heads). Earlier layers tend to capture syntactic relationships; later layers encode higher-level semantic concepts. Between attention sub-layers sit feed-forward networks that apply learned non-linear transformations, giving the model its representational depth.

Positional encoding

The attention mechanism has a subtle limitation: it is position-agnostic. When a token computes its Query-Key dot products, it attends based on semantic similarity — not on where in the sequence the other token appears. Left uncorrected, 'The dog bit the man' and 'The man bit the dog' would produce identical attention patterns — same tokens, different meaning.

Positional encodings solve this by adding a position-dependent signal to each token's embedding before it enters the Transformer layers. The original 2017 Transformer used fixed sinusoidal encodings — mathematical functions of position that produce a unique pattern for each sequence slot. Modern LLMs almost universally use RoPE (Rotary Position Embedding), which encodes position by rotating the Query and Key vectors before the dot product. This makes attention scores naturally decay with distance, and crucially enables context window extension: by adjusting the rotation frequency at inference time, models can generalise to longer contexts than they were trained on — one mechanism behind how providers extend 8K base models to 128K or beyond.

Mixture of Experts

Most frontier models today are not dense Transformers — they use a Mixture of Experts (MoE) architecture. In a dense Transformer, every parameter activates for every token. MoE replaces the feed-forward sub-layer in each Transformer block with a set of specialised 'expert' networks — typically 8 or 16 — and a learned router. For each token, the router selects a small subset of experts (usually 2) to process it; the remaining experts do not activate and incur no compute cost.

This decouples total parameter count from active parameter count. Mixtral 8x7B has 46.7 billion total parameters but only 12.9 billion active per token — delivering quality comparable to a ~46B dense model at the inference cost of a ~13B one. GPT-4 is widely understood to use MoE, as do Gemini and most other frontier models. The trade-off: MoE models require more total VRAM to hold all experts in memory simultaneously, and training requires careful load-balancing to prevent certain experts from being underused. For inference throughput, the active-parameter advantage is substantial.

Inference: generating one token at a time

When you send a message to an LLM, it doesn't retrieve a pre-written answer. It generates a response one token at a time, each token being a probabilistic selection based on everything that came before. This is why outputs vary between identical prompts, why models can be confidently wrong (the next token is probable, not necessarily true), and why longer responses take longer to generate — the model produces each token sequentially.

Mechanically: the model passes the full context through all its layers and produces a logit (a raw score) for every token in its vocabulary — typically 50,000 to 128,000 entries. These logits are converted to probabilities via a softmax function. One token is sampled, appended to the context, and the entire process repeats. At temperature 0, the model greedily selects the highest-probability token every time, producing deterministic output. At higher temperatures, logits are divided by T before softmax, flattening the distribution so lower-ranked tokens get a meaningful share of the probability mass. Top-p sampling (nucleus sampling) further trims this to the smallest set of tokens whose cumulative probability exceeds a threshold p — cutting off the long tail of improbable vocabulary entirely.

Context windows

The context window is the total amount of text an LLM can see at once — your conversation history, any documents provided, system instructions, and the model's own previous responses. Modern frontier models support windows from 128,000 to over one million tokens. When content exceeds this window, earlier context is dropped. This is a real engineering constraint in enterprise applications that need to reason over long documents or extended conversations.

Parameters: what the numbers actually mean

When a model is described as having '7 billion parameters' or '405 billion parameters', those numbers refer to its weights — individual numerical values (typically stored as 16-bit floats) that encode everything the model learned during training. A 7B model stores roughly 14 GB of data at 2 bytes per parameter; a 70B model requires ~140 GB; a 405B model exceeds 800 GB. This is why running large models locally requires substantial GPU memory: the full weight set must fit in VRAM before any inference can begin.

Scaling laws (established by DeepMind's Chinchilla paper, 2022) showed that optimal model performance depends jointly on parameter count and training token volume. Doubling parameters without also scaling training data yields diminishing returns. This insight drove the shift toward training smaller models on vastly more tokens — producing models like Llama 3 8B that significantly outperform earlier 30B models trained on less data. Parameter count is a proxy for capacity, not a direct measure of capability.

Frontier models vs. the rest

Frontier models are the most capable models available at any given time — the ones actively pushing the boundary of what AI can do. As of 2025, these include GPT-4o (OpenAI), Claude 3.5 and 3.7 Sonnet (Anthropic), Gemini 2.0 and 2.5 (Google DeepMind), and Llama 3 (Meta). Below the frontier sit smaller, faster, cheaper models — often distilled or fine-tuned variants — that handle most enterprise tasks at a fraction of the cost. Choosing between frontier and sub-frontier is primarily an economics and capability trade-off, not a prestige decision.

Open-source vs. closed and proprietary

Closed models — GPT-4o, Claude, Gemini — are accessed only via API. You cannot inspect or modify the underlying weights. Open models — Llama 3, Mistral, Gemma — release their weights publicly, allowing anyone to run, inspect, and fine-tune them on their own infrastructure. Open models offer data sovereignty and cost control at the expense of setup complexity. Closed models are typically more capable at the frontier but create vendor dependency and offer less visibility into how the model behaves internally.

Running models locally

Open-source models can run entirely on your own hardware — no API calls, no data leaving your machine, no per-token cost after setup. Ollama is the simplest entry point: a command-line tool that downloads quantised models and runs them with a single command. It exposes an OpenAI-compatible API endpoint, making it a drop-in local replacement for cloud APIs during development. LM Studio provides a desktop GUI for browsing, downloading, and running models — suited to users who prefer not to use the command line. llama.cpp is the underlying C++ inference engine powering most local tools; it supports GGUF-quantised models and can fall back to CPU when GPU VRAM is insufficient.

The practical constraint is hardware. A 7B model at INT4 quantisation requires roughly 4–5 GB of VRAM or unified memory; a 13B model requires ~8 GB; a 70B model at INT4 requires ~40 GB. Modern MacBooks with Apple Silicon (M2/M3/M4) run 7B and 13B models well via their unified memory architecture. Running locally makes most sense for: development and testing without API latency or cost, privacy-sensitive workflows where data cannot leave the machine, high-volume inference where annual API spend would exceed hardware costs, and exploring model behaviour with unrestricted access to generation parameters.

The major labs

OpenAI introduced GPT and ChatGPT, which has roughly 500 million users and the strongest brand recognition in the space. Anthropic, founded by former OpenAI researchers, builds the Claude series with a safety-first approach and strong performance on reasoning and long-document tasks. Google DeepMind produces the Gemini series, with particular strength in multimodality and deep integration across Google's product ecosystem. Meta AI releases the Llama series fully open-source, making it the most widely used base for fine-tuning and research. Mistral is a European lab producing highly efficient models with a strong open-source presence and growing enterprise adoption.

Fine-tuned and specialised models

Base pre-trained models are rarely used directly in products. Most AI applications use fine-tuned variants — models further trained on domain-specific data for coding (GitHub Copilot, Cursor), legal analysis (Harvey), medicine (Med-PaLM), or specific enterprise workflows. When a vendor says they use a customised AI model, this is usually what they mean: a general base model adapted for a specific task.

Benchmarks and capability claims

AI benchmarks — MMLU, HumanEval, GPQA, MATH — measure performance on specific tasks: professional exam questions, coding problems, graduate-level reasoning. They are useful reference points but imperfect guides. Models can be specifically optimised to perform on popular benchmarks without being broadly better. When evaluating a model for your organisation's use case, treat benchmarks as a starting filter, then test against your actual tasks and data.

The stack: model to user

Most people interact with AI through a product interface — a chatbot, an embedded assistant, a search tool. What they see represents only the top layer. Beneath it is a stack: the foundation model (the LLM itself), an orchestration layer (code that manages conversations, tools, and context), a data layer (documents and knowledge the model can access), and the application layer (the interface users see). Understanding this stack helps you diagnose problems, evaluate vendor products, and make better architectural decisions.

Embeddings

Embeddings are numerical representations of text — vectors of hundreds or thousands of numbers that encode semantic meaning. Two sentences with similar meanings will have similar embeddings, even if they use completely different words. This is the foundation of semantic search and AI retrieval systems. When an AI product searches a knowledge base, it is almost certainly comparing embeddings, not doing keyword matching. This is why you can search for employee holiday entitlement and retrieve a document about annual leave policy.

A typical embedding model produces a vector of 1,536 dimensions (OpenAI's text-embedding-3-small) or 3,072 dimensions (text-embedding-3-large). Similarity is measured using cosine similarity: the cosine of the angle between two vectors. Identical meaning → angle near 0° → cosine similarity near 1.0. Unrelated meaning → angle near 90° → cosine similarity near 0. Finding the most similar vectors across millions of stored embeddings uses approximate nearest-neighbour indexing algorithms — HNSW (Hierarchical Navigable Small World graphs) is the most widely deployed — enabling millisecond search across document stores that would otherwise take seconds with brute-force comparison.

Vector databases

Vector databases — Pinecone, Weaviate, pgvector — are purpose-built for storing and querying embeddings at scale. They can find semantically similar content across millions of documents in milliseconds. Any enterprise AI system that needs to search large bodies of knowledge will include a vector database or equivalent. Understanding this layer also explains why AI knowledge bases have a delay when new content is added — documents must be embedded and indexed before they become searchable.

Retrieval-Augmented Generation (RAG)

RAG is the most important architectural pattern in enterprise AI. The problem it solves: you cannot fit all your organisation's knowledge into a model's context window, and fine-tuning the model on all your data is prohibitively expensive and creates a static snapshot. RAG retrieves the most relevant documents at query time and injects them into the prompt. The model then generates a response grounded in that retrieved content. Most enterprise AI assistants — knowledge bases, document Q&A tools, customer support bots — use RAG.

RAG quality depends heavily on engineering decisions beyond the basic pipeline. Chunking strategy — how documents are split before embedding — is critical: chunks too large dilute retrieval precision; chunks too small lose surrounding context. Typical chunk sizes are 256–1,024 tokens with 10–20% overlap between adjacent chunks to avoid splitting relevant content across boundaries. Retrieval precision can be improved with re-ranking: after the initial vector search returns top-k candidates, a cross-encoder model scores each chunk against the specific query and reorders results by true relevance — more accurate than vector similarity alone but slower to compute.

APIs and function calling

Models are almost always accessed via API — a standard interface for sending text in and receiving text or structured data back. Modern LLM APIs support function calling: the model can request that your code execute a specific function and return the result. This is how AI assistants can search the web, query your database, check calendar availability, or look up a customer record mid-conversation. The API layer is where model capability meets your data and systems.

System prompts vs. user prompts

Every LLM application has two types of input: the system prompt and the user prompt. The system prompt is written by the developer, sets the model's persona, defines its task and constraints, and the end user typically never sees it. The user prompt is what the user actually types. The system prompt is extraordinarily powerful — it frames everything that follows. A customer service chatbot that feels tuned for a specific company is mostly a well-crafted system prompt layered on top of a general model. When you use a company's AI assistant, you are interacting with their system prompt as much as with the underlying model.

Temperature and sampling

When the model selects the next token, it does not always choose the most probable option. Temperature is a parameter that controls randomness: at 0, the model always selects the highest-probability token (deterministic, consistent, sometimes repetitive); at 1, it samples more freely across possibilities (creative, varied, sometimes unpredictable). Most production applications set temperature between 0.2 and 0.7. Use lower temperature for factual extraction and structured outputs; higher temperature for creative generation and brainstorming.

Mechanically, temperature T is applied by dividing the model's raw logits by T before the softmax step. When T < 1, logit differences are amplified: the highest-scoring token becomes even more dominant. When T > 1, differences are compressed: probabilities spread across more candidates. Top-p sampling (nucleus sampling) works alongside temperature by restricting selection to the smallest group of tokens whose cumulative probability mass exceeds threshold p — at p = 0.95, the lowest-probability long tail is excluded entirely, preventing very improbable tokens from ever being selected even at high temperatures.

Other generation parameters

Temperature and top-p are the most commonly tuned parameters, but production LLM applications regularly use several others. top_k restricts sampling to the k most probable tokens at each step (typically 40–50), providing a simpler alternative to top-p. repetition_penalty applies a multiplicative discount to tokens that have already appeared in the output — reducing the looping and repetitive phrasing that long generations tend to produce. frequency_penalty and presence_penalty (OpenAI's terminology) are similar but distinguish between tokens that appeared frequently (frequency_penalty reduces their probability proportionally to count) versus tokens that appeared at all (presence_penalty applies a fixed one-time penalty).

stop_sequences are strings or tokens that halt generation immediately when produced — essential for structured outputs where generation must terminate at a defined delimiter. max_tokens caps output length and is a critical cost-control parameter: without it, an unusually verbose response can multiply API costs unexpectedly. seed fixes the random state for deterministic outputs — at temperature 0 with a fixed seed, responses should be identical across calls, enabling reliable automated testing. Understanding these parameters collectively gives you precise control over the consistency, style, length, and cost of model outputs.

Context window management in production

Every token counts against the context window: the system prompt, the full conversation history, retrieved documents, the current message, and the model's own response. In long conversations or document-heavy applications, the window fills. When it does, either earlier content is truncated (the model effectively forgets it), or the application must summarise and compress prior context. This is a genuine engineering challenge — most users never notice because well-designed applications handle it invisibly, but it is a core constraint shaping every enterprise AI deployment.

Why prompts fail

Hallucination occurs when the model generates plausible but incorrect information. This is a fundamental property of the architecture: the model selects probable tokens, not verified facts. It cannot reliably distinguish between what it knows and what it infers. This cannot be fully eliminated with better prompting — it can be mitigated with retrieval grounding (RAG), output verification, and human review processes.

Instruction complexity degrades reliability. Models follow one clear instruction better than five ambiguous ones. Complex multi-step prompts produce less consistent outputs than decomposed, sequential ones. Prompt injection is a distinct security concern: malicious instructions embedded in user inputs or retrieved documents can override system prompts and alter model behaviour — a real risk in agentic and document-processing systems.

Few-shot prompting

Zero-shot prompting gives the model a task with no examples. Few-shot prompting includes two to five examples of the desired input-output pattern before the actual task. Few-shot consistently improves performance on structured, domain-specific, or format-sensitive tasks because it demonstrates the expected reasoning style and output format — reducing the model's uncertainty about what a good response looks like in your specific context.

Chain-of-thought and extended thinking

Chain-of-thought (CoT) prompting elicits step-by-step reasoning by providing examples that demonstrate intermediate reasoning steps, or simply appending 'Let's think step by step' to the prompt. This works because of the auto-regressive nature of LLMs: generating intermediate reasoning tokens makes subsequent correct tokens more probable — the model talks itself through the problem before committing to a final answer. On multi-step reasoning benchmarks, CoT prompting improves accuracy by 20–40 percentage points, but only on models above roughly 10 billion parameters — suggesting that explicit reasoning is an emergent capability that appears at scale.

Extended thinking — implemented in models like Claude 3.7 Sonnet (thinking mode) and OpenAI o3 — is a systematic version of this principle. These models generate extended internal reasoning (sometimes thousands of tokens of scratchpad) before producing a visible response. This shifts compute from training time to inference time: harder problems get more tokens of reasoning. The trade-off is latency and cost — more thinking tokens means slower and more expensive responses — but for complex reasoning tasks the accuracy gains are substantial.

Context engineering

Prompt engineering focuses on the instructions you give the model. Context engineering is the broader discipline of deciding what goes into the context window — and in what form. As windows expand toward one million tokens, the question of what to include becomes as important as how to phrase the instruction. Blindly filling the context with everything potentially relevant degrades performance: models exhibit a 'lost in the middle' failure mode where relevant information buried in a long context is less reliably used than information near the beginning or end.

Context engineering decisions include: conversation history management (how many prior turns to include; when to summarise rather than truncate); document injection strategy (full document vs extracted passages vs summaries); structured vs prose context (tables and JSON can be more token-efficient than natural language for certain data types); and context ordering (placing the most critical content at the start or end rather than the middle). In agentic systems, it also covers what intermediate results to retain across steps and what to discard to prevent context overflow in long-running tasks. As context windows grow, context engineering is increasingly the dominant skill in applied LLM work.

What makes a system agentic

For the first years of the LLM era, most AI interactions were transactional: a user sends a message, the model replies, done. Agentic systems change this fundamentally. An agent is an LLM that can take actions, not just generate text. It operates in a loop: observe input, plan a response or action, execute that action (call a tool, search the web, write code, send a message), observe the result, and repeat — until the task is complete or the model determines it is done. The key distinction is that agentic systems persist and act across multiple steps rather than responding in a single shot.

Tool use and function calling

Modern LLMs support tool use: you provide the model with a list of available tools — web search, database query, email send, code execution, calendar check — with descriptions of what each does. The model decides which tools to call, when to call them, and how to interpret their outputs. It then incorporates those results into its ongoing reasoning. This is how AI assistants move beyond conversation into genuine task completion.

Function calling works at the protocol level: tool definitions are included in the API request as a JSON schema specifying each tool's name, description, and parameter types. When the model determines it should use a tool, it returns structured JSON — rather than prose — specifying the function name and arguments. Your application code executes the function and returns the result, which is appended to the context for the model's next call. The model never directly executes anything: it only requests that your code does. This separation is important for security — it means you control exactly what actions an agent can take, and you can log, audit, and gate every tool invocation.

Agentic design patterns

Several repeating patterns have emerged in how capable agent systems are structured. ReAct (Reason + Act) is the most common: the model alternates between reasoning about its situation and taking an action, making its thinking visible and auditable in the trace. Reflection adds a second step where the model reviews its own output and iterates — effectively self-editing before returning a result. Planning agents decompose a complex task into a structured sequence of subtasks before executing any of them, reducing compounding errors on long-horizon work.

Parallelisation runs multiple agent calls simultaneously — useful when subtasks are independent (e.g. researching several topics at once) — then merges results. Specialisation splits work across purpose-built agents (a research agent, a writing agent, a code-review agent), each with a focused system prompt and limited tool set. An orchestrator agent routes tasks and synthesises results. These patterns are often combined: a planning orchestrator that fans out parallel specialised agents and then applies reflection before returning a final answer.

Reliability degrades quickly as agent chains grow longer. Each LLM call has some probability of error; errors compound multiplicatively across steps. This means the most robust production agents are either short-chained (two or three hops) or include explicit verification steps — another agent checking the output, a structured self-critique loop, or a deterministic validation function before the result is accepted.

The Model Context Protocol (MCP)

Until 2024, every agent framework had its own proprietary way of defining tools, connecting to external services, and passing results back to the model. This created fragmentation: a tool built for LangChain could not be used in AutoGen, and every new integration required custom glue code. The Model Context Protocol (MCP), introduced by Anthropic in late 2024, is an open standard that defines a uniform interface between AI models and the data sources or tools they connect to.

MCP works as a client–server protocol. An MCP server exposes a set of tools, resources, or prompts through a standardised JSON-RPC interface. An MCP client — which could be Claude Desktop, an IDE plugin, or a custom agent framework — discovers and calls those tools through the same protocol regardless of what the server is built on. This means a single MCP server wrapping, say, a company's CRM can be used by any MCP-compatible model or host without rewriting the integration. The ecosystem is growing rapidly: hundreds of community-built MCP servers now exist for databases, APIs, file systems, browser control, and SaaS platforms.

For developers, MCP is significant because it standardises the interface that has historically been the most painful part of building agents. Rather than writing bespoke function-calling schemas for every capability, you build or install an MCP server once and it becomes available to any compliant model. The protocol also defines resource exposure (streaming file contents, database rows) and prompt templates — making it broader than pure tool-calling. MCP is increasingly treated as infrastructure-level for agentic systems the way HTTP is for web services.

Multi-agent systems

Complex tasks can be distributed across multiple specialised agents. A research agent gathers information, a writing agent drafts content, a critique agent reviews it, and an orchestrator agent coordinates the workflow. This pattern allows more reliable completion of long, multi-step tasks by decomposing them — but it introduces coordination complexity, higher cost (each agent makes its own LLM calls), and compounding error risk: a mistake early in the chain can propagate and amplify through subsequent steps.

Memory architecture

Short-term memory is the context window — everything the model has seen in the current session. Long-term memory is external storage — facts, preferences, or summaries of past interactions persisted in a database and retrieved when relevant. Episodic memory refers to summaries of previous sessions injected at the start of new ones. Most current agent frameworks have limited and unreliable long-term memory. This remains one of the most actively researched problems in the field, and a major practical limitation for agents that need to learn from experience over time.

Real-world examples

You have already encountered agentic systems: GitHub Copilot completing a multi-file refactor autonomously, Perplexity chaining multiple web searches to answer a complex question, Claude Projects maintaining context across sessions, or an AI sales assistant that researches a prospect, drafts an outreach email, and schedules a follow-up without human intervention. Enterprise agentic deployments — AI that takes actions inside business systems — are accelerating significantly through 2025 and beyond.

Why GPUs?

Graphics Processing Units were designed for the parallel computations required to render graphics — performing thousands of simple operations simultaneously rather than a smaller number of complex ones sequentially. Training and running neural networks requires almost identical mathematical operations (matrix multiplications at very large scale). This is why NVIDIA, originally a gaming hardware company, became the most important infrastructure supplier in the AI industry. A modern frontier model training run uses thousands of GPUs operating in parallel for weeks or months.

The core operation in a neural network is a matrix multiplication — thousands of simple numerical operations applied simultaneously across large grids of numbers. CPUs are designed for sequential, complex tasks and have a small number of powerful cores. GPUs have thousands of simpler cores designed to run in parallel, which is exactly what neural network workloads require. This makes a modern GPU not just faster but fundamentally better suited to AI than a CPU — and it is why NVIDIA, a gaming hardware company, became the most important infrastructure supplier in the AI industry. Beyond raw processing speed, the speed at which data can be moved between GPU memory and its processing cores matters enormously — and frontier GPUs are optimised for this as much as for raw computation.

NVIDIA's dominance and the CUDA moat

NVIDIA's H100 and H200 GPUs are the gold standard for AI training and inference. But the hardware alone does not explain NVIDIA's position — the CUDA software ecosystem does. Built over 15 years, CUDA is a parallel computing platform that most AI frameworks, libraries, and tools are optimised for. Switching away from NVIDIA GPUs requires not just different hardware but porting significant amounts of software. This creates a durable moat that competitors from AMD, Intel, and Qualcomm are working to bridge.

Custom silicon

Google's Tensor Processing Units are purpose-built chips for neural network workloads — highly efficient for training and inference at Google's scale, accessible exclusively through Google Cloud. Other major players — Amazon (Trainium, Inferentia), Microsoft (Maia), Meta (MTIA) — have developed custom silicon to reduce dependence on NVIDIA and control their own cost structures. These chips remain largely internal tools, but their existence signals the scale at which the major labs are operating.

Data centres and energy

A single NVIDIA H100 GPU draws approximately 700 watts of power. A cluster of 10,000 GPUs — modest by frontier training standards — consumes roughly 7 megawatts, equivalent to powering a small town. Training a frontier model like GPT-4 is estimated to have consumed 50–100 gigawatt-hours. At scale, energy availability and cost have become the primary constraint on AI development. Microsoft, Google, and Amazon have all signed agreements with nuclear power providers to secure dedicated generation capacity for AI data centres.

Cloud vs. on-premise inference

Most organisations run AI inference in the cloud — AWS, Azure, Google Cloud, or specialist providers like CoreWeave. On-premise inference makes sense in two scenarios: data sovereignty requirements that prohibit cloud processing, or query volumes so high that annual API spend would exceed the capital cost of owning and operating hardware. For most organisations at current scale, cloud inference is the right default. The break-even point for on-premise typically requires substantial usage and a dedicated ML infrastructure team.

Training vs. inference — two very different cost centres

AI has two distinct cost structures. Training is a largely one-time expense — the compute required to create the model. Training GPT-4 is estimated to have cost $50–100 million. Llama 3 70B cost Meta roughly $10–20 million. These numbers are falling with algorithmic improvements, but frontier training remains a capital expenditure accessible only to well-funded labs. Inference is the ongoing cost of running the model to serve requests — what you pay every time someone uses an AI product. Inference costs have fallen dramatically and are the relevant budget line for almost every organisation.

API pricing in practice

Most organisations access AI via API, paying per token. Pricing varies substantially by model tier. Frontier models such as GPT-4o and Claude 3.5 Sonnet typically cost $3–15 per million input tokens. Mid-tier models such as GPT-4o mini and Claude Haiku cost $0.15–1 per million tokens. Open-source models hosted via third-party providers often cost under $0.10 per million tokens.

For context: one million tokens is approximately 750,000 words — a substantial volume of text. Most enterprise applications consuming AI at scale spend thousands to tens of thousands of dollars per month on API calls. Understanding token consumption is essential for accurate AI budget planning, since usage-based pricing requires different financial controls than per-seat SaaS.

Make vs. buy decisions

Should your organisation fine-tune its own model or use a commercial API? For the overwhelming majority of organisations, the answer is: buy. Fine-tuning requires ML engineering expertise, data curation, training infrastructure, and ongoing maintenance. It makes economic sense in a narrow set of circumstances: data privacy requirements that prevent external API use, query volumes so high that annual API spend exceeds the cost of owning and operating a model, or capability gaps that commercial models genuinely cannot fill. Start with a commercial API. Revisit when scale and requirements justify the investment.

The cost trajectory

AI inference costs have followed a consistent pattern: roughly 10x cheaper every 12–18 months. This trajectory has significant strategic implications. Use cases that are economically unviable at today's pricing — processing millions of customer records, running real-time analysis across all support interactions — may be trivially affordable in 18 to 24 months. Evaluate AI investments against the expected cost trajectory, not just today's pricing. The business case for automation that looks marginal now may be compelling within two years.

Training data and copyright

Frontier models are trained on vast datasets scraped from the internet, including content that may be under copyright. This has led to significant litigation — most notably The New York Times versus OpenAI — with outcomes still being determined in courts. For enterprise users deploying AI products, the practical exposure is limited. You are not redistributing training data; you are using the model's generated outputs. The legal risk sits primarily with the model providers, not their API customers. Monitor for developments, but this should not be a reason to avoid AI deployment.

What happens to your inputs

This is the question every enterprise procurement and legal team asks. The answer depends on contract type. By default, OpenAI, Anthropic, and Google do not use API inputs to train future models — the API channel is treated differently from consumer products. Enterprise agreements — Azure OpenAI Service, Anthropic's enterprise tier, Google Vertex AI — typically include contractual guarantees that your data is not used for model training.

Consumer products (ChatGPT free, Gemini free) operate under different terms where inputs may be used for training unless opted out. Do not extrapolate consumer product terms to enterprise API agreements — they are not the same document, and the distinction matters significantly for compliance.

Fine-tuning on proprietary data

Fine-tuning a model on your organisation's internal data — documents, emails, customer records — raises specific questions: Who holds the resulting model weights? What data was exposed during the training process, and to whom? Is the fine-tuned model itself a security risk if accessed without authorisation? These are solvable problems but require deliberate data governance decisions before beginning. Establish data classification, access controls, and retention policies before engaging in any proprietary fine-tuning project.

Data residency and sovereignty

Regulatory frameworks in the EU, financial services, healthcare, and government require that certain categories of data remain within specific geographic boundaries. Major cloud providers offer region-locked deployments — Azure OpenAI in EU regions, AWS Bedrock with data residency guarantees, Google Cloud's regional endpoints — that satisfy most regulatory requirements. On-premise deployment of open-source models is the most restrictive-compliant option but carries operational overhead. Data residency is now a procurement decision as much as a technical one.

Prompt injection

Prompt injection is a security vulnerability specific to AI systems: malicious instructions embedded in user inputs or retrieved content can override system prompts and cause the model to behave in unintended ways. An AI assistant that processes incoming emails could be manipulated by a carefully crafted email body designed to exfiltrate information, override access controls, or produce harmful outputs. Mitigations include input validation, output monitoring for anomalous behaviour, limiting the actions agents can take autonomously, and applying the principle of least privilege to AI system permissions.

Alignment: the core challenge

Alignment refers to the challenge of ensuring AI systems do what humans actually want — not merely what they have been literally instructed to do. An LLM trained purely to predict text can produce harmful, false, or manipulative outputs without any intent to do so. Alignment research attempts to close the gap between following the literal instruction and doing what was genuinely intended, in context, including situations the instruction did not anticipate.

This problem scales with capability. A more capable but misaligned system can cause more harm. This is the foundational concern driving AI safety research, and it becomes increasingly important as systems gain greater autonomy.

RLHF: teaching models human preferences

Reinforcement Learning from Human Feedback is the primary alignment technique used by frontier model developers. Human raters are shown pairs of model outputs and asked to indicate which is better. This preference data trains a reward model — a separate neural network that learns to assign a scalar score to any (prompt, response) pair, predicting what human raters would prefer. The reward model then drives a reinforcement learning loop: the LLM is fine-tuned using Proximal Policy Optimisation (PPO) to generate responses that earn higher scores. A KL-divergence penalty prevents the policy from drifting too far from the base pre-trained distribution — without it, the model would 'collapse' into a narrow set of reward-hacking responses that score well without being genuinely better.

RLHF is effective but imperfect. It encodes the biases and blind spots of its human raters, can lead to reward hacking (the model learns to produce outputs that score highly on the reward model without being genuinely better), and can make models overly cautious or deferential in ways that reduce usefulness. The rater pool — who they are, what they are instructed to optimise for, and how they are paid — has substantial influence on the resulting model's behaviour, making it one of the least transparent aspects of frontier model development.

Constitutional AI

Anthropic's Constitutional AI approach provides the model with a set of principles and uses AI feedback rather than human feedback to evaluate outputs against those principles. The model critiques and revises its own outputs based on this constitution before producing a final response. This scales more efficiently than human feedback, produces models that can articulate why they declined a request, and is the primary alignment technique behind the Claude model series.

Red-teaming and jailbreaks

Red-teaming is the deliberate, systematic attempt to find failure modes in AI systems before deployment — adversarially prompting the model to produce harmful outputs, bypass safety measures, or behave unexpectedly. Major labs conduct extensive internal red-teaming and commission external red teams before major model releases. Jailbreaks are successful circumventions of safety guardrails. Despite extensive red-teaming, jailbreaks continue to emerge. Safety measures should be understood as a significant reduction in risk, not an elimination of it.

Output filtering and defence-in-depth

Model-level alignment is one layer of safety. Most production deployments add additional filtering layers: classifiers that detect harmful or policy-violating content in model outputs, input monitoring for known attack patterns, and human review pipelines for high-stakes automated decisions. The right approach is defence-in-depth — multiple independent safety layers, so the failure of any single layer does not produce an unacceptable outcome.

The EU AI Act

The EU AI Act — the world's first comprehensive AI regulation — establishes a tiered risk framework. Unacceptable-risk applications are prohibited: AI-based social scoring, real-time biometric surveillance in public spaces, subliminal manipulation of vulnerable groups. High-risk applications face substantial obligations: conformity assessments, mandatory human oversight, transparency requirements, logging and audit trails, and registration in an EU database. High-risk categories include AI used in hiring and employment decisions, credit scoring, healthcare diagnosis, educational assessment, law enforcement, and critical infrastructure.

Limited-risk applications face disclosure obligations: chatbots must identify themselves as AI systems. Minimal-risk applications — most AI tools in common use — face no specific obligations. Any organisation deploying AI that affects EU citizens or operates within the EU needs to assess where their applications fall. Fines for non-compliance reach up to €35 million or 7% of global annual turnover for the most serious violations.

US frameworks

The US approach has been more fragmented: executive orders, voluntary commitments from major AI developers, and sector-specific guidance from financial, healthcare, and other regulators. The NIST AI Risk Management Framework provides a voluntary governance structure — govern, map, measure, manage — that has been widely adopted as a de facto enterprise standard. Sector-specific regulators including the SEC, FDA, CFPB, OCC, and EEOC have issued or are actively developing AI-specific guidance for their domains. Comprehensive federal AI legislation remains pending, but the regulatory direction is clear: disclosure, accountability, and human oversight requirements will grow.

Enterprise AI governance

Regardless of regulatory requirements, robust AI governance is a risk management priority. An enterprise AI governance programme typically covers five areas: an inventory of all AI systems deployed, by whom, for what purpose, and on what data; risk assessment covering what decisions each system influences and the impact of errors; accountability with clear ownership and shutdown authority; audit trails logging inputs, outputs, and decisions; and human oversight defining which decisions require human review rather than full automation, built into system design from the start.

What professionals need to act on now

If your organisation operates in or sells to EU markets, audit your AI deployments for high-risk classification under the AI Act. Adopt the NIST AI RMF as your internal governance baseline if you do not already have one. Review vendor contracts for AI-specific data processing terms, audit rights, and liability clauses — standard SaaS agreements rarely cover AI adequately. Establish an AI system inventory now, even if it is just a spreadsheet. The governance infrastructure that takes a week to build today will take months to reconstruct under regulatory scrutiny.

Why LLM evaluation is fundamentally different

Traditional ML models produce discrete, verifiable outputs: a classifier predicts a label; a regression model predicts a number. Correctness is a binary, computable property. LLMs produce open-ended text where quality is multidimensional — accuracy, helpfulness, fluency, tone, format compliance, and safety matter simultaneously. There is rarely a single correct answer. Two responses can both be correct while differing substantially in quality.

Standard metrics inherited from NLP — BLEU and ROUGE — measure surface-level token overlap between a generated response and a reference string. They fail badly for modern LLM outputs: a response using entirely different words from the reference but semantically superior will score poorly; a response that paraphrases the reference closely but is factually wrong can score highly. BLEU correlates weakly with human judgement in conversational AI contexts and should not be used as a primary signal for evaluating LLM systems.

LLM-as-a-Judge

The dominant approach to reference-free LLM evaluation is LLM-as-a-Judge: using a capable frontier model (typically GPT-4o or Claude Opus) to evaluate the outputs of another model. The judge receives the original prompt, the response being evaluated, and a structured rubric specifying criteria — accuracy, helpfulness, faithfulness, format compliance. It produces a numeric score and a written rationale. G-eval (2023) formalised this approach, demonstrating that LLM judges correlate with human judgements at rates comparable to inter-annotator agreement between humans.

Two modes are standard: pointwise scoring (evaluate one response on a 1–5 scale against a rubric) and pairwise preference (show two responses, ask which is better and why). Pairwise preference is generally more reliable — it is easier to rank two responses than to assign an absolute score. Known failure modes include position bias (preference for whichever response is shown first), verbosity bias (preference for longer responses regardless of quality), and self-preference (a model subtly favours outputs stylistically similar to its own). Mitigate these by randomising response order across evaluation calls and averaging scores over multiple runs.

Component-level evaluation for RAG

A RAG pipeline has two stages that fail independently: retrieval and generation. Evaluating only end-to-end output quality obscures which stage is causing problems — a poor answer could result from retrieving wrong documents (retrieval failure) or from the model misrepresenting the retrieved documents (generation failure). These require different fixes. Evaluating components separately is what makes RAG systems debuggable.

The RAGAS framework defines four metrics that decompose RAG evaluation. Context precision measures whether retrieved chunks are relevant to the query. Context recall measures whether all relevant documents are being retrieved. Faithfulness is the RAG-specific hallucination metric: does every claim in the generated answer have supporting evidence in the retrieved context? A model that generates accurate-sounding statements not found in the retrieved documents is hallucinating, even if those statements happen to be factually correct in the world. Answer relevance measures whether the answer addresses the question asked. All four metrics can be computed automatically using an LLM judge, making them practical at scale.

Multi-turn and task-completion evaluation

Single-turn evals measure one response in isolation. Most production applications are multi-turn or agentic — they span conversations or multi-step workflows. Multi-turn evals assess whether the model maintains coherent context across a conversation, resolves ambiguities gracefully, and handles contradictions without losing track of prior context.

For agentic systems, task-completion rate is the most important metric: given a defined goal, does the agent reach it? This requires constructing test suites with verifiable end states. Coding agents can be evaluated against test suites; research agents against verifiable factual claims; customer support agents against resolution criteria. Trajectory evaluation goes further — assessing not just whether the agent succeeded but whether it did so efficiently, without unnecessary tool calls or redundant reasoning steps. A correct answer reached via 30 LLM calls when 5 would suffice is an engineering problem.

Building a production evaluation framework

Evaluation in production has two distinct purposes. Offline evaluation runs test suites before deployment to catch regressions and validate that prompt changes or model upgrades actually improve quality. Online evaluation samples live traffic to detect distributional shift and real-world failure modes that test suites did not anticipate. Both are necessary; neither alone is sufficient.

A practical eval stack for a production LLM application typically combines: a curated golden dataset of representative examples that must not regress; automated LLM-as-a-Judge scoring on a sample of live traffic; component-level metrics for RAG systems; and human review queues for low-confidence or flagged outputs. This turns evaluation from a one-time pre-launch check into a continuous engineering practice. Frameworks that operationalise this include RAGAS (RAG-specific metrics), DeepEval (unit-test-style assertions for LLM outputs with CI integration), and Promptfoo (prompt regression testing in CI/CD pipelines).

The full training pipeline

The model you interact with via API is the product of up to four distinct training stages, each building on the last. Stage 1 (pre-training) trains on trillions of tokens of raw text, producing a base model that can continue any piece of text but is not conversational. Stage 2 (supervised fine-tuning, SFT) trains on curated instruction-response pairs, making the model helpful and conversational. Stage 3 (preference fine-tuning, RLHF or DPO) uses human preference comparisons to push the model toward helpful, harmless, calibrated outputs. Stage 4 (reasoning fine-tuning, GRPO or similar RL methods) trains on verifiable reasoning tasks — this is what distinguishes models like o3, DeepSeek-R1, and Claude's extended thinking mode from standard instruction-tuned models.

When a vendor describes a model as 'fine-tuned', they almost always mean Stages 2 and 3. When your organisation fine-tunes a model, you are performing an additional Stage 2 (or 3) on top of a base that has already completed the full pipeline — you are adapting a highly capable foundation, not training from scratch.

Why full fine-tuning is impractical for most organisations

Full fine-tuning means updating every single weight in the model — all 7 billion of them in a 7B model, or 70 billion in a 70B model. Doing this requires storing not just the model itself but additional data tracking how each weight should change, which multiplies the memory requirement several times over. A single full fine-tuning run on a 7B model requires roughly 120 GB of GPU memory; a 70B model requires over a terabyte. Very few organisations have hardware at that scale, and renting it is expensive. This is what makes full fine-tuning impractical for most teams.

Beyond hardware, full fine-tuning risks catastrophic forgetting: overwriting the general capabilities the base model acquired during pre-training in order to specialise for a narrow task. The result can be a model that performs well on your specific use case but has degraded on everything else.

LoRA: parameter-efficient fine-tuning

Low-Rank Adaptation (LoRA) solves the compute problem with a clever shortcut: rather than updating the original model weights, it freezes them entirely and adds a small set of new trainable parameters alongside them. These additions are tiny — think of them as a thin layer of adjustments on top of a frozen foundation. Only the adjustments are trained; the original model is untouched. The result is that instead of updating billions of parameters, you are updating a fraction of a percent of that number while achieving comparable task performance.

This reduction is dramatic in practice. A full fine-tuning run on a 7B model might update billions of parameters; LoRA on the same model might update a few million — roughly a 99% reduction. After training, the adjustments can be merged back into the base model with no slowdown at inference time. QLoRA extends this further by also compressing the frozen base model to use less memory, making it possible to fine-tune a 7B model on a single consumer GPU rather than a specialised cluster. This is why fine-tuning has become accessible to individual practitioners and small teams in a way it was not even two years ago.

SFT vs preference fine-tuning vs reasoning fine-tuning

Supervised fine-tuning (SFT) trains the model on (prompt, ideal response) pairs using standard cross-entropy loss — the same objective as pre-training, just on your curated data. The model learns to imitate the provided responses. SFT is effective for teaching output format, domain tone, task-specific vocabulary, and consistent response structure. Its ceiling is the quality of your training data: the model cannot produce outputs better than its examples.

Preference fine-tuning (RLHF or DPO) moves beyond imitation. DPO (Direct Preference Optimisation) presents the model with pairs of responses to the same prompt and updates it to prefer the better one — without a separate reward model or RL training loop, making it significantly simpler than RLHF while achieving comparable results. Reasoning fine-tuning (GRPO) takes a different approach: rather than showing correct outputs, it uses verifiable tasks (maths, code, logic puzzles) and rewards the model for reaching correct answers, allowing it to develop its own reasoning strategies through trial and error. This is the technique behind DeepSeek-R1 and is now widely used to produce models with strong chain-of-thought reasoning.

When to fine-tune: the decision framework

Prompting, RAG, and fine-tuning address different failure modes and should be attempted in that order. Prompting costs nothing and enables instant iteration — exhaust it first. RAG solves knowledge gaps cheaply and keeps your knowledge base current without retraining. Fine-tuning changes how the model behaves at a fundamental level — its style, format, reasoning approach, and implicit assumptions.

Fine-tune when: the model consistently produces incorrect format or tone despite well-crafted system prompts; the task requires domain-specific reasoning patterns that prompting cannot reliably elicit; or latency constraints prevent a retrieval step. Do not fine-tune to add factual knowledge — the model will memorise your facts imperfectly and that knowledge becomes stale the moment training ends. For knowledge, RAG is always the better answer.

Why LLM deployment is different

Deploying an LLM is not like deploying a conventional API. Standard web services are largely stateless: a request arrives, the server processes it in milliseconds, a response is returned. LLMs are autoregressive: they generate one token at a time, sequentially, with each token depending on all previous ones. A 500-token response requires 500 forward passes through a multi-billion-parameter model. Response latency is measured in seconds, not milliseconds, and scales with output length — a simple question with a long answer takes longer than a complex question with a short one.

This creates engineering constraints that differ fundamentally from conventional software. GPU memory is the binding resource rather than CPU or RAM. Throughput and latency are in direct tension: serving more concurrent requests increases throughput but may increase individual response latency. Cost scales directly with token volume, creating financial exposure that per-seat SaaS pricing does not. These constraints demand a specialised operational discipline — LLMops — that conventional DevOps does not fully cover.

KV caching

When a model generates text, it processes every token in the context at each generation step — not just the new token it is about to produce. Without caching, this means re-processing the entire preceding conversation or document on every single step. For a long prompt generating a long response, the redundant computation adds up quickly.

KV caching solves this by storing the model's internal representation of all previously processed tokens in GPU memory. On each new generation step, only the newest token needs to be computed fresh; everything prior is retrieved from the cache. This makes generation significantly faster and cheaper — the longer the context, the greater the benefit. The trade-off is memory: keeping a large cache for many concurrent users consumes significant GPU memory, which is why managing it efficiently is one of the core challenges of LLM serving. This is the problem that vLLM's PagedAttention was specifically designed to solve.

Quantisation

Model weights are typically stored as 16-bit floats (FP16 or BF16) during and after training. At inference, weights can be reduced to lower precision — 8-bit integers (INT8) or 4-bit integers (INT4) — with minimal quality degradation for most tasks. A 7B model in FP16 requires 14 GB of VRAM; the same model at INT8 requires 7 GB; at INT4, approximately 3.5 GB. This is why quantised 7B models run on consumer GPUs with 8 GB VRAM, and why 70B models can be served on a single high-end GPU at INT4 rather than requiring a multi-GPU cluster.

Modern quantisation methods — GPTQ, AWQ, and GGUF — minimise quality loss by identifying weight-sensitive layers and preserving them at higher precision while aggressively quantising less sensitive layers. The practical trade-off: INT8 has negligible quality loss for most production tasks; INT4 shows measurable degradation on complex multi-step reasoning but remains usable for retrieval, summarisation, classification, and general Q&A. For organisations running open-source models, quantisation is the single highest-leverage lever for reducing inference hardware costs.

Inference engines and continuous batching

Naive LLM serving processes one request at a time or groups requests into fixed batches. This is GPU-inefficient: utilisation collapses between requests, and within a fixed batch some sequences finish generating before others, leaving their GPU allocations idle while the batch completes. vLLM addressed both problems. PagedAttention manages KV cache memory using non-contiguous memory pages — similar to how operating systems handle virtual memory — rather than requiring a contiguous block reserved per request upfront. This dramatically increases the number of concurrent requests that can be served from the same hardware.

Continuous batching (iteration-level scheduling) adds new requests to the running batch as soon as existing ones finish generating, keeping GPU utilisation near 100% rather than waiting for an entire batch to complete. Together, PagedAttention and continuous batching give vLLM 10–20x higher throughput than naive serving on the same hardware. For teams self-hosting open-source models, vLLM is the standard inference engine. For cloud API users these optimisations are handled by the provider — but understanding them explains why throughput-optimised API tiers (e.g. batch APIs) are significantly cheaper than real-time endpoints.

Observability: what to instrument

Observability for LLM systems means being able to answer: which calls failed and why, what is the cost per user or feature, where is latency coming from, and are outputs degrading over time? Standard application observability — request logs, error rates, p99 latency — captures part of this but misses the LLM-specific signals that actually matter for debugging and cost control.

Every LLM call should log: the full prompt (system and user), the full response, input and output token counts, latency broken down into time-to-first-token and total generation time, cost, model name and version, and any relevant user or session identifiers. For agentic and RAG systems, traces should capture the entire execution chain — retrieval steps, tool calls, intermediate model outputs — not just the final response. Without this, diagnosing whether a poor output came from a bad retrieval result, a malformed prompt, or model behaviour is guesswork. LangFuse (open-source, full trace support), Helicone (lightweight API proxy requiring no code changes), and Arize (enterprise-grade with drift detection) are the main tools that operationalise this.

Speculative decoding

Speculative decoding is a latency optimisation that exploits a fundamental asymmetry in autoregressive generation: verifying that a token is correct is faster than generating it from scratch. A small, fast draft model generates several candidate tokens ahead; the large target model then verifies all of them in a single parallel forward pass. Accepted tokens are kept; rejected tokens trigger regeneration from the point of mismatch. The final output is guaranteed to be identical to what the target model would have produced alone — there is no quality trade-off.

When the draft model's predictions are accurate — which is common for routine or predictable text — speculative decoding achieves 2–3x throughput improvement with no quality loss. It is most effective for tasks with predictable output patterns: code completion, template-following, structured data extraction. Applied alongside KV caching, quantisation, and continuous batching, it forms part of the complete optimisation stack that makes large-model inference commercially viable at scale.