Showing raters two outputs for the same prompt without revealing which model produced which, and asking which is better.
Running two versions side by side on comparable users to measure which is actually better.
Granting access based on attributes — of the user, the resource, and the context (department, time, sensitivity) — rather than fixed roles.
Restricting which documents a given user is allowed to retrieve.
In the paper, accuracy is the percentage of problems solved correctly out of a total.
Standard metrics for objective tasks like classification.
A non-linear function applied to a neuron's weighted sum before passing the result to the next layer.
A LoRA variant that adaptively allocates rank across layers.
A small trainable module inserted into a frozen model (the original PEFT idea); also a loose name for a LoRA's saved weights.
The wrapping applied around every sub-layer: `output = LayerNorm(x + SubLayer(x))`.
Capping the queue and per-user load; rejecting fast under overload instead of accepting unservable work.
In policy gradient RL, the difference between actual return and baseline: A = reward - V(prompt).
An AI system that works in a loop — read the situation, decide, act with tools, observe the result, repeat — instead of answering a single message.
The repeating cycle of **Think → Act → Observe** until the goal is reached.
Copilot autonomously plans and performs multi-step tasks (editing files, running commands).
A setup where the model (acting as an **agent** in a decide-act loop) chooses whether, what, and how many times to retrieve, rather than running a fixed one-shot pipeline.
Software that performs tasks once thought to require human intelligence, like understanding language or making decisions.
A system that uses an LLM as its "brain" to decide, step by step, what actions to take to accomplish a goal — using tools, memory, and reasoning.
Designing experiences for an uncertain, fallible, probabilistic tool; also a safety mechanism against overreliance.
A period of reduced funding and interest in AI research, typically following a wave of over-promising and under-delivering.
A competition mathematics exam (15 problems, 3 hours).
A BERT variant that reduces parameters by factorising the embedding matrix and sharing weights across Transformer layers.
Automated paging when SLOs are at risk; alert on user-felt symptoms (latency, errors), not just machine stats.
The process of training an AI system to behave in ways that align with human values, intentions, and safety constraints.
Making language models behave in accordance with human values and preferences.
A grid where each row corresponds to a target word and each column to a source word.
The small neural network inside the attention mechanism that scores how well a decoder state matches each encoder hidden state.
A communication pattern where every GPU sends data to every other GPU.
A list of approved tools/recipients/domains the agent is restricted to.
A sequence of mathematics competitions for students (AMC 8, 10, 12).
Fast search that finds *almost certainly* the closest vectors, trading a tiny bit of accuracy for huge speed.
Psychological harm experienced by humans who repeatedly review harmful content (violence, abuse, self-harm).
Other ANN indexes: trees (Annoy, static), quantization+pruning (ScaNN), SSD-resident graph (DiskANN, billion-scale cheaply).
A tempting-but-wrong approach; a common, costly mistake.
A permissive open-source license allowing commercial use without restriction.
A file or output Claude produces and saves during a session (a document, a diagram, a report) that you can reopen and reuse later, especially in Cowork.
A mechanism introduced the same year as seq2seq (Bahdanau et al., 2014)
The computational cost of attention, typically measured in FLOPs (floating point operations).
One of several parallel attention computations; more heads = richer attention, bigger KV cache.
The key innovation in Transformers that allows each token to consider the relevance of all other tokens in the sequence.
The probability-like number, between 0 and 1, representing how much the decoder at decoding step t focuses on source position i.
A key-value detail attached to a span (model name, token count, cost, etc.).
Proving *who* you are (a user or service).
Deciding *what* an authenticated identity is allowed to do.
The tendency of human reviewers to over-trust the AI and stop truly checking.
How much an agent decides and acts on its own versus asking a human.
The decoder's mode of operation: generate one token at a time, feed the generated token back as input, generate the next.
A model that generates a sequence by predicting one token at a time, conditioning each prediction on all previously generated tokens.
A training objective where the model learns to predict the next token given all previous tokens.
Adjusting replica count to match traffic; for LLMs, scale on queue length / KV-cache utilization / TTFT-vs-SLO, not CPU.
An additional loss term added to the main cross-entropy language modelling loss during MoE training.
A post-training quantization method for fast, accurate GPU inference.
A higher-level, config-driven fine-tuning framework.
The process of updating node statistics (visit counts, accumulated rewards) as you trace back from a leaf node to the root after a rollout.
A command or job Claude starts and lets run without waiting — useful for long builds, test suites, or watchers.
Short for "backward propagation of errors." The algorithm that computes the gradient of the loss function with respect to every weight in a multi-layer network, by applying the chain rule backwards from the output layer to the input layer.
The standard way of training RNNs and LSTMs.
A repo with only `.git` contents and no working files; what servers use.
A model that has only been through pretraining.
A reference point (e.g., the un-fine-tuned base model) you compare your model against on the same test set.
A group of examples processed together for one weight update; batch size = how many.
Running non-urgent requests together at a discount.
The number of training examples (or sequences) processed in a single gradient update.
Processing many requests together to use the GPU efficiently; the main cost lever in serving.
An inference-time decoding algorithm.
Standardised tests for evaluating language model quality (e.g., MMLU for general reasoning, GSM8k for math, HumanEval for coding).
vLLM's built-in load/benchmark scripts.
An embedding-based metric comparing meaning rather than exact words.
A strategy where you generate N independent solutions to the same problem and select the best one according to some criterion (e.g., a Process Reward Model score).
The DPO parameter controlling how strongly to stay near the reference (SFT) model.
An embedding model that encodes the query and each document *separately* then compares.
Bi-encoder embeds query and doc separately (fast, searchable); cross-encoder scores them together (slow, precise) — used for reranking.
A representation built from both left and right context simultaneously.
An LSTM that processes a sequence in both directions — forward and
Two recurrent networks — one reading left to right, one right to left — whose hidden states are concatenated at each position.
The task of deciding whether an input belongs to one of two categories — yes or no, 0 or 1, cat or dog.
1 bit per number; up to 32× smaller, Hamming-distance search, very fast.
A selective PEFT method that trains only the bias terms.
The library that handles 4-bit/8-bit quantization for QLoRA.
Word-overlap metrics for generation tasks; weak proxies, use cautiously.
Bilingual Evaluation Understudy.
Stores the raw contents of one file (no name, no path).
Computing attention in blocks (query chunk × KV chunk) rather than all-at-once.
A classic keyword-ranking algorithm; the "sparse" half of hybrid search.
The training dataset for GPT-1: approximately 7,000 unpublished novels scraped from the web, totalling ~800 million words.
A process where improvement in one component (the model) enables improvement in another (the data), which feeds back to improve the first component further.
The design flaw in plain seq2seq (Paper 06): the entire source sentence's meaning must be compressed into a single fixed-size vector before decoding can begin.
A subword tokenisation algorithm that splits words into common subunits.
A probabilistic ranking model from statistics, used here to model human preferences.
A movable pointer to a commit; physically a one-line file in `.git/refs/heads/`.
Enforced rules on a branch: require PRs, approvals, passing checks, signed commits; block force-pushes and deletion.
Compare the query to every vector.
Limits and automatic stops that prevent runaway spending from bugs, spikes, or abuse.
Self-hosting GPUs (cheaper at high steady volume) vs managed per-token APIs (cheaper/simpler at low or spiky volume).
A user granted a narrow exception to a ruleset.
Reusing prior results to avoid paying again.
Helping users trust the AI exactly as much as it deserves — avoiding both over-trust and under-trust.
A small set of representative texts used to set quantization scales well.
Releasing a change to a small percentage of traffic first, watching metrics, then expanding.
A vector of proposed updates to the cell state, produced by a tanh
The handshake where client and server announce which features they support, so each side only uses what the other understands.
A multiplier that sets the maximum number of tokens each expert can process per batch: `capacity = (batch_tokens / n_experts) × capacity_factor`.
Estimating how many replicas/GPUs a workload needs: peak demand ÷ single-replica capacity × safety factor.
How many distinct values a label can have.
When a model loses knowledge from pretraining while being fine-tuned on new data.
Self-attention where position i is prevented from attending to positions j > i (future tokens).
Same as autoregressive language modeling: predict the next token given previous tokens.
A (T × T) upper-triangular boolean mask applied in the decoder's self-attention.
In autoregressive language modelling, preventing the model from attending to future tokens (tokens that come after the current position).
One of the two Word2Vec training tasks.
The "notebook" of an LSTM — a vector that flows from one time step to
A rule in calculus for differentiating composed functions.
A prompting technique where intermediate reasoning steps are shown in few-shot examples, causing language models to generate their own step-by-step reasoning before producing a final answer.
Conversational Q&A with Copilot about your code.
The model-specific format (with special tokens) for laying out system/user/assistant turns.
A system that responds to a single message.
An improvement on the compute-optimal frontier (from DeepMind's Chinchilla paper, 2022).
A thought experiment proposed by philosopher John Searle in 1980 as a critique of the Turing Test.
A small passage of a document (often a paragraph) stored and retrieved as a unit.
Breaking long prefills into chunks interleaved with decode steps, so big prompts don't stall everyone's TPOT.
Splitting documents into chunks.
Automated testing and deployment on every change; for AI, evals run here to gate changes.
Automatically stops sending requests to a clearly-failing dependency for a while, using the fallback instead.
Linking the documents an answer is based on, so users can verify.
Anthropic's command-line coding agent: an AI that reads, writes, and runs code in your project, using tools, memory, and your instructions.
A plain-Markdown briefing file Claude reads automatically at the start of a session.
The component inside the host that maintains a connection to one MCP server and speaks the protocol on the host's behalf.
Keeping exactly N requests in flight vs sending requests at an arrival rate (queuing if busy).
A reading comprehension exercise invented in 1953 where words are systematically removed from a passage and the reader must fill them in.
Inline "ghost text" suggestions as you type.
Line-by-line feedback on a PR before merging.
Static analysis that finds vulnerability patterns in your code and flags them on PRs.
A deterministic rule check (valid JSON?
One vector per token + MaxSim matching; cross-encoder-like quality, still scalable, larger storage.
The total fine-tuning loss: L₃ = L_task + λ · L_language_model.
One saved snapshot of your project, with a unique hash, author, date, and message.
Points to a root tree, parent commit(s), author info, and message.
The amount of data that must be transferred between GPUs.
When a conversation gets long, Claude summarizes earlier context to free up room while keeping the important facts.
A mathematical property of problems: a problem is "computable" if it can be solved by a Turing Machine (i.e.
The total computational resources available for training, measured in FLOPs (floating point operations).
Limited by calculation speed (cores busy, data plentiful).
The simultaneous execution of computation and communication.
Achieving the best accuracy for a given computational budget.
The boundary of efficient training allocations: the curve of (N, D) pairs that minimize loss for a given compute budget C.
The choice of which inference-time strategy (Best-of-N vs.
A security flaw where a trusted component is tricked into misusing its authority for someone else — a key risk MCP authorization design must prevent.
The school of thought in AI that believes intelligence emerges from the interactions of many simple connected units (like neurons), rather than from explicit symbolic rules.
The subjective experience of being aware, of having an inner life.
The user's explicit approval before an app acts on their behalf (e.g.
The original paper's name for the additive structure of the cell state.
A written document specifying principles that an AI should follow.
An alignment methodology that replaces human feedback with AI feedback.
A portable bundle of app + dependencies that runs identically everywhere; the fix for GPU environment ("works on my machine") pain.
Objects are named by the hash of their content, so identical content is stored only once.
Everything Claude can "see" right now: your messages, files it has read, tool results, and instructions.
The craft of deciding what to put into Claude's limited context — and what to leave out — so it has exactly what it needs and isn't distracted by noise.
The maximum number of tokens a model can process in a single input.
The maximum amount of text (in tokens) a model can consider at once, e.g., "8k context" = 8,192 tokens.
Distributing a long sequence across multiple GPUs along the sequence dimension (distinct from data, tensor, or pipeline parallelism).
Passing trace context so each span knows its parent and the trace tree assembles correctly.
Cramming too much into a prompt in the hope something helps.
Also called the **thought vector**.
The maximum number of tokens the model can see at once.
The maximum number of tokens (prompt + output) a model can handle at once, e.g.
Extra next-token-prediction training on in-domain text to inject domain knowledge before SFT.
Refreshing the batch every token step, removing finished requests and adding new ones.
Automatically testing every push/PR to catch breakage early.
In machine learning, a model has converged when its weights have settled and further training produces no improvement.
Open chat (flexible but a blank-box burden) vs.
During training, the recurrence x_t = Āx_{t-1} + B̄u_t can be unrolled and rearranged as a convolution: output = conv(input, kernel).
A cloud agent you assign an issue; it works in the background and opens a PR.
A GitHub-hosted, repo-scoped long-term memory; Copilot saves, validates, refreshes, and expires facts (~28 days unused).
Letting requests share identical KV blocks (e.g.
An individual processing unit; GPUs have thousands.
A body of text used to train a model.
Patterns where the model critiques retrieved context and re-retrieves or falls back if it's insufficient.
A measure of similarity between two vectors based on the angle between
Breaking spend down by feature, customer, model, or endpoint to see what/who drives cost.
Making AI cost visible, accountable, and continuously optimized.
The dollar cost of one request, and per-user/per-conversation/per-outcome cost.
GPU $/hour ÷ tokens/hour at the operating point; the metric leadership cares about.
The trade-off: you can't maximize cheap, good, and fast all at once; you choose the balance per use case.
The collaborative surface for Claude (schedules, artifacts, plugins, shared sessions) that goes beyond a single terminal chat.
A processor with a few very capable cores; great at complex sequential tasks.
An agent that checks others' work for errors.
A prompt asking an AI model to evaluate its own or another model's output against a specific principle.
Attention where the query comes from one sequence (the decoder) and the keys and values come from another sequence (the encoder).
A model that reads query and document *together* to judge relevance.
The standard loss function for language modeling.
NVIDIA's platform for running general computation on their GPUs.
Recording a whole step's GPU operations and replaying them as one unit to cut launch overhead.
Persistent files Copilot folds into requests so you don't repeat rules.
The dimension of all input and output vectors in the Transformer.
The process of having humans provide labels (e.g., preference comparisons) for training data.
A short document recording a dataset's source, size, splits, cleaning, and limitations.
When test (or validation) examples also appear in training, making evaluation falsely optimistic.
Running multiple full copies (replicas) of a fitting model behind a load balancer to add throughput.
Connects agents to your data for retrieval (LlamaIndex…).
The number of training tokens (or training examples).
Library for loading and processing training data.
Agents discuss/challenge each other to reach a better answer.
The repeatable nine-step procedure for designing any serving deployment (brief → fit → precision → parallelism → optimizations → config → benchmark/size → operate → cost).
The second phase: generating output tokens one at a time, sequentially.
The second half of the seq2seq architecture.
A Transformer that uses only the decoder stack (masked self-attention + feed-forward), without the encoder-decoder cross-attention.
Breaking a big task into smaller tasks.
Breaking a complex problem into smaller, simpler subproblems and solving each one before combining results.
Removing exact and near-duplicate examples from a dataset.
Layering controls (prevent → detect → block → review → recover) so no single failure is catastrophic.
Clear markers (triple backticks, XML-style tags, headings) that separate parts of a prompt — instructions vs.
The standard approach in neural networks: every parameter fires for every input.
Embedding/vector search (vectors are "dense" with numbers).
Dense = embeddings (meaning); sparse = word-based vectors (exact terms).
GitHub feature that alerts on vulnerable dependencies and opens PRs to update them.
The map showing how every technique traces back to a few root facts (token-by-token generation; separate GPU compute/memory; memory-bound decode and the KV cache).
Converting quantized numbers back to higher precision for computation; adds small overhead.
Same input always gives the same output (normal software).
The ordered set of checks for diagnosing slow or broken serving, each mapping to an earlier module's concept.
How many numbers are in a vector (e.g., 384, 768, 1536).
The *user* types malicious instructions to bypass the AI's rules.
Running prefill and decode on separate GPU pools so the two phases stop interfering; a large-scale technique.
The process of converting a continuous-time differential equation (dx/dt = Ax + Bu) into a discrete recurrence (x_t = Āx_{t-1} + B̄u_t).
A compressed version of BERT created by knowledge distillation (training a small model to mimic the outputs of a larger one).
Using a strong "teacher" model to generate outputs to train a smaller "student" model.
Every clone holds the full history, so you can work offline and each copy is a backup.
Data's shape changing over time, staling IVF clusters / PQ codebooks; fix by retraining/re-indexing.
The linguistic claim (often attributed to Firth, 1957) that words
When the RL policy generates responses very different from the distribution the reward model was trained on.
The dimension of the Query and Key vectors.
A LoRA variant that decomposes weights into magnitude and direction; often a small quality gain.
Quantizing the quantization constants too, saving extra memory (from QLoRA).
A simpler alternative to RLHF that trains directly on chosen/rejected pairs, no reward model or RL needed.
Behavior decaying over time: *model drift* (the provider changed the model), *data drift* (inputs changed), *knowledge drift* (your knowledge base went stale).
The philosophical position, associated with René Descartes, that mind and matter are fundamentally different kinds of thing.
Halting training when validation loss stops improving, to prevent overfitting.
`per_device_batch_size × gradient_accumulation_steps` — the real batch size after accumulation.
Modified attention patterns (sliding window, global, sparse) that reduce computation from O(n²) to O(n log n) or O(n), making long sequences tractable.
A scalar λ such that Av = λv for some eigenvector v.
Multiplication of two vectors of equal length, slot by slot.
A server primitive that lets a server ask the user for additional input mid-task, through the client.
A chatbot created in 1966 by Joseph Weizenbaum at MIT.
A dense, low-dimensional vector representation of something — a word,
The length of each word vector.
The (V × d) matrix whose rows are the embeddings for each vocabulary
The model that turns text into embeddings.
Capabilities that appear when a language model reaches a certain scale, but were not present (or were very weak) in smaller versions.
A capability that appears in large language models above a certain size threshold, even though it was not explicitly trained for.
Flags turning on the corresponding techniques from Modules 04–05.
The first half of the seq2seq architecture.
The vector produced by the bidirectional encoder for source position i.
The Transformer's two-part structure for seq2seq tasks (e.g., translation).
A philosophy, popularised by this paper, where a single neural network
A specific, fine-grained permission a given identity holds (e.g.
A term in RL that encourages exploration by rewarding policy entropy (randomness).
Memories of events / general facts / how-to skills.
One complete pass through all training examples.
Sending only the cases that need judgment (low confidence, flagged, high-stakes) to humans, and to the right human.
A clear way for the user to reach a human or bypass the AI when it isn't working.
Measuring prompt quality systematically against a set of test cases with known good answers, so you can tell whether a change actually helped.
A collection of example tasks with expected outcomes, used to score the agent and catch regressions.
Measuring the quality of an agent's outputs, ideally with numbers across many examples.
Systematically measuring AI output quality across many cases.
Generating data by prompting a model to make existing examples harder and more varied.
Giving an agent more power/tools than it needs, so one mistake or injection causes outsized damage.
One of n specialised feed-forward networks in an MoE layer.
A training failure mode where the gating network routes most tokens to a small number of popular experts, leaving the rest undertrained and effectively unused.
Distributing a Mixture-of-Experts model's experts across GPUs.
The opposite failure mode: the gradient grows without bound when
The fundamental challenge in search and learning: should you exploit what you've learned (focus on high-reward nodes) or explore new options (try under-explored nodes)?
The power in a power law.
Extending a fitted line to predict values beyond the observed range.
Meta's industry-standard ANN *library* (not a server).
Whether every claim in an answer is supported by the retrieved context; the hallucination detector.
Whether an answer sticks to its provided sources rather than making things up.
A worse-but-working alternative when the primary path fails (another model, a cache, a non-AI path, or a human).
A 2016 extension of Word2Vec (by Bojanowski et al.) that represents
A switch to turn a change on/off instantly without redeploying.
A two-layer MLP applied position-wise after attention: `FFN(x) = max(0, xW₁ + b₁)W₂ + b₂`.
The per-token processing part of a layer; holds many parameters.
Including a few worked examples in the prompt to show the model the pattern you want, instead of only describing it.
The ability to perform a task from a small number of examples — typically tens to hundreds.
A technique where a language model is shown a small number of examples (typically 2-8) before being asked to solve a new problem.
The per-token processing sub-layer in a Transformer block; holds much of the model's parameters and knowledge.
Restricting search by metadata.
Adapting a pre-trained model to a specific task by continuing training on labelled task data with a small learning rate.
An exact, memory-efficient attention kernel that avoids writing big intermediates to slow memory; speeds attention, especially for long contexts.
Floating-point Operations Per Second; raw compute speed.
The improvement loop: production failures feed the eval set, which catches them before the next release, which improves the system.
A sigmoid-valued vector that decides which slots of the previous cell
Your own full copy of someone else's repo, for contributing without write access.
The computation that flows from input to output through the network: input → layer 1 → layer 2 → ...
A large model pre-trained on broad data that can be adapted to many downstream tasks.
32-, 16-bit floating-point precisions.
Number formats of 32/16/16/8/8/4 bits (= 4/2/2/1/1/0.5 bytes per parameter).
Modern 8-bit float, natively accelerated on the newest GPUs; memory savings + real compute speedup with small quality loss.
Software that handles the repetitive plumbing of building agents (loop, tool parsing, memory, coordination).
Updating all of a model's weights.
The model outputting a structured (usually JSON) request to use a tool, which the surrounding program then executes.
Periodic cleanup that packs loose objects and eventually removes unreferenced ones.
A simpler variant of the LSTM proposed by Cho et al.
A single chokepoint all model calls pass through, used to centralize caching, routing, rate limits, budgets, and logging.
The learned routing function `G(x) = Softmax(TopK(H(x), k))`.
Examples of data-protection and AI regulations imposing real obligations (consent, deletion, transparency, sometimes mandatory human oversight).
Gaussian Error Linear Unit: GELU(x) = x · Φ(x), where Φ is the Gaussian CDF.
The smallest variant (~2–7B parameters) of Gemini, optimized for on-device inference on mobile phones and edge devices.
The balanced variant (~50B parameters estimated) of Gemini, deployed for most production use (Google Bard, Workspace, Search).
The largest variant (~1.3T parameters estimated) of Gemini, achieving the highest benchmarks (90.04% MMLU) but requiring significant compute.
The ability of a model to perform well on new, unseen data (test set).
Whether a trained reward model (or policy) performs well on new, unseen tasks or domains.
A file format (llama.cpp/Ollama) for running quantized models efficiently on CPU/Mac.
Automation (CI/CD) defined as YAML in `.github/workflows/`.
The bundle of advanced security features (secret scanning, code scanning, etc.) for private/enterprise repos.
An AI pair programmer (an LLM wired into your tools) that suggests, explains, and writes code.
### Outcome Reward Model (ORM)
A 2014 alternative to Word2Vec, from Stanford.
General Language Understanding Evaluation.
A 1931 result by mathematician Kurt Gödel: in any formal mathematical system powerful enough to describe arithmetic, there are true statements that cannot be proved within that system.
Curated questions + correct answers + source chunks, used to measure retrieval and generation quality.
Throughput counting only requests that met their latency SLO.
The one-line summary of the serving engineer's objective.
A post-training quantization method for efficient GPU inference.
A processor with thousands of simpler cores; great at massively parallel work like the matrix math in LLMs.
A "GPU is busy" reading that can sit at 100% during memory-bound decode while compute is idle — so it's a misleading efficiency signal for LLMs.
Flag for the fraction of GPU memory vLLM may use; the key throughput/stability knob (bigger KV cache vs less headroom).
Designing the system to get slower/simpler/fall back under stress rather than failing outright.
Whatever decides how good an output is: code-based, human, or LLM-as-judge.
The vector of partial derivatives of the loss with respect to every
Summing gradients over several mini-batches before updating, to simulate a larger batch with less memory.
A memory optimisation technique where intermediate activations are discarded during forward pass and recomputed during backward pass.
The optimisation algorithm that trains neural networks.
Moving a task from tighter to looser human oversight as the AI proves itself (and back if quality drops).
Extracting entities and relationships into a **knowledge graph** and retrieving sub-networks; strong for multi-hop/global questions.
The simplest decoder strategy.
Tying the model's answer to provided source text rather than its own memory — the basis of trustworthy, citable output.
A variant of Multi-Head Attention where multiple query heads share the same key-value head.
A benchmark dataset of 8,500 grade-school math word problems, ranging from simple arithmetic to multi-step reasoning.
A check around the model.
Checks and limits (enforced in code) on what an agent can take in, put out, or do.
Input/output filtering to keep harmful or unsafe content in check.
The phenomenon where a language model generates plausible-sounding but factually incorrect or entirely made-up information.
One agent passing the task (and needed context) to another.
In older statistical machine translation, each target word was explicitly assigned to exactly one source word.
An algorithm designed with GPU memory hierarchy in mind.
A principle in many AI constitutions stating that the AI should avoid providing information, advice, or assistance that could lead to physical, financial, psychological, or emotional harm.
A 40-character fingerprint computed from content.
High-capacity GPU memory (e.g., 80GB on an H100), with lower bandwidth than SRAM.
One of h = 8 parallel attention computations in multi-head attention, each operating in a lower-dimensional subspace (dₖ = d_model / h).
The biological learning rule proposed by psychologist Donald Hebb in 1949: "neurons that fire together, wire together." When two neurons are active simultaneously, the connection between them strengthens.
The alignment criteria used to train InstructGPT: helpful (answers user queries well), harmless (doesn't enable or encourage harmful acts), honest (doesn't hallucinate or mislead).
A principle stating that the AI should genuinely assist the human in achieving their goals.
A layer of neurons between the input layer and the output layer.
How wide each token's internal representation is; bigger = more capacity, memory, compute.
The LSTM's "spoken" output at each step.
A high-level plan whose steps each break into their own sub-plans.
A popular, fast, accurate ANN indexing method (higher memory use).
A principle stating that the AI should be truthful and not deliberately mislead the human.
A command the harness runs automatically at a defined moment — before or after a tool call, when a session starts, and so on.
The AI application the user interacts with (e.g.
Labels provided by humans comparing two AI outputs and indicating which one is better.
Judgments by human raters about which model outputs are better.
Measure of how often different human raters agree on which output is better.
Requiring human approval before the agent performs risky/irreversible actions.
The AI acts autonomously but a human monitors and can intervene.
Fully autonomous AI; humans only review aggregates and samples afterward.
Combining **dense** (vector) and **sparse** (BM25 keyword) search, fused (e.g.
Generate a hypothetical answer, embed *it*, and search with that; the fake answer is often closer to real passages than the question.
A setting you choose before/around training (learning rate, epochs, rank, etc.), as opposed to the learned weights.
Carrying the original user's identity through each hop — host → server → downstream API — so permissions are enforced as that real user, not a shared service account.
Performing a task by providing examples in the prompt, without updating model weights.
The ability to retrieve specific facts from long context.
The offline phase: parse → clean → chunk → embed → store, producing a searchable knowledge base.
Malicious instructions hidden in *content the model processes* (a web page, document, email).
The process of running a trained model on new inputs to generate predictions.
The time to generate a single token during inference.
Inference: generating tokens one-by-one (autoregressive).
The broader principle of improving model performance by allocating more compute at inference time, rather than only at training time.
A high-speed network fabric used in data centres (200+ GB/s).
Fast inter-node networking technologies that make multi-node serving practical.
The first lifecycle message that opens an MCP session and triggers capability negotiation.
The text you send in.
A sigmoid-valued vector that decides how much of the candidate vector
A matrix or function that projects the input u_t into the state space.
Tokens you send vs.
GPT-1's technique for reformatting any NLP task's input as a flat token sequence wrapped in special tokens, allowing the unmodified pre-trained model to handle diverse task formats.
The risk that AI-suggested code contains vulnerabilities (injection, weak crypto, hard-coded secrets).
Trusting model output blindly and passing it into a database, shell, or page, enabling classic attacks.
A base model that has gone through post-training (SFT + preference tuning) so it follows instructions and chats.
Teaching language models to follow user instructions accurately and safely.
A data shape of instruction / optional input / output (e.g., Alpaca-style).
Personal → path-scoped → repo-wide → AGENTS.md → org; all merge, higher wins on conflict.
The ability of a language model to accurately follow user instructions and respond helpfully.
Adding the code that creates spans/logs/metrics.
8- and 4-bit integer precisions used in quantization.
Any GPU-to-GPU or node-to-node link; its speed dictates which parallelism you can use where (the governing rule: chatty parallelism → fastest link).
Variants of DPO with different objectives or data needs (KTO uses single good/bad labels, not pairs).
Keeping one tenant's (or session's) data and execution separated from another's, so they can't see or affect each other.
A GitHub thread tracking a task, bug, or feature.
One complete cycle of: MCTS search → solution verification → data collection → model training.
Clustering-based ANN; partition space into cells, search only cells near the query.
A crafted prompt that tries to bypass a model's safety rules or system instructions.
An LLM from AI21 Labs (2024) that alternates Mamba and Attention blocks in its architecture.
A setting that forces the model to return valid JSON, making its output safe to parse by a program.
The lightweight remote-procedure-call format (JSON-RPC 2.0) MCP uses for all messages — requests, responses, and notifications.
A file format with one JSON object per line; the common format for training data.
The number of experts selected per token.
A small program that runs one operation on the GPU.
Merging several small operations into one kernel to reduce memory traffic.
The "advertisement" projection of each position.
A regularization term in the RL objective that constrains the policy to stay close to the SFT baseline: β · KL[π_RL || π_SFT].
The task: find the k most similar stored vectors to a query.
Your prepared, searchable collection of chunks.
The date after which a model knows nothing, because its weights were frozen at training time.
LLM/ML-aware serving layers built on Kubernetes.
The standard orchestrator for running, scaling, and self-healing many containers across machines.
The fixed-size unit of KV memory, and the per-request map from logical token positions to physical blocks.
The memory buffer storing Key and Value vectors from all previous tokens during autoregressive (token-by-token) generation.
A segment of the key and value matrices corresponding to a subset of the sequence.
Moving cold KV cache to CPU/disk when GPU memory is tight; trades speed for capacity.
Storing the KV cache in 8-bit (or lower) to roughly halve its memory, enabling more concurrency / longer context.
Computing loss only on the assistant's tokens, so the model learns to respond, not to parrot the prompt.
A probability distribution over sequences of tokens.
A neural network trained to predict the next token in a sequence, using next-token prediction as the training objective.
The time taken to produce a response.
Making communication latency disappear by overlapping it with computation.
A plot of throughput vs p99 latency across concurrency levels; the most important serving chart.
Bigger batches raise throughput but can raise per-user latency.
One repeating unit of a Transformer (attention + MLP).
Applied after each sub-layer.
A small positive number (e.g.
How the learning rate changes over training (e.g., cosine decay).
Giving the agent only the tools/permissions it actually needs.
Giving the model and its tools the minimum power needed, to limit the blast radius of any failure.
The defined stages of an MCP connection: initialize → operate → shut down.
A recurrence relation of the form x_t = Āx_{t-1} + B̄u_t where future x_t depends only on previous x_{t-1}, not on all past history.
A statistical method to fit a straight line through data points.
A property of a dataset: two classes are linearly separable if you can draw a straight line (in 2D) or a flat hyperplane (in higher dimensions) that perfectly separates all examples of one class from all examples of the other.
Health checks: "is it alive?" (restart if not) vs "is it ready to serve?" (don't route traffic until the model is loaded — crucial for slow-loading LLMs).
A lightweight engine for running quantized models on CPU/GPU/Mac.
An AI model (GPT, Claude, Gemini, Llama…) that predicts the next chunk of text.
Using a strong model to score or compare outputs, for filtering data or evaluating models.
Using another LLM to score an output against criteria (also used in evaluation).
The final layer that converts the model's internal representation into a probability for every token in the vocabulary.
The goal of ensuring that all n experts receive roughly equal numbers of tokens over training.
Tools to drive realistic concurrent/rate-based traffic.
Politely rejecting or queuing low-priority traffic under extreme load to protect the whole system.
A timestamped record of a single event ("what exactly happened in this one case").
A graph where both axes are logarithmic.
A raw, unnormalised score before softmax.
Raw, unnormalised scores output by a neural network before applying softmax.
The full name of the paper.
External storage (database/files) that persists across sessions and can be huge.
A single compressed object file in `.git/objects/ab/cdef...`.
An efficient fine-tuning method that adjusts a small add-on set of weights instead of all of them.
LoRA = small add-on weight patches that customize a base model.
The LoRA scaling factor; effective scaling is `alpha / r`.
Dropout applied within the LoRA path to reduce overfitting.
A single number measuring how wrong the model's predictions are.
A mathematical function that measures how wrong the network's prediction is.
Models attend most to the start and end of context and may overlook material in the middle; place key chunks at the edges.
Hashing that sends similar vectors to the same bucket.
The RNN variant used by both the encoder and decoder in this paper.
BERT's primary pre-training objective.
A dataset of 12,500 competition-level math problems from AMC (American Mathematics Competitions) and AIME (American Invitational Mathematics Examination).
The core math operation in LLMs; millions of independent multiply-adds, ideal for parallel GPU hardware.
Embeddings whose first N dimensions are themselves usable; truncate for speed, expand for accuracy.
A scale (Level 0 "Vibes" to Level 4 "Optimized") for assessing how production-ready an AI system is across all pillars.
The token cap per training example; longer ones get truncated.
A cap on how long the model's output can be.
Flag bounding max context length, which bounds KV-cache size per request.
Flag for max tokens processed per step; balances prefill vs decode and tail latency.
Flag for max concurrently batched requests; trades throughput against latency and memory.
A standard for packaging tools/data sources so any compatible agent can plug into them — like "USB for AI tools."
A central proxy that sits between hosts and many MCP servers to enforce auth, routing, rate limits, logging, and policy in one place.
A program that speaks MCP and offers tools, resources, or prompts (e.g.
A connector giving Copilot capabilities to act on external systems; complements skills.
Facts Claude persists across sessions in files, so it remembers your preferences, project details, and past decisions without you re-explaining them.
How fast data moves between GPU memory and cores (GB/s or TB/s); largely decides *decode speed*.
How much data (GB) the GPU can hold; decides *whether a model fits*.
Tiers from tiny-fast (registers, SRAM/L1) to big-slow (VRAM), then off-GPU (system RAM, disk).
With P GPUs using Ring Attention, per-GPU memory is O((n/P) × d), scaling linearly with the number of GPUs.
Limited by data-movement speed (cores idle, waiting for data).
Folding a LoRA adapter into the base weights so there's zero inference overhead.
Tags stored with each chunk (source, date, section, page, permissions) enabling filtering, citations, and access control.
A number measured and aggregated over time ("how is the system doing overall").
Multi-Head / Multi-Query / Grouped-Query Attention: design choices trading KV-cache size against quality.
A small chunk of work kept flowing through a pipeline to fill bubbles.
A layer type where multiple expert networks are available, and a router learns which expert(s) to use for each input.
The practices and tooling for deploying, monitoring, and continuously improving ML/LLM systems in production.
A benchmark of 57 diverse academic subjects (history, law, science, medicine) with 14,042 multiple-choice questions.
When generated data lacks diversity, repeating similar outputs.
The underlying Claude that powers a session.
Degradation that can occur when models are trained repeatedly on model-generated data.
Combining multiple fully fine-tuned models' weights into one without retraining.
Who supplies the LLM (OpenAI, Anthropic, Google, Hugging Face, local via Ollama).
Sending each request to the cheapest model that can handle it; escalating to a bigger model only when needed.
The observation that CoT prompting's effectiveness depends critically on model size.
The number of parameters in a neural network.
A skill chosen automatically by description match vs.
A drop-in replacement for the FFN sub-layer in a Transformer.
Tracking whether known things are wrong (vs.
A search algorithm that explores a decision tree by: (1) selecting promising nodes using UCB, (2) expanding the tree with new candidate moves, (3) running rollouts to simulate outcomes, (4) backing up the results to update node statistics.
Retrieval-quality metrics: MRR (rank of first hit), nDCG (graded ranking quality), precision@k (fraction of shown results that are relevant).
Leaderboards for embedding quality, retrieval quality, and index speed/recall.
Multiple specialized agents collaborating (or competing) toward a goal.
`Concat(head₁, ..., headₕ) · W^O`.
Generate several paraphrases of a question, retrieve for each, and combine.
An attention variant where all query heads share a single key-value head.
Chaining several thinking steps to solve harder problems.
Serving many independent customers (tenants) from one shared system, while keeping each tenant's data and access strictly separate.
Ensuring shared infrastructure doesn't leak one tenant's data to another (watch shared prefix caches).
Capable of processing and reasoning over multiple modalities (text, images, audio, video) simultaneously.
Retrieving over images, charts, tables, and scanned pages, not just text.
Total number of expert networks in one MoE layer.
Training a single model jointly on multiple modalities from the start, rather than training text-first and bolting on vision later.
Human language as it is actually spoken and written — English, Hindi, Tamil, etc.
The training trick that made Word2Vec fast.
The umbrella term for translation systems built entirely from neural
BERT's second pre-training objective.
The pre-training objective: given all previous tokens, predict the probability distribution over the next token.
The 4-bit format used by QLoRA, designed to match how weights are distributed.
One physical server (often holding multiple GPUs).
The specific gating formulation from the 2017 paper: raw logits have Gaussian noise added before top-k selection during training.
Information given to the model at question time via the prompt; fresh, exact, citable.
Scaling a vector to length 1.
A JSON-RPC message that expects no reply — used for one-way signals like progress updates.
A generation strategy where you only consider the top tokens that make up a certain cumulative probability (e.g., top_p=0.9 means consider tokens until their probabilities sum to 90%).
Ensuring computed values don't overflow, underflow, or lose precision.
Lets containers access the host's GPUs.
NVIDIA's high-speed GPU interconnect (576 GB/s per link).
NVIDIA's very fast GPU-to-GPU links *inside a server*; what makes tensor parallelism viable.
The modern authorization framework MCP builds on for granting scoped, delegated access without sharing passwords.
Being able to see everything an agent did (thoughts, tool calls, results).
Software that extracts text out of an image (e.g.
Off-policy: Learning from data generated by other policies (e.g., supervised data).
Running your system against a fixed test set before shipping a change.
The Python library (`LLM.generate`) for batch jobs vs the API server (`vllm serve`) for live traffic.
A simple tool for running GGUF models locally; great for prototyping.
A vector of length V with a single 1 and the rest zeros.
Performing a task with exactly one example in the prompt.
Measuring quality on live production traffic via feedback and sampled LLM-judge scoring.
An incremental softmax computation (using logsumexp trick) that maintains running statistics (max, sum of exponentials) as you process blocks.
Online = live users, latency-sensitive.
A model whose weights you can download and run yourself, rather than calling a provider's API.
A de-facto standard API shape most engines (including vLLM) support, so clients can switch backends easily.
An open, vendor-neutral standard for creating traces, spans, and metrics, so you aren't locked to one tool.
PQ with a learned rotation first, for better recall at the same memory.
For a given compute budget C, the best way to split resources between model size (N) and data size (D) to minimize loss.
The algorithm that applies gradients to weights, with adaptive step sizes and momentum.
The code that assembles the prompt and manages the flow of a request (retrieval, history, model call, output handling).
Builds and coordinates the agent loop / multiple agents (LangGraph, CrewAI, AutoGen…).
The "boss" agent that breaks a goal into tasks and assigns them to workers.
A manager delegates sub-tasks to workers and combines results.
A method combining SFT and preference tuning into one step, no reference model needed.
A model that scores only the final output (right or wrong), without evaluating intermediate steps.
A sigmoid-valued vector that decides which slots of the cell state
Code that reads the model's response and extracts the structured data you need, often paired with a schema.
A matrix or function that projects the hidden state x_t back to the output space.
Training error decreases, but test error increases.
Users trusting AI output (including hallucinations) too much and acting on it.
A standard list of the top security risks for LLM applications; a useful threat checklist.
Median and tail response times; optimize and alert on the tail (p99).
Many objects bundled and compressed together by `git gc` into `.git/objects/pack/`.
Concatenating multiple short examples into one sequence to reduce wasted padding and speed training.
Filler tokens added so examples in a batch are the same length; ignored in the loss.
An optimizer that offloads state to CPU RAM during memory spikes to avoid crashes (from QLoRA).
vLLM's memory-management technique enabling efficient batched serving.
A hardware-friendly algorithm (e.g., Blelloch scan) that computes a recurrence y_t = f(x_{t-1}, u_t) in parallel by decomposing it into a tree of subproblems.
One of the billions of tunable numbers inside a model that together encode what it "knows." More parameters = more memory and compute needed.
The learnable weights in a neural network model.
The numbers that make up a model.
Knowledge stored in the model's weights from training; fast and broad but frozen, blurry, and uncitable.
The commit that came before a given commit.
Match on small chunks but return the larger surrounding passage for generation.
Pulling clean text out of source files (PDF, Word, HTML, slides, scans).
A metric that evaluates whether at least one out of K generated solutions is correct.
A phishing-resistant, passwordless sign-in credential.
A small rectangular region of an image, typically 14×14 pixels.
A slower general-purpose bus; TP over PCIe-only is often too slow.
An extension to LSTMs (Gers & Schmidhuber, 2000) in which the gates
The family of methods that train only a tiny number of new parameters while freezing the base model.
A way to summarize a distribution.
The value below which 50% / 95% / 99% of requests fall.
The artificial neuron Rosenblatt described: takes several inputs, multiplies each by a weight, sums the results, and outputs 1 if the sum exceeds a threshold, 0 otherwise.
How much Claude is allowed to do without asking — from confirming every action to running freely within an allowlist.
A metric for language models derived from cross-entropy loss.
A core data structure in pre-2014 statistical translation.
Personal data (names, emails, etc.) that must be protected, redacted, and handled per privacy law.
Agents work in a fixed order, each passing output to the next (assembly line).
Idle gaps in pipeline parallelism; reduced by keeping many micro-batches in flight.
Splitting the model by layers across GPUs/nodes, assembly-line style; less communication, so it can cross nodes.
A LoRA variant with smarter initialization from the base weights' principal components.
A mode where Claude researches and proposes a plan for approval before making any changes, so you can steer before code is written.
Writing the whole plan up front vs.
Decomposing a goal into an ordered set of doable sub-tasks.
A packaged add-on that extends Claude with extra skills, commands, or integrations.
K8s units: the running container(s), and the controllers that keep N replicas alive (operationalizing data parallelism).
RL algorithms that improve a policy (probability distribution) by taking gradient steps that increase expected reward.
The language model being trained and improved across rounds.
The property of a word having multiple meanings.
Tendencies of LLM judges to favor a certain answer position or longer answers; must be controlled for.
A fixed vector added to each input embedding to inject position information.
The fine-tuning steps (SFT, preference tuning) applied after pretraining to turn a base model into a helpful assistant.
A mathematical relationship where one variable is proportional to another raised to a power: y = a * x^b.
A stable reinforcement learning algorithm used in the RL stage.
Compress vectors by splitting into chunks and replacing each with a codebook ID.
An already-quantized model you download and serve directly.
Training a model on large-scale, typically unlabelled data before fine-tuning.
How many bits are used to store each weight; more bits = more exact but more memory.
How many bits store each number.
Of the K retrieved chunks, how many were actually relevant.
Data shaped as prompt / chosen / rejected, used for preference tuning.
Teaching a model judgment/taste by training on "this response is better than that one." (M12)
The first phase: the model reads the whole prompt at once (in parallel), builds the KV cache, and produces the first token.
Reusing the cached KV of shared prefixes (system prompts, RAG context, chat history) to skip redundant prefill.
Additive PEFT methods that prepend trainable "virtual tokens" (soft prompts) to the input.
Word vectors trained on a large corpus by someone else and then
The initial, expensive phase where a model learns language by predicting the next token across enormous amounts of text.
A core building block MCP defines.
Same input can give different outputs (AI).
A machine-learning model trained to evaluate the quality of individual steps in a multi-step reasoning process.
The live environment real users use, as opposed to a demo or test setup.
Solving problems by writing Python code instead of natural language reasoning.
Protocol features that let a long-running operation report how far along it is, and let the caller stop it.
Staged loading of skills: L1 name+description (always) → L2 full body (on match) → L3 resources (on demand).
On-demand expertise vs.
Tools to collect (scrape) and visualize metrics.
The text instructions given to a model.
Reusing/caching a stable part of the prompt to save cost and time.
Breaking a task into a sequence of prompts where each step's output feeds the next, instead of asking for everything at once.
The practice of carefully designing the text prompt to get better outputs from a language model.
The specific structure and wording of a prompt.
Malicious instructions hidden in content the agent reads (web page, email, doc) that trick it into harmful actions.
A reusable prompt with placeholders you fill in at runtime, so the same well-tested wording serves many inputs.
The company running the model you call over the internet (e.g., OpenAI, Anthropic, Google).
A GitHub proposal to review and merge a branch's commits.
Blocks a push that contains a recognized secret *before* it enters history.
The process of executing Python code to check if a solution is correct.
The underlying deep-learning framework that runs the math on the GPU.
The four weight matrices in attention; the most common place to attach LoRA adapters.
LoRA on top of a 4-bit quantized base model, enabling fine-tuning of large models on a single GPU.
Converting weights to lower precision to save memory, with little quality loss if done well.
Named trade-offs of size vs quality for GGUF models.
The "what am I looking for?" projection of each position.
Per-token vectors: Query ("what I seek"), Key ("what I offer"), Value ("info I provide").
The subset of query vectors on a given GPU.
In multi-head attention, one of n_heads independent attention mechanisms.
Retrieve relevant info → Augment the prompt with it → Generate the answer.
A popular framework of RAG evaluation metrics.
The size of a LoRA patch's bottleneck; controls its capacity.
Building a tree of summaries over a corpus and retrieving at the right level of detail; good for global/summary questions.
Capping request volume — the provider's limits on you, and your limits on each user (protects availability, cost, and against abuse).
Granting access by assigning users to roles (admin, editor, viewer) that bundle permissions.
Adjusting the plan when a step fails or returns surprising results.
The core pattern of alternating **Thought → Action → Observation**, combining reasoning with tool use.
The cognitive process of chaining ideas together across multiple steps to arrive at a conclusion.
Granting access based on relationships between entities ("owner of," "member of") — the model behind systems like Google Zanzibar.
The fraction of truly-relevant items that were actually found.
Was a relevant chunk in the top K retrieved; the most important retrieval metric.
In deep networks, the range of input positions that influence a given output position.
During token-by-token generation, apply the recurrence directly: x_t = Āx_{t-1} + B̄u_t.
A neural network that processes inputs one step at a time, feeding its
Deliberately attacking your own system to find weaknesses before real attackers do.
An agent reviewing its own output, finding flaws, and revising — without a human pointing them out.
Writing a "lesson learned" after a failure and storing it to do better next time (reflection + memory).
A catalog where MCP servers are published and discovered, so hosts can find and install them.
Something that used to work but broke after a change.
Checking the fine-tuned model didn't get worse at general tasks it should still handle.
A three-stage training pipeline for aligning language models: (1) Supervised Fine-Tuning on human demonstrations, (2) training a Reward Model on human preference comparisons, (3) using Reinforcement Learning (PPO) to optimize the policy aga
A data generation strategy: generate many candidate solutions, keep only the correct ones, discard the rest.
Whether the system keeps working under real load and failures.
Another full copy of a repo, usually on GitHub; the default is `origin`.
One complete serving instance (a single GPU or a TP/PP group); the unit you data-parallelize and autoscale.
Copies of data across machines (fault tolerance + read throughput).
A project tracked by Git: your files plus a hidden `.git` folder holding the entire history.
The broader idea — Word2Vec is an early instance — that useful
The paired JSON-RPC messages: a request asks for something and carries an id; the matching response returns the result or an error.
A second, more accurate (cross-encoder) scoring pass over retrieval candidates; the highest-value upgrade beyond basic retrieval.
Adding the sub-layer's input directly to its output: `x + SubLayer(x)`.
A server primitive exposing readable data (files, records, documents) the model can pull into context — read-only, addressed by URI.
Finding the most relevant chunks for a query.
Re-attempting a failed call, waiting progressively longer (backoff) with randomness (jitter), only for retryable errors, with a cap.
Cascading overload caused by naive client retries; mitigated by backoff and circuit breakers.
Sutskever's empirical hack: feed the source sentence to the encoder in
A point where the flow pauses for human judgment, placed by stakes and uncertainty.
A prompt asking an AI to rewrite its own output to address a critique.
In MCTS, the function that assigns a reward to a rollout outcome.
When a model games the reward signal to score high without genuinely being better.
When the RL policy finds ways to get high reward scores without actually being helpful.
A neural network trained in the second stage of RLHF to predict which of two responses humans prefer.
A distributed attention algorithm where P GPUs are arranged in a ring topology.
An arrangement of P GPUs in a logical circle where GPU i communicates with GPU i-1 (receives data) and GPU i+1 (sends data).
The second stage of Constitutional AI.
The stage of Constitutional AI where an AI (rather than a human) provides feedback on which response better follows the constitution.
The original preference-tuning approach: train a reward model, then optimize the LLM against it with RL.
Robustly Optimized BERT Pretraining Approach.
Telling the model who to act as ("You are a careful editor…") to shape tone, depth, and focus.
Quickly reverting to the previous version when a change goes wrong.
Update strategies: replace replicas gradually / run new alongside old then switch / send a small traffic slice to the new version first.
In MCTS, a simulation of completing a partial solution to a full solution.
A client primitive that tells a server which directories or scopes it is allowed to operate within.
A technique to extend a model's usable context length by adjusting its positional encoding.
A method of encoding token position information by rotating query and key vectors.
Classifying a query and sending it to the right index/tool (docs vs SQL vs web).
A simple, robust method to merge two ranked result lists (e.g.
A LoRA variant that rescales so high ranks train more stably.
Extra capacity (e.g.
A client primitive that lets a server ask the host's model to generate text — so servers can use the LLM without holding their own API key.
A hyperparameter in language model decoding that controls randomness.
An isolated, safe environment for running untrusted code so it can't harm the real system.
`Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V`.
Increasing the size of neural networks (more parameters, more data, more compute).
The observation that larger models have qualitatively different capabilities (reasoning, instruction-following) that smaller models lack.
vLLM's three conceptual roles: decides what runs (continuous batching), manages KV memory (PagedAttention), and runs the model on the GPU.
A precise specification of the fields and types an output must contain — handed to the model so its structured output is predictable.
In OAuth, the specific permissions an access token grants (e.g.
Any credential that grants access (password, API key, token, private key, `.env`); never commit it.
Revoking a leaked credential and issuing a new one; always do this *before* scrubbing history, since a pushed secret may already be copied.
GitHub feature that detects committed secrets and alerts you.
One of three embeddings summed to form each token's input representation.
An SSM where the input projection (B), output projection (C), and step size (Δ) are functions of the input u_t, not fixed constants.
Attention where Q, K, and V all come from the same sequence.
A technique that improves chain-of-thought reasoning by sampling multiple independent reasoning chains from the same prompt and taking a majority vote on the final answer.
The process of an AI model reading its own output and identifying whether it violates constitutional principles.
A bootstrapping process where: (1) a model generates candidate solutions using search, (2) solutions are verified automatically, (3) correct, high-quality solutions become training data, (4) the model is trained on this data, improving for
Generating new instructions (and answers) from a small seed set to bootstrap a dataset.
The loop of generate → self-feedback → refine → repeat.
Learning without human labels by using the data itself as the answer (e.g., predicting the next word).
Caching answers to frequently-asked (by meaning) questions to skip the pipeline.
Grouping sentences by meaning so each chunk is one coherent idea.
Searching by meaning (via embeddings) rather than exact keywords.
A subword tokenizer that converts text into tokens using a learned vocabulary.
Vectors in cheap object storage, stateless compute searches them; elastic and cost-efficient.
The encoder-decoder architecture from Paper 06 (Sutskever et al., 2014).
Parallelising the sequence dimension of tensors.
A strategy where you iteratively refine a solution, using feedback from one attempt to improve the next.
A program that exposes tools, resources, and prompts over MCP for clients to use (e.g.
Running a model as a continuous service that handles many users' requests over an API.
One continuous working conversation with Claude, with its own context.
The configuration file that controls the harness — permissions, environment variables, hooks, and model choice.
Fine-tuning on input→desired-output examples; the workhorse technique.
Running a new version alongside production without showing its output to users, just logging/evaluating what it would have done.
Splitting data across machines (capacity + write throughput; queries fan out).
The context window — the current task and recent steps; temporary.
The activation function σ(z) = 1/(1+e⁻ᶻ).
A function that squashes any real number into the interval (0, 1).
A commit cryptographically signed (GPG/SSH) so GitHub can show a **Verified** badge proving authorship.
How closeness is measured: **cosine** (angle/direction, default for text), **dot product** (direction + magnitude), **Euclidean/L2** (straight-line distance).
The throughput one replica sustains within SLO at your length profile; the basis for sizing.
A reusable procedure you teach Claude once (as a folder with instructions) so it can perform a multi-step task the same way every time — invoked by name.
The other Word2Vec training task, and the one people usually mean
The first stage of Constitutional AI.
A contractual performance promise, usually looser than the SLO.
A reusable shortcut typed as `/name` that runs a saved prompt or procedure — your own custom commands plus built-in ones.
An attention variant where each token attends only to the last W tokens (a sliding window), not all previous tokens.
A committed target like "99.9% success" or "p95 < 3s." Define one per feature.
On a log-log plot, the slope of a line is the exponent of the power law.
INT8 methods that handle activation outliers so 8-bit quantization stays accurate.
Git's mental model for what a commit stores: a complete photo of the project at a moment, not a list of edits.
Attention's approach: each target word is generated using a *weighted blend* of multiple source words, not a hard assignment to one.
Trainable input vectors that act like a learned prompt expressed as numbers.
A function that turns a vector of raw scores into a probability
Smooth approximation of ReLU: softplus(x) = log(1 + e^x).
A single timed unit of work within a trace (e.g., "retrieval took 120 ms"), with attributes.
The opposite of dense: only a fraction of parameters are active for any given input.
Keyword search (e.g.
The problem where an AI system finds a way to satisfy the letter of a specification while violating its spirit.
Using a small **draft model** (or heuristic) to guess several tokens, then verifying them in one parallel pass of the big model.
Store numbers as int8 instead of float32.
Stanford Question Answering Dataset.
GitHub's three ways to merge a PR (combine into one commit / merge commit / linear replay).
Tiny, ultra-fast on-GPU cache (e.g., 192KB per core).
Credentials for authenticating to GitHub (key pair / personal access token).
The "loading dock" where you gather exactly what goes into the next commit; physically the `.git/index` file.
A continuous or discrete linear dynamical system.
An n×n matrix governing how the hidden state x evolves over time.
Naive batching that waits for all requests in a batch to finish; wastes GPU slots.
The dominant pre-2014 translation approach.
A scalar (or per-head scalar) that controls the discretisation rate.
A string that tells the model to stop generating when it appears — useful for clean, bounded output.
When one GPU is slower than others (older hardware, thermal throttling, interference), it becomes the bottleneck.
Showing the answer as it's generated, word by word, to slash perceived latency.
Output constrained to a defined shape (JSON, a schema) so a program can use it directly, rather than free-form prose.
A prior SSM architecture (Gu et al., 2021) that imposes structure on the A matrix (e.g., diagonal, plus rank-1 update) for efficiency.
A separate Claude instance the main agent spawns to handle a focused sub-task (research, review) with its own fresh context, reporting results back.
Another Git repo embedded at a pinned commit inside yours.
A Word2Vec training trick where very common words (like "the", "of",
Key metrics — % of tasks done correctly / how long it takes / tokens or dollars used.
Replacing long history with a short summary to save context space.
The first stage of RLHF.
The full set of dependencies and sources a server is built and distributed from — a security surface, since a compromised dependency compromises the server.
Protecting against risks from third-party code and actions (e.g.
Google's 2021 simplification of MoE: k=1 routing (route each token to exactly one expert, no blending).
When a model agrees with users even when the user is wrong, in order to be pleasing.
A point where all P GPUs pause and wait for the slowest GPU to finish.
Training examples generated by an AI model rather than collected from humans.
The developer-written setup that defines the agent's role, rules, tools, and process.
The standing orders in a prompt that set the AI's behavior, rules, and tone.
An annotated tag pointing to a commit, with its own metadata.
The slow end (p95/p99) of the latency distribution; where real user frustration lives.
A function that squashes any real number into the interval (−1, +1).
Which weight matrices receive LoRA adapters (e.g., q/k/v/o projections, MLP layers, or "all-linear").
A long-running unit of work tracked over time, letting a server perform work asynchronously and report results when ready.
A training technique.
A publication style (unlike peer-reviewed research papers) that allows companies to present results without the formal review process.
A hyperparameter controlling randomness in generation.
Special GPU units that do low-precision matrix math (FP16/INT8/FP8) very fast; a reason quantization speeds compute.
Splitting each layer's math across GPUs; makes a big model fit and adds bandwidth, but needs constant fast communication — keep within one NVLink node.
Held-out, ideally real, examples used once at the end to score the model honestly.
A curated collection of test inputs (and often expected outputs) you run your system against.
Spending additional computation at inference time (rather than training time) to improve performance.
Having the model write a database query; the right tool for structured/aggregate questions (not RAG).
Hugging Face's production model-serving engine.
Alternative engines: Hugging Face's TGI, NVIDIA's highly optimized TensorRT-LLM, SGLang (structured generation/prefix sharing), and the CPU/local llama.cpp + Ollama world.
The point of maximum throughput still within your latency SLO; your target operating point (≈ goodput maximum).
Another name for the **context vector** — Hinton's evocative label for
A structured view of who might attack a system, how, and what's at stake — used to design defenses deliberately.
The minimum weighted sum required for the Perceptron to output 1.
The number of queries a system can handle per unit time.
How long until the *first* word of a streamed response appears.
The two components of latency: how fast output starts, and how fast it streams.
A maximum wait time for a call; exceeding it triggers a fallback.
A unit of text, roughly a word or subword.
The total number of tokens (words or subwords) available for generating a solution.
When an expert receives more tokens than its capacity allows, excess tokens skip the MoE layer and pass through the residual connection unchanged.
Practices that reduce how many tokens a task consumes — trimming context, reusing caches, being concise — to save cost and stay within the window.
The index of a token in the sequence (0 to n-1).
The process of converting input (text, images, audio) into discrete tokens.
The component that converts text into tokens (and back).
Marking a vector deleted without removing it from the graph; cleaned up later by compaction.
Code the agent can call to act or fetch info it can't do alone (search, calculator, send email, run code).
Giving the model the ability to call functions (look up an order, do math, search) and use the results.
A structured description of a tool (name, description, inputs) that tells the model when and how to use it.
The operation that keeps the k largest values in a vector and sets all others to −∞.
An alternative to temperature: the model samples only from the most probable tokens whose combined probability reaches p.
Average gap between subsequent output tokens.
The full recorded log of one agent run (inputs, steps, tools, observations, output, cost).
The identifier that ties all the steps of one request together.
A local branch linked to a remote branch for easy push/pull.
Dividing data into what the model learns from, what you watch for overfitting, and what you score once at the end.
The process of setting a model's weights by showing it data and nudging the weights to predict better.
The text corpus used to train a language model.
Computation used to train the model initially.
The path/process an agent took (not just the final answer).
The reuse of knowledge (model weights) learned on one task/dataset for a different but related task.
A neural network architecture based on self-attention, introduced in "Attention Is All You Need" (2017).
The architecture used in GPT models: a stack of self-attention and feedforward layers that process tokens left-to-right (causally).
The standard library for loading models and tokenizers.
A key benefit of Constitutional AI: the principles are written in human-readable natural language, making the intended values explicit and auditable.
How MCP messages physically travel between client and server — commonly stdio (local process) or streamable HTTP (remote).
A directory listing: names → blob/tree hashes plus permissions.
Exploring several possible reasoning paths/plans and picking the best.
Library providing ready-made trainers (`SFTTrainer`, `DPOTrainer`).
Cutting off text that exceeds the max sequence length; can silently break examples.
The line where data or control passes between parties with different trust levels — every crossing needs validation and authorization.
Time from sending a request to the first output token.
An abstract mathematical machine Turing described in 1936 — not a real physical device, but a thought experiment.
The test proposed by Turing: a machine passes if a human interrogator, communicating only by typed text, cannot reliably distinguish it from a human.
A second sign-in factor (authenticator app, passkey) beyond a password; the top account protection.
When the model hasn't learned enough; both train and validation loss stay high.
When a language model generates intermediate reasoning steps that sound logical and plausible but don't actually reflect how the model arrived at its answer.
A library that makes LoRA/QLoRA fine-tuning faster and more memory-efficient.
A formula that balances exploitation (choosing nodes with high average reward) and exploration (trying under-explored nodes).
The actual request from the user in a turn, as opposed to the system prompt that frames the whole session.
vLLM's re-architected core for lower overhead and cleaner async behavior.
The "what I send when selected" projection of each position.
In RL, an estimate of expected future reward used to reduce gradient variance.
The process of encoding organizational or societal values into an AI system.
The problem where gradients shrink toward zero as they propagate backwards through many layers (especially through sigmoid activations).
The phenomenon where, during BPTT on a long sequence, the gradient
In the context of scaling laws, the spread of loss values across multiple runs or models.
A database that stores embeddings and quickly finds the most similar ones (e.g., Pinecone, Chroma, Weaviate, FAISS, Qdrant).
Library (FAISS): in-process index only.
A component that evaluates whether a proposed solution is correct.
Tracking changes to prompts, models, retrieval config, guardrails, and code so behavior is reproducible and rollback is possible.
A Transformer applied to images by dividing them into patches and treating patches as tokens.
A high-performance GPU serving engine with continuous batching and LoRA support; the default for serious serving.
The set of words the model knows about.
A cluster region and its center, used by IVF.
The GPU's dedicated memory, separate from system RAM, measured in GB.
Pre-loaded spare capacity kept ready because LLM scale-up is slow.
Early requests that pay startup costs; discard them and measure steady state.
Starting with a tiny learning rate and ramping up over the first steps, for stability.
A number attached to an input connection in a neural network.
A regularization technique that discourages large weights, helping prevent overfitting.
The values given to weights before training begins.
Using the same weight matrix for both the token input embedding and the output projection (UW and UWᵀ).
Compressing just the stored weights (big memory/bandwidth win, easy on quality) vs also quantizing the live activations for compute speedups (harder, due to outliers).
The billions of numbers inside a neural network; where a model's learned knowledge and abilities are stored.
The fraction of comparisons your model wins in A/B evaluation.
How many words on each side of the target count as "context" in
An evaluation task of the form "A is to B as C is to ___".
Another name for a word embedding — a dense, low-dimensional vector
Collective name for the two 2013 papers (Mikolov et al.) and the
BERT's subword tokenisation algorithm.
An agent with one focused job (research, writing, coding).
The real files you see and edit.
An isolated copy of your git repository where an agent can work without touching your main checkout — handy for parallel or experimental changes.
Weights x-bit, activations y-bit (e.g.
A special token prepended to every BERT input.
Special token inserted between input segments (premise and hypothesis, question and answer) during fine-tuning.
Special token appended at the end of the input sequence during fine-tuning.
The special token used to replace selected tokens during MLM pre-training.
A separator token appended after each sentence in BERT's input.
Special token prepended to every input sequence during fine-tuning.
The repository's actual database — objects, refs, HEAD, index, config, hooks, logs.
Lists files Git should not track; only affects untracked files.
End-of-sentence token.
Start-of-sentence token.
Stage changes for the next commit.
A cross-tool standard file giving AI agents project guidance.
Frontmatter listing tools a skill may use without per-use confirmation.
Frontmatter glob that limits a path-specific instruction to matching files.
Binary-search history to find the commit that introduced a bug.
Show who last changed each line and in which commit.
List/create branches and move between them.
Copy a single commit onto the current branch.
Delete untracked files (not recoverable — use `-n` first).
Download a full copy of a remote repository once.
The repository's local settings and remotes.
The repo-wide instruction file at `.github/copilot-instructions.md`, applied to everyone.
The frontmatter field Copilot matches against your request to decide whether to load the skill.
The recommended tool to rewrite history and purge a file (e.g.
Scripts Git runs automatically at events (pre-commit, pre-push, etc.).
The staging area, in binary form.
A personal, uncommitted ignore list.
Create a new repository (`.git` folder).
Records every position HEAD has held; powers the reflog.
A bookmark of where HEAD was before a big operation, for easy recovery.
Your locally cached view of the remote's branch; updates only when you sync.
The conventional name for the default remote.
Many refs compressed into one file for efficiency.
Upload commits / download without merging / download and merge.
Replay your commits onto another branch for a linear history; never on shared commits.
Show the history of HEAD's movements; the recovery safety net.
Folder of pointer files: `heads/` (branches), `tags/`, `remotes/`.
Move the branch pointer back: `--soft` (keep staged), `--mixed` (keep unstaged), `--hard` (discard).
Unstage a file or discard working-directory changes.
Add a new commit that undoes an earlier one; safe for shared history.
A skill's entry file: YAML frontmatter (`name`, `description`, optional fields) plus an instructions body.
Temporarily shelve uncommitted changes.
See pending changes / line-level changes / history.
Mark a commit with a permanent name (e.g.
Convention for the original repo when you've forked it.