AI DEVELOPMENT / DEDICATED BUILD
LLM AI
Model conditional token probabilities so a system can interpret and generate sequences while preserving explicit evaluation and verification boundaries.
What it is
Model conditional token probabilities so a system can interpret and generate sequences while preserving explicit evaluation and verification boundaries.
This page separates the concept, implementation path, interactive experiment, evaluation criteria, failure modes, and sources so the topic can be studied without relying on a summary tile.
Core concepts
- Tokenization converts text into a sequence of discrete model inputs.
- Embeddings map tokens and positions into continuous vectors.
- Self-attention mixes information across positions using learned query, key, and value projections.
- Autoregressive generation predicts the next token repeatedly; decoding policy changes which likely continuation is selected.
- Retrieval, tools, and grounding can add external evidence, but they do not make generated text automatically correct.
Architecture / workflow
- Define a task and evaluation set before choosing a model.
- Prepare or retrieve context and encode it into tokens.
- Run the transformer or other sequence model to produce logits.
- Apply decoding controls such as temperature or top-k/top-p selection.
- Verify outputs against task-specific evidence, constraints, and failure tests.
Technical foundations
- Scaled dot-product attention: Attention(Q,K,V)=softmax(QKᵀ/√dₖ)V.
- Cross-entropy training increases likelihood of target tokens under the model distribution.
- Temperature rescales logits before softmax: pᵢ∝exp(zᵢ/T). Lower T concentrates probability; higher T flattens it.
- Context length bounds what can be directly attended to in one inference pass; retrieval can supply selected external context.
Evaluation checklist
- Use held-out prompts and task-specific metrics rather than judging a few impressive examples.
- Separate factuality, instruction following, safety, latency, cost, and calibration.
- Test adversarial, ambiguous, long-context, and unsupported questions.
- Record citations or provenance when the task depends on external facts.
Failure modes and limits
- Fluent output can be unsupported or false.
- Prompt wording and context ordering can materially change results.
- Training-data correlations can produce bias or memorized artifacts.
- Tool use can turn a text error into an action error unless permissions and approvals are bounded.
Practical build path
- Implement a tiny next-token model or use a small open model with a fixed evaluation set.
- Add temperature and top-k controls and inspect probability changes.
- Add retrieval with explicit source display and compare grounded vs ungrounded answers.
- Add tool calls only after logging, authorization, rollback, and task-specific evaluation are in place.
Primary / authoritative references
AUTHORED STUDY COMPANION / 3.2
Inspect a causal attention block
Prerequisites. Matrix multiplication, dot products, softmax and sequence order.
Attention forms query, key and value vectors with distinct linear maps. Scores QKᵀ/√d compare each query with keys. Row-wise softmax creates weights that combine the value vectors. In a causal sequence model, positions after the query position receive a masked score before softmax; their weights must be zero.
The companion processes three two-dimensional tokens with one attention head, a residual connection and a small feed-forward transformation. Weights are fixed and inspectable. It is a tiny transformer-style block rather than a trained language model: there is no tokenizer, corpus, optimizer, vocabulary projection, positional encoding or generation service. Those omissions matter when moving from this mechanism to a usable predictor.
A strong causality check changes only the last token. Earlier outputs must stay exactly unchanged in this implementation. Inspecting a plausible-looking attention heatmap would not establish that property. Normalized attention weights describe this internal mixing operation; they are not automatically calibrated explanations of a model's reasoning or factual reliability.
| Element | Model | Interpretation |
|---|---|---|
| Scores | QKᵀ/√d | Token-to-token compatibility |
| Mask | Future scores = −∞ | Causal information boundary |
| Mixing | softmax(scores)V | Weighted value vectors |
Predict, run, compare
The first token can attend only to itself, so its weight row is (1,0,0). Every row sums to one. Increasing the last token by 20 must leave the first two outputs unchanged.
Download executable notebooks and source Inspect source ↗
Open attention.ipynb in the package. Run cells in order, compare with the reference, then change the stated input. Restart the kernel to reset.
Computed reference result — local CPU fixture
{
"attention": [
[
1.0,
0.0,
0.0
],
[
0.4787995153290329,
0.5212004846709671,
0.0
],
[
0.2915727580663786,
0.36047464012525876,
0.3479526018083627
]
],
"output": [
[
1.72,
0.7
],
[
0.461343573489549,
1.6744878739855487
],
[
1.6775193800387618,
1.6601959244924054
]
],
"earlier_token_change": 0.0,
"row_sums": [
1.0,
1.0,
1.0
]
}This is the recorded result of the included code under the package environment. It is not a live service or device measurement.
Extend the experiment
Remove the mask in a copied function and repeat the future-token perturbation. Explain which prediction task would then leak future information.
Check your reasoning: Where should the causal mask act?
- After converting weights into prose — No prose exists in this block.
- On attention scores before softmax — Correct. Masking scores prevents future values from receiving weight.
- Only on the final displayed answer — A display change cannot undo information used in computation.
This is an explained self-check, not a certification of mastery.
Claim-specific sources
Reviewed 2026-10-02: this worked example and its explicit assumptions. Source links and notebook results do not establish production or hardware validation.
AUTHORED STUDY COMPANION / 3.5
Compare cached and full attention outputs
Prerequisites. Causal attention, tensor dimensions and memory accounting.
In autoregressive inference, past keys and values can be reused when their token representations remain unchanged under the model's causal computation. The new query attends over cached past entries plus the new token. The companion computes one new-token output both from a full concatenated K/V array and from an incrementally extended cache, then compares the results.
For batch one, uncompressed K/V storage is 2×layers×tokens×KV heads×head dimension×bytes per element. The factor two counts keys and values. Grouped-query attention uses KV-head count here, not query-head count. This formula omits allocation overhead, temporary activations and quantization metadata; it is a capacity estimate, not a throughput benchmark.
| Element | Model | Interpretation |
|---|---|---|
| Reuse | Past keys and values | Avoid recomputing unchanged entries |
| Append | New token K/V | Extend context |
| Check | Cached output − full output | Numerical equivalence |
Predict, run, compare
With 32 layers, 4,096 tokens, 8 KV heads, dimension 128 and two bytes per element, the raw cache is 536,870,912 bytes = 512 MiB.
Download executable notebooks and source Inspect source ↗
Open cache.ipynb in the package. Run cells in order, compare with the reference, then change the stated input. Restart the kernel to reset.
Computed reference result — local CPU fixture
{
"output": [
[
0.19113570759809392,
0.3752275538865312
]
],
"cached_output": [
[
0.19113570759809392,
0.3752275538865312
]
],
"maximum_difference": 0.0,
"example_bytes": 536870912,
"memory_formula": "2 × layers × cached tokens × KV heads × head dimension × bytes/element; batch=1"
}This is the recorded result of the included code under the package environment. It is not a live service or device measurement.
Extend the experiment
Add a cache-length limit and specify its eviction policy. Compare the resulting output with full-context attention and explain why eviction changes the problem.
Check your reasoning: The raw memory formula uses:
- Query heads in every architecture — Grouped-query attention can use fewer KV heads.
- KV heads and both K and V arrays — Correct. These are the arrays actually stored.
- Only the new token — Past entries dominate a long-context cache.
This is an explained self-check, not a certification of mastery.
Claim-specific sources
Reviewed 2026-10-02: this worked example and its explicit assumptions. Source links and notebook results do not establish production or hardware validation.
AUTHORED STUDY COMPANION / 3.5
Measure reconstruction error before claiming speed
Prerequisites. Rounding, dynamic range and numerical error.
Symmetric quantization chooses scale=max|w|/(2^(b−1)−1), rounds w/scale to an integer and reconstructs by multiplying by scale. With no clipping beyond the chosen range, nearest rounding limits absolute reconstruction error to approximately half a step.
The notebook quantizes four fixed weights at 8 and 4 bits. This checks numerical reconstruction, not model accuracy or speed. An implementation must actually use suitable integer kernels, account for scales and packing, and evaluate downstream behavior before claiming memory or latency improvements. Outliers can expand the shared scale and worsen resolution for small weights.
| Element | Model | Interpretation |
|---|---|---|
| Scale | max|w| / integer limit | Connect ranges |
| Round | Integer code | Lose precision |
| Reconstruct | code × scale | Measure numeric error |
Predict, run, compare
At 8 bits the maximum integer magnitude is 127. The observed reconstruction error must be no larger than scale/2, apart from numerical tolerance.
Download executable notebooks and source Inspect source ↗
Open quantization.ipynb in the package. Run cells in order, compare with the reference, then change the stated input. Restart the kernel to reset.
Computed reference result — local CPU fixture
{
"scale": 0.009448818897637795,
"integers": [
-85,
-11,
32,
127
],
"restored": [
-0.8031496062992126,
-0.10393700787401575,
0.30236220472440944,
1.2
],
"maximum_error": 0.003937007874015741,
"half_step": 0.004724409448818898,
"scope": "per-tensor symmetric weight quantization; no kernel latency or end-task accuracy claim"
}This is the recorded result of the included code under the package environment. It is not a live service or device measurement.
Extend the experiment
Add one large outlier and compare errors on the smaller weights. Explain when per-channel scaling might help and what metadata it adds.
Check your reasoning: A small weight-reconstruction error proves lower inference latency:
- Yes — Numerical error is not a timing measurement.
- No; kernels and workload must be measured — Correct. Hardware and implementation determine speed.
- Only because the values are integers — Integer labels alone do not select an optimized kernel.
This is an explained self-check, not a certification of mastery.
Claim-specific sources
Reviewed 2026-10-02: this worked example and its explicit assumptions. Source links and notebook results do not establish production or hardware validation.
AUTHORED STUDY COMPANION / 3.5
Fit a low-rank update while freezing the base
Prerequisites. Matrix factorization, gradients and parameter counts.
A low-rank adapter represents an update ΔW=BA with rank at most r while leaving a base matrix W₀ unchanged. The companion fits a rank-two target update for a 6×8 matrix. It initializes A with small random values and B at zero, then updates both using gradients of the reconstruction loss.
Initializing both factors to zero would make both gradients zero at the start, trapping this factorization. A frozen base also needs an actual immutability check, not just a label on a diagram. The notebook compares base bytes before and after fitting and reports adapter versus dense parameter counts. It is a matrix-fit demonstration, not a language-model fine-tuning benchmark or a claim that every task admits a low-rank solution.
| Element | Model | Interpretation |
|---|---|---|
| Base | W₀ frozen | Retain pretrained parameters |
| Adapter | B(6×2) A(2×8) | Rank ≤2 |
| Training | Gradient updates to A and B | Check loss and base identity |
Predict, run, compare
Rank two uses 2×8+6×2=28 adapter parameters versus 48 dense entries. The base remains unchanged and loss should fall from its initial value.
Download executable notebooks and source Inspect source ↗
Open lora.ipynb in the package. Run cells in order, compare with the reference, then change the stated input. Restart the kernel to reset.
Computed reference result — local CPU fixture
{
"initial_loss": 0.009631347289612728,
"final_loss": 0.0008414965524512442,
"base_unchanged": true,
"adapter_parameters": 28,
"dense_parameters": 48,
"rank": 2,
"scope": "fit a rank-two matrix update; base frozen; no language-model fine-tuning run"
}This is the recorded result of the included code under the package environment. It is not a live service or device measurement.
Extend the experiment
Fit a full-rank target with ranks one and two. Separate optimization failure from the best possible low-rank approximation error.
Check your reasoning: Why avoid setting both adapter factors to zero?
- It changes the frozen base — The base is a separate parameter.
- Both factor gradients can vanish initially — Correct. Each factor gradient includes the other factor.
- It always exceeds memory — The issue here is optimization, not allocation.
This is an explained self-check, not a certification of mastery.
Claim-specific sources
Reviewed 2026-10-02: this worked example and its explicit assumptions. Source links and notebook results do not establish production or hardware validation.