test-time-training · long-context · fast-weights · llm-memory · evaluation
Write what the model has already read into a fixed-size block of mid-layer parameters. Can it later cite the actual facts in that block?
Protocol: EXPERIMENT.md. Checklist: TODO.md.
Phase 0 in progress. Frozen Qwen3-4B-Base on a 5090. Budget K=2702, gate 1, and ceiling passed. Live L3 rules: last digit / K-M region. reverse+add7 discarded. Next: lr band [3e-5, 1e-3] and giveaway passed. Then needle just outside the 512 window.
No Phase 2 scores yet.
The usual setup keeps the whole history in context. Longer history, larger KV cache. Truncate the window, and the early facts are gone.
A line of work writes that history into weights instead. Most of those papers report long-document perplexity and mixed long-context scores. Almost nobody, on a frozen pretrained model, measures the two things separately: what actually landed in the fixed-size state, and what got pushed out. Almost nobody runs a control that separates memory from a bit of extra adaptation to that kind of text.
That is the question here.
The state counts as rolling only if all three hold:
- Capacity is fixed. It does not grow with the conversation.
- Each step is written from the previous state plus the new input.
- New writes decay, evict, or overwrite the old.
KV cache fails (1). Swapping a different slice of neurons without writing new content fails (2) and (3). This is not MoE, and it is not a longer attention window.
In-Place TTT showed that a drop-in write on a 4B model can move numbers on long-context tasks. TTCD changed the write objective to "distant information that will matter later," and put the state in the MLP down projection. Those papers ask whether the write improves scores. Both evaluated in the pretrain or continual-pretrain setting, not on a frozen backbone.
We reuse their write and ask something narrower: after the write, can the model cite a specific fact; after more tokens are written, is the earliest fact still there.
SR-TTT v2 is why the evaluation has to be written this way. v1 reported needle-retrieval gains. v2 retracted them: the loss and the metric read logits at the answer positions, which trains the model to copy an answer already in the input; the cache also attended to future tokens. After the leaks were closed, a from-scratch TTT model with no attention window scored 0% generation exact match at every length and depth they tested. Early-context needles were almost never stored.
We do not stop there. We start from a pretrained backbone and keep a short local window. That difference is real. It also forces a constraint: the no-state baseline must use the same window length as the with-state run. Otherwise you cannot tell the state from the window.
| Model | Role |
|---|---|
Qwen/Qwen3-4B |
Frozen backbone. Main experiments. The write config is tuned here |
Qwen/Qwen3-8B |
Frozen backbone. Same config, no retuning. Scale check |
Both run in the first phase. Every knob — window length, state size, decay, learning rate, chunk size — is fixed on 4B and carried to 8B unchanged. A knob tuned separately on 8B makes the two rows incomparable.
Granite-3.1-1B-A400M was dropped. Sparse routing sends each update to a few experts only, so a weak result cannot be told apart from a routing artifact.
The write is the strongest existing setup: IP-TTCD's location (MLP down projection) and its teacher-student objective. The contribution is the measurement table and the controls, not a new write rule.
S is a fixed-size low-rank correction to W_down. Report its size two ways in every table:
- Trainable parameters,
2 · r · (d_ff + d_model) · (number of adapted layers). - The same number expressed as equivalent KV tokens,
bytes(S) / bytes(KV per token), with the per-token figure read off the model config, not assumed.
Without (2) a reader cannot tell whether the comparison against a short window is fair.
Decay is explicit: S ← λ·S + ΔS. Report λ and the effective half-life in tokens. λ = 1 is a valid arm; it fails criterion (3) and should overwrite badly, which is a prediction to test.
Diagnostic. Write with short-window next-token loss. That loss is dominated by frequent, easy tokens. Names and IDs that appear once contribute almost no gradient. We expect perplexity may drop and fact recall to be poor. This round is to see what the loss is actually optimizing. It is not evidence that memory works.
Main. Write with the hidden-state gap between a long-window teacher and a short-window student. That target is aimed at distant information that will be useful later.
One disclosure: the main-round write runs a long-window teacher, so memory during writing grows with the teacher window. "Memory does not grow with document length" holds on the read side only. The diagnostic round can be bounded on both sides. The longest teacher window that fits on 32 GB is a hard limit; record the measured number and the dtype.
Scoring is generation match, not logits at the gold position. Needles are synthetic names, dates, and IDs, not world facts the pretrained model might already know.
A startup self-test must pass before any score is reported:
- Perturb future tokens. Past outputs must not change.
- Toggle writes on and off. Unperturbed positions must agree.
If the self-test fails, the run is void. This leak has already happened on this line of work.
Fix these before the first run and do not change them afterward. Values below are the intended defaults; confirm them against what fits on one GPU.
| Parameter | Setting |
|---|---|
| Haystack | Long documents, held out, no overlap with the needle vocabulary |
| Context lengths | Several fixed points, held constant across all arms |
| Needle depths | Several fixed relative positions, including one near the start |
| Needles per condition | Enough for a usable confidence interval, reported with one |
| Local window | Fixed length, identical in every arm |
| Fact recall | Generated answer, normalized exact match against the needle value |
| Verbatim | Generated span against the original wording, fixed span length, one string metric |
| Overwrite | Fact recall at each "N tokens written" checkpoint |
Five arms. Drop one, and the table cannot be read.
| Arm | What it isolates |
|---|---|
| State on / state off | Same window length. The gap is from the state, not from the window |
| Write the true fact | The reference condition |
| Write a swapped fact | Same document; only the needle is replaced by another value of the same type. Score which value the model produces. Producing the swapped value means the state carried content. Producing the original value means a leak or a prior |
| Write no fact | Same document, needle removed, length and domain matched. This is the guessing floor. Any arm at or below it measured nothing |
In-place W_down / new adapter |
Whether any gain comes from weights the model already knows how to read |
Capacity and decay are swept on top of these, not in place of them.
Overwrite test: do not write during questions. For each "N tokens written" point, ask from that checkpoint. Do not ask once and then keep writing — that measures repeating a question just asked.
Full context is an upper bound only. It is not rolling; KV grows.
Three lines, reported separately. Do not merge them into one score.
- Fact recall. After the evidence has left the window, is the fact answered correctly.
- Verbatim. Can the original wording be reproduced. A fixed-size state is lossy; this line will be worse than (1). The gap is how much meaning survived without the words.
- Overwrite. After N new tokens are written, is the earliest fact still there. KV does not overwrite. This state must. That is the cost relative to KV.
Memory is reported twice per run: steady-state read-side, and peak write-side.
On a frozen model with a short window kept, fact recall may be above zero, and still far below full context. Early positions will be worse than late ones. That position bias is a main result, not a nuisance.
Verbatim near zero is a reasonable outcome, not a failed run.
Content that needs reasoning or stitching across passages will mostly not come back. A new adapter may help almost not at all. Those two are hypotheses.
If all three curves are zero, that is still a result, provided the evaluation can show it did not leak. This line of work is short on negative results that can prove that.
- A new rolling architecture, or a new write rule.
- Lower long-document perplexity on its own.
- Open-ended generation fluency.
- Pretraining from scratch.
Fast Weight Programmers; Dynamic Evaluation (Krause et al., 2018); TTT layers (Sun et al., 2024); Infini-attention; Titans (Behrouz et al., NeurIPS 2025); SEAL. Titans and Infini-attention train the write from scratch; we do not.
Closest in design, and the write we reuse:
- In-Place TTT (Feng et al., ICLR 2026, arXiv:2604.06169) — adapts the existing
W_down. - TTCD / IP-TTCD (Wang, Dang et al., arXiv:2608.01672, code at dangxingyu/ttcd) — the teacher-student write objective.
Against the premise:
- Latent Context Compilation (arXiv:2602.21221) argues that writing context into weights produces stateful parameters that complicate concurrent serving, and keeps buffer tokens instead. If the three curves come back flat, this is the reading that wins.
- TTT-KVB (arXiv:2602.21204) argues TTT is closer to learned linear attention than to store-and-retrieve. The swapped-fact and no-fact arms are there to answer that.
Protocol:
- SR-TTT v2 (arXiv:2603.06642) — the evaluation leak this repo self-tests against.
Python 3.10 or later. NVIDIA driver with CUDA 12.8 support.
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
python scripts/download_model.py --repo-id Qwen/Qwen3-4B
python scripts/verify_inference.py --model-dir models/Qwen3-4BInstall torch first from the CUDA index. requirements.txt does not list torch, so the CUDA build is not replaced by the default wheel.
TODO.md phases and fail criteria (Chinese)
README.zh-CN.md Chinese writeup
configs/ model and generation configs
scripts/ download and smoke tests
models/ local weights, not in git
experiments/ run logs
notebooks/ analysis
src/rlm/ write rules, windows, evaluation — planned, currently a stub
MIT. See LICENSE.