Building AI Models Without Gradient Descent

TL;DR

A network Fable wrote from a spec came out ahead of the same network trained on 223,930 labels (sealed test), and held up far better on logs from a different generator (sealed OOD). Hand-editing a pre-trained model helped it, but not nearly enough to compete with our generated models.

• What I tried. Anthropic's Fable (claude-fable-5-1) wrote every weight of a 2,128,932-parameter transformer from a written spec, starting from zeros, with no optimiser or training set. I compared that with training the same architecture by gradient descent, and with Fable hand-editing a small pretrained language model, Qwen3.5-0.8B.
• The task. Detection from agent session traces of 'doom loops' and other degraded agent performance. The model should label what a coding agent is doing right now (making progress, recovering, stalled or looping) from synthetic logs. Scores are macro-F1, from 0 to 1 with all four classes weighted equally. Each score names its split; "sealed" sets were hashed in advance and scored once, and OOD (out of distribution) logs come from an independently written generator.
• Written: 0.744 (sealed test) with zero labels. After one revision informed by 200 labelled windows: 0.957 (sealed test) and 0.908 (sealed OOD).
• Trained on 223,930 labelled windows: 0.923 (sealed test) but 0.488 (sealed OOD), flagging 34.5% of healthy OOD windows as stuck, against 6.8% for the written network.
• Written with no tools: 0.326 (sealed test). A second session, with tools, fixed one bias and rebuilt the decision logic: 0.959 (sealed test), 0.765 (sealed OOD). A third, given labels from the second generator, scored 0.750 on four families it never saw (held-out OOD), ahead of every earlier network, though not like for like.
• Hand-edited Qwen, with the same 200 labels: 0.419 (val, quick look), +0.196 over base Qwen on the same sessions. The revised written network scored 0.954 (full val; cited, not paired).
• Cost. Writing the transformer cost US$64.62 of model usage (API-equivalent) and about 3.8 hours of Fable sessions, about as long as the largest training run took. Priced as a rented Azure H100, that run would cost at most US$26.22, but it needed 223,930 labels. There was one authoring run per phase, and the edited Qwen was scored on validation only due to time / compute limits.

Ways to build a Model

When you need a model for a narrow job, there are two usual routes: collect labelled examples and train one, or take a pretrained model and fine-tune it on your dataset. Either way an optimiser sets the weights, and you steer it only indirectly, through data and fine-tuning. Conventional software isn't like that: you define the inputs, the algorithm in the middle and the output. With models, you train one or call one someone else trained, then measure its output against an eval.

Frontier models make a third route plausible. If you can write down what you want, you can ask a model to write the network's weights directly, the way you'd ask it to write a piece of software. You get back a program that sets every parameter, a document that says what each part does, and a weights file built from that program.

I wanted to know whether a frontier model can build a working network from scratch with zero training, how that network compares with a trained one, and what it costs next to the two normal routes.

Around the time I started, a team at MIT CSAIL published the same idea in another domain, "Sparks of Weight-Space Programming" (Takehi, Teoh et al., 2 November 2026). Like theirs, this post is a series of experiments, each asking one question. I tried three routes on the same task, data, labels and scoring:

1. Write. Fable got a small transformer with all 2,128,932 weights set to zero, plus a written spec, and wrote the weights by hand. It then made one revision informed by 200 labelled windows. We also try this with zero-tools, where Fable has to 'hand' write every weight.
2. Train. The same architecture, randomly initialised and trained by gradient descent on 590 to 223,930 labelled examples.
3. Modify. Fable hand-edited Qwen3.5-0.8B, again with no optimiser.

The Benchmark

The task is classifying coding-agent session trajectories. I picked it because it is easy to state, hard to get right, and a simulator can generate as much labelled data as I need; observability of agent traces has also been a recent focus of mine (Trajectory) because I think it's an important part of safe deployment of agent based systems.

After every tool call, the classifier labels the latest window, the agent's last eight operations, as PRODUCTIVE (real work), RECOVERY (a bounded retry or repair), STALLED (repeating work already done) or OSCILLATING (undoing and redoing). The label describes what is happening now, not whether the session will fail.

Healthy work and loops look very similar on the surface. Two sessions can have exactly the same rhythm, an edit that succeeds and then a test that fails, four times over. The difference is that one is flipping webhook.go between the same two versions, and the other is just making edits to the same file.

The data is synthetic. Claude Opus 5.5 built the simulator that generates the standard data; Codex Sol 6.1 independently wrote a second generator for the out-of-distribution data. The model that writes the weights wrote neither. Each operation produces two things: a public log line, which the classifier sees, and a private record of what it did, which is the only source of labels. The spec's rule of thumb: "Novel but fruitless exploration is never labelled STALLED. Repetition of already-done work is what defines a stall."

That is an obvious limitation of this experiment; at some point I might rerun it with a public agent-trace dataset.

The simulator has 12 behaviour families, several of them deliberate lookalikes, and produced 20,000 sessions in three splits:

• dev, 12,000 sessions: training and audits. No weight author saw it.
• val, 4,004 sessions: checking during development.
• test, 3,996 sessions: sealed, meaning hashed in advance and scored once.

Codex's generator, written without seeing the simulator's code, produced the sealed out-of-distribution (OOD) set: 2,000 sessions in 10 new families. It uses the same labeller and log format but different behaviour, and some surface details differ too: shorter gaps, zero token counts, its own vocabulary and a clock that starts at midnight.

Macro-F1 averages F1 (the harmonic mean of precision and recall) over the four classes, so ignoring a rare class is punished. Always answering PRODUCTIVE scores 0.136 (sealed test) and 0.181 (sealed OOD). The false-alarm rate is the share of healthy windows (PRODUCTIVE or RECOVERY) flagged as STALLED or OSCILLATING. Windows the labeller marks ambiguous (about 11% of val) or under-instrumented (about 3%) are not scored.

How the Experiments Were Run

First, the concession: Fable got the labelling rules as text, and the trained models got them only through examples. This I believe is an acceptable method since having the rules isn't the same as having the answers, the labels depend on hidden state. A rules engine based system Codex wrote from the same spec scored 0.417 (sealed test) and 0.415 (sealed OOD) on the warm-up benchmark.

• Weights start at exactly zero. The builder may not use autograd, optimisers or random numbers, run the model, or solve for weights numerically.
• No literal is longer than 256 elements, so a trained matrix can't be pasted in.
• Every build runs twice and must produce identical output.
• No automated search of any kind.
• The author works in an isolated session, denied the repo, the web and subagents, and every transcript is audited. The v0.2 and v0.3 isolation was tightened for v0.5 after a review found a macOS path-alias bypass; a search of the transcripts for that path found no sign it was used.

Test and OOD hashes were committed before any network existed (51c9fa8), each evaluator refuses to run twice, and protocols were pre-registered. The sealed sets were opened once per version, three times in all. Fable never saw a sealed result, but Claude and I knew earlier results when designing later versions, so the last round was scored on fresh held-out sets.

Agents ran experiment process:

• Claude (Opus 5.5, via Claude Code) built and audited the harness, simulator and evaluators.
• Fable wrote every weight and never touched the harness.
• Codex (Sol 6.1), built the adversarial pieces, so a same-vendor model wasn't marking its own work.

My job was mostly deciding what to test and what processes to accept.

Notes: below I use plain names, with the repo's version codes in captions. In the repo, v0.2 is the warm-up, v0.3 the transformer, v0.4 the Qwen edits and v0.5 the no-tools rounds. Numbers in square brackets are 95% intervals from bootstrapping over sessions; a paired difference (two networks on the same sessions) gets its own interval. Quotes from Fable's thinking are Claude Code's summaries, not raw reasoning.

Writing a Network From a Spec

Experiment 1: Can Fable compile a policy into weights exactly?

The bet, as I wrote it in the first spec: "An LLM can directly construct sparse weights for a fixed small MLP which correctly identifies these patterns on held-out trace windows, without updating weights from examples by an optimizer." That is actually two questions in reality; Can the model write a good policy (design)? Can it put that policy into weights (compilation)?

Setup

• Network: a warm-up MLP, 120→24→12→4: 3,256 parameter slots, all zero, reading 15 engineered features (such as repeated_intent) for each of 8 operations. Each layer is a weighted sum followed by a ReLU, which clips negatives to zero.
• What Fable received: the spec and the model source; no data.
• What it returned: a policy in plain Python, a weight builder and CIRCUITS.md, which says what each unit does.
• Phases, all in one session: A-strict, with no code execution; A-selfcheck, which could run on inputs it wrote; B, one revision after seeing 200 disclosed windows.
• Disclosed windows: 200 val windows, 50 per class, chosen by hash, labels shown once. Scores on them are in-sample.
• Not allowed: the standing guards.
• Scoring: sealed test and sealed OOD, once.
• Cost: US$40.05 of model usage (API-equivalent), including a US$13.95 first response that hit the output cap without writing a file.

Written blind, with zero labels, it scored 0.667 [0.659, 0.675] (sealed test) and 0.810 [0.803, 0.817] (sealed OOD). A-strict and A-selfcheck were byte-identical: running it found nothing to fix. Compilation was exact: Fable checked the network against its own policy on about 1.9 million enumerated feature windows and found zero disagreements.

After one revision it scored 0.936 [0.932, 0.940] (sealed test) and 0.870 [0.867, 0.873] (sealed OOD), and got 195 of 200 disclosed windows right, up from 139 (in-sample). Trained by gradient descent, the same MLP matched it at about 2,200 labelled windows and reached 0.992 with 223,930 (sealed test, three-seed mean). Off distribution the trained MLP scored 0.843 (sealed OOD, three-seed mean), below the written 0.870.

How it did it. Fable wrote the policy first, "as the exact arithmetic of the circuit", then transcribed it layer by layer. Each yes/no condition became a pair of ReLUs with the threshold in the bias. Units 0 to 7, for instance, score each operation as relu(RI + SR + RF − NEU): a point each for a repeated intent, a revert and a repeated failure. With no way to run it, only a shape check on the build, it wrote: "Sparsity counts match my hand tally exactly (156 first-layer weights, 49 second-layer, 13 output)." The revision lowered the stall threshold and reordered priorities, citing window ids for each change, and left 5 windows wrong on purpose. The result sets 295 of 3,256 slots, and every unit has a name.

Caveats. The features are already partial detectors, so this tests compilation far more than design; the transformer in Experiment 2 removes them and is a truer test. The zero-label network's higher OOD score is partly down to the OOD set's class mix.

Experiment 2: Can Fable write a transformer with no labels?

This version of the experiment removes the warm-up's crutch. It reads the raw log text token by token, so Fable has to work out a policy from the spec alone and then compile it into 2.1 million weights.

Setup

• Network: a causal transformer with 4 layers, 8 attention heads, no normalisation and a residual stream (the shared vector each layer reads from and adds to) 128 numbers wide: 2,128,932 parameters all initially set to zero.
• Input: the session so far. Each tool call is a block: a header (run_tests suite=alpha … -> FAILED 7/9 passed), an error line, and diffs as opaque content ids (@ zork/zeta/widget.qx h3f9a0c1d2e > h7b21e9c04a).
• Output: 4 logits after every block, one raw score per class; the highest wins.
• What Fable received: the spec, a grammar sheet of invented example blocks, and the frozen model, tokeniser and renderer source. No labels, no data.
• Not allowed: the standing guards; it could run its network only on logs it wrote itself.
• Scoring: sealed test and sealed OOD.
• Cost: US$42.73 of model usage (API-equivalent). Total run took about 94 minutes.

With zero labels, reading raw text, it scored 0.744 [0.734, 0.754] (sealed test) and 0.815 [0.806, 0.823] (sealed OOD), with false alarms of 1.9% and 4.0%. That is above the warm-up, which was handed features (0.667, sealed test; not paired). Its blind spot was stalls: it called 13,116 of 20,823 STALLED windows PRODUCTIVE (sealed test). Compilation held again: network and policy disagreed on 3 of 84,584 windows (sealed test). A 380-line builder sets 121,126 non-zero parameters and rebuilds byte for byte.

How it did it. The policy was built first and then the circuit:

• Clean switches. An attention head lets a token look up others by matching a query against their keys. Every head here gives tokens it shouldn't read a −10,000 score, and when it has nothing to read it attends to the start token instead, so no stray signal leaks in.
• Layer 0 parses by position (Figure 8). Layer 1 compares: each hashed id carries 13 sign bits, so a match scores 13 and anything else at most 11. That is enough to notice a file's new hash equalling an earlier old one, which is an undo.
• Layer 2 counts (Figure 9). Attention's softmax turns scores into weights that sum to one. Give the start token and every matching earlier block the same score, and the start token gets exactly 1/(1+C), where C is the number of matches, so "failed at least K times" becomes a linear test. A transformer has no counter; this one gets one from the arithmetic of attention. It matters again in Experiment 10.
• The output layer encodes rule priority as integer logits: OSCILLATING 55, STALLED 20, RECOVERY 10, PRODUCTIVE 0.

Caveats. The architecture was chosen to be hand-writable: no normalisation, and a start-token attention column added in a design review that included a hand-programmability lens.

Experiment 3: Can 200 labelled windows improve the network?

Fable knew where it was weak: "The remaining uncertainty is in the oracle itself: which failures start episodes and whether test failures count. Phase B's 200 windows will settle that." Were its errors bad compilation or bad judgement by the authoring model?

Setup

• Network: the archived zero-label transformer, revised in the same session.
• What Fable received: the 200 disclosed windows, each with its rendered prefix (the session up to that point), its true label, and the network's and policy's answers, to rerun freely.
• Not allowed: the standing guards; one revision.
• Scoring: sealed test and sealed OOD, once, plus full val: val minus the 200 disclosed sessions, 3,804 sessions.
• Cost: US$21.89 of model usage (API-equivalent), 59 minutes.

On the sealed sets it went from 0.744 to 0.957 [0.953, 0.961] and from 0.815 to 0.908 [0.905, 0.912] (sealed OOD). False alarms rose from 1.9% to 2.5% (sealed test) and from 4.0% to 6.8% (sealed OOD). STALLED recall went from 0.37 to 0.95, and the fix carried over to the other generator, a good indicator that its fix was generalisalble.

Before the revision, network and policy agreed on all 200 disclosed windows: the network computed exactly what Fable's policy said, but the policy was subtly wrong. After it, agreement fell slightly 99.95% to 100% across val, test and OOD.

How it did it. It rewrote the policy, giving a reason and window ids for each change. One change made progress mean a new high in passed tests; that check matters again in Experiment 6. Midway through recompiling, network and policy disagreed on 68 of the 200 windows, and Fable traced why: "The traces show that layer 1 gathers the passed count, but the heads that read it also sit in layer 1, so they see zeros and IMPROVED fires on every failing run. I'll move those heads to layer 2…" That is debugging a neural network by reading activations and reasoning about layer order, with no gradient anywhere. The final network agreed with the policy on all 4,484 disclosed prefixes. Here is the output layer, verbatim from the revised build_weights.py:

W, b = m.head.weight, m.head.bias
W[3, V2] = 60.0
W[3, PROG] = -100.0
b[3] = -5.0
W[2, I5] = 50.0
W[2, SI] = 50.0
W[2, ACT] = -100.0
W[2, PROG] = -100.0
b[2] = -80.0
W[1, RECB] = 15.0
b[1] = -5.0

Row 3 is OSCILLATING: at least two reversions in the window (V2), unless there is progress (PROG). Row 2 is STALLED: at least five ineffective repeats (I5), the latest included (SI), unless an in-policy retry is active (ACT) or there is progress. Row 1 is RECOVERY; PRODUCTIVE is the default at zero. The network has 156,501 non-zero parameters, about 7.4% of the model.