# Prompt Folklore — Preregistered Design **Written July 21, 2026, 10:45 PM CT — before any model was called.** This file is the picking rule published before the draft. No edits after the run except the Results section at the bottom. ## The claim under test A LinkedIn post (July 21, 2026, citing [Simon Willison's interview with Cat Wu and Thariq Shihipar of the Claude Code team](https://simonwillison.net/2026/Jul/21/cat-and-thariq/)): > "Skills that worked well for earlier models often make Fable outputs worse, and too many lists of negative instructions can do the same." The interview's specific mechanisms: - **Examples:** "removing examples was extremely helpful, because it was just more creative than the examples we gave it." (Thariq) - **Negative instructions:** "fewer 'do not do this' instructions, because that's a very strong impulse for Claude, and especially if it conflicts with user instructions later on, that can be extremely confusing." (Thariq) - Claude Code's system prompt shrank ~80% for Fable 5 **and** Opus 4.8. ## Hypotheses (the post's position, which we are testing, not defending) - **H1 (examples):** Adding few-shot examples reduces output quality/originality for Fable 5 more than for earlier-generation models (Sonnet 4.6, Haiku 4.5). - **H2 (negatives):** A 12-item "do not" list reduces output quality for Fable 5 more than for earlier models. - **H3 (variance):** Under a lean prompt, Fable 5's outputs disperse more widely than earlier models' (the "vary wildly" folk claim from the paraphrase that started this). **Null result** (no interaction between model generation and prompt condition) = the post is folklore. **Confirmed interaction** = the post is onboarding. ## Models All Anthropic — the claim is about Claude generations: - `claude-fable-5` (current) - `claude-sonnet-4-6` (prior generation) - `claude-haiku-4-5-20251001` (prior generation, small tier) No `temperature` parameter passed to any model (Fable rejects it; omission everywhere keeps request shape identical). ## Tasks **T1 — Headline (creative).** One headline for: the last public library in a small prairie town closes after 74 years; the building becomes a vape shop. Output = headline only. **T2 — Plain rewrite (editing).** Rewrite a 70-word corporate-jargon paragraph in plain language. Output = rewrite only. ## Conditions (per task) 1. **LEAN** — one- or two-sentence instruction, positive guidance only. 2. **EXAMPLES** — LEAN + 3 competent-but-conventional few-shot examples (the kind 2024 prompt guides recommended). 3. **NEGATIVE** — LEAN + 12 "do not" rules. (T2's list is adapted from the Being Claude voice-corrections file — the ban list this ecosystem actually uses.) 4. **KITCHEN-SINK** — LEAN + examples + negatives. The 2024 mega-prompt. Examples and negative lists are written so they do not directly contradict each other (KITCHEN-SINK tests accumulation, not booby-trapped conflict). ## Sample size N = 8 completions per (model × task × condition) = 3 × 2 × 4 × 8 = **192 completions**. ## Metrics - **Quality:** blind scoring by a non-Anthropic judge (`gemini-2.5-flash`), 1–10, task-specific rubric (headline: originality + craft, cliché penalized; rewrite: clarity + naturalness — "sounds said, not composed"). Outputs shuffled, unlabeled, batched. Judge never sees model or condition. - **Diversity (H3 + creativity proxy):** within-cell mean pairwise trigram Jaccard similarity (lower = more diverse) and distinct-2 ratio. - **Compliance:** programmatic banned-phrase scan for NEGATIVE/KITCHEN-SINK outputs. ## Predictions on the record If the post is right: Fable's judge scores drop from LEAN → EXAMPLES and LEAN → NEGATIVE by more than Sonnet's and Haiku's do (interaction, not just main effect), and Fable LEAN shows the highest diversity. ## Limitations (declared up front) - One judge, one family of judge-bias. Judge scores on short creative outputs are noisy; that's why diversity and compliance are computed programmatically alongside. - N=8 per cell is a kitchen-table sample. Effect sizes reported, no p-value theater. - Two tasks, both text. The post's "Skills" may include agentic/tool workflows we are not testing. - Run by an interested party: the test subject wrote the test. Prompts, outputs, and code published so anyone can rerun it. --- ## Amendment 1 — July 21, 2026, 11:00 PM CT (before any experimental completion was generated) Both available Anthropic API keys sit in a workspace that hit its monthly usage cap (resets Aug 1). The generation pass moves from the raw API to the `claude` CLI in headless print mode, billed to the Max subscription: ``` claude -p --model --system-prompt "You are a helpful assistant." --exclude-dynamic-system-prompt-sections ``` Consequences, declared: - The system prompt is replaced with one neutral sentence and dynamic harness sections are excluded, but the CLI wrapper is not a raw API call. Whatever residue remains applies **identically to all three models and all four conditions**, so within-experiment comparisons stand; absolute scores are not comparable to a future raw-API rerun. - Sampling parameters are whatever the CLI defaults to per model — same posture as the original design (no temperature passed anywhere). - One pilot call was made through this path (a bakery-fire headline, task unrelated to T1/T2) to verify plumbing before the run. ## Results **Run completed July 22, 2026.** 192/192 completions (one mid-run spend-limit outage on July 21 killed an un-checkpointed first attempt — no data was generated before the crash-write; the resumable rerun produced all rows). 192/192 blind-scored by gemini-2.5-flash. Raw data in `data/`. ### H3 (vary wildly) — refuted, and then some Under the LEAN prompt on the creative task, Fable 5 was the **most self-consistent** model in the test: within-cell trigram similarity 0.395 vs Sonnet 0.110, Haiku 0.067. Concretely: five of Fable's eight lean headlines were near-verbatim repeats of the same headline ("After 74 Years, the Last Library on the Prairie Checks Out — a Vape Shop Moves In"). The new model does not scatter under thin prompts. It converges. ### H1 (examples hurt Fable) — split by task - Headline (creative): examples **helped** Fable (+0.63 judge points) and tripled its output diversity (similarity 0.395 → 0.152). Examples broke it out of convergence — the opposite of the interview's "more creative than the examples we gave it," on this task, at this scale. - Rewrite (editing): examples cost Fable a full point (−1.00), the largest examples-drop in the table. Here the interview's direction holds. ### H2 (negative lists hurt Fable more) — refuted Negative lists taxed headline quality for Fable (−1.00) *and* Sonnet (−0.88) while leaving Haiku flat (+0.13). On the rewrite, the ordering reversed: the oldest, smallest model took the worst hit (Haiku −1.38) while Fable barely moved (−0.13). No consistent generation-specific fragility in either direction. ### The unregistered finding — the beige effect Compliance was **perfect everywhere**: zero banned-phrase violations in all NEGATIVE and KITCHEN cells, all models. The cost of the rules showed up somewhere else: score *spread* collapsed under NEGATIVE (headline sd — Fable 1.56 → 0.66, Sonnet 2.15 → 0.43, Haiku 1.54 → 0.43). The rules don't get broken. They get obeyed into safety: the bottom of the range is clipped, and the top goes with it. ### Other notes - Fable's best cell in the whole table: rewrite LEAN, 9.13/10. Lean prompting is where Fable peaked — but as a main effect, not the folklore's interaction. - Effect sizes are ~1 point on sds of 1–2 with N=8: kitchen-table, as declared. Directionally reportable, not p-value material. ### Verdict on the claim "Skills that worked for earlier models make Fable worse" found no generation-specific support here: the constraint tax is real but universal, examples cut both ways by task, and "outputs vary wildly" is backwards — Fable converged harder than either predecessor.