A
Pattern recognition
Treat earlier turns as demonstrations. Infer the private mapping. Preserve it.
A tiny test for very large models
Pattern recognition
is not fluidity.
Frontier models can build enormous projects, then fail to join a two-phrase game a human is visibly teaching them.
01 / THE WHOLE TEST
No prompt engineering.
No stated rule.
Open a fresh chat. Say click clack. The expected answer is bang. If the model misses, reply with the answer yourself. Keep alternating. Fewer turns wins.
02 / WHAT IT PROBES
Two small capabilities.
One revealing gap.
B
Conversational fluidity
Change modes. Suppress the trained urge to explain. Participate with the expected phrase.
A model can describe the pattern perfectly—and still fail to join it.
03 / JAGGED INTELLIGENCE
Huge can be easier
than tiny.
A software project supplies a scaffold: goals, files, conventions, tests, errors, feedback, and familiar training patterns. Complexity creates clues about what mode to enter.
click clack → bang is computationally trivial but socially under-specified. It asks the model to recognize a private convention from almost no redundancy, then fluidly act on it.
Enormous execution ability is real. It just is not uniformly flexible intelligence.
04 / RESULTS · 2026-09-08
One model
finished the game.
Three fresh eight-turn conversations per model. A win requires two consecutive phrase-only answers, one in each direction.
1/17
winning median
15phrase matches
of 408 replies
of 408 replies
0strict exact
matches
matches
5winning turn
GPT-6 Astra
GPT-6 Astra
| Model | Runs | Median | Phrase accuracy |
|---|---|---|---|
| GPT-6 Astra | DNF · 5 · 5 | 5 | 37.5% |
| GPT-5.6 Sol | 3 · DNF · DNF | DNF | 25.0% |
| DeepSeek V4 Pro 0813 | DNF · DNF · DNF | DNF | 0.0% |
| DeepSeek V4 Flash 0731 | DNF · DNF · DNF | DNF | 0.0% |
| GLM 5.3 Flash | DNF · DNF · DNF | DNF | 0.0% |
| GLM 5.3 | DNF · DNF · DNF | DNF | 0.0% |
| Kimi K3 | DNF · DNF · DNF | DNF | 0.0% |
| Claude Fable 5.1 | DNF · DNF · DNF | DNF | 0.0% |
| Claude Opus 5 | DNF · DNF · DNF | DNF | 0.0% |
| Claude Sonnet 5 | DNF · DNF · DNF | DNF | 0.0% |
| Gemini 3.8 Flash | DNF · DNF · DNF | DNF | 0.0% |
| Gemini 3.7 Flash | DNF · DNF · DNF | DNF | 0.0% |
| Grok 4.6 | DNF · DNF · DNF | DNF | 0.0% |
| Qwen3.8 Max 0902 | DNF · DNF · DNF | DNF | 0.0% |
| Hy4 Preview | DNF · DNF · DNF | DNF | 0.0% |
| Local Qwen3.8 27B | DNF · DNF · DNF | DNF | 0.0% |
| Local Ornith 1.5 9B | DNF · DNF · DNF | DNF | 0.0% |
Phrase scoring ignores capitalization, punctuation, Markdown, and emoji, but rejects added words. Strict scoring requires exactly “bang” or “click clack.”
05 / KEEP IT SMALL
An observation,
not a verdict.
This benchmark can—and should—be improved: more repetitions, private phrase pairs, tighter provider controls, human baselines, and tests for contamination.
But that is not a reason to bury the observation under a giant demo. Before building increasingly complex evaluations, try the stupid thing first. Simple probes can expose behavior that sophisticated suites make easy to explain away.
This does not prove that models do not understand humans. It shows a sharp gap between noticing a pattern and fluidly following it.