The Fourth
Axis: Most of the brain goes on arXiv. The notebook goes to the patent office.
Depth, tokens, time. Three solved. The fourth is where the money went.
The filing
Record · [08] [01]
OpenAI holds 69 published patent families. Forty-nine were filed by OpenAI itself, the earliest with a priority date of July 2022; the other twenty were acquired. A patent analyst who read all 69 found the portfolio protects products, not the models themselves. The exceptions are few: image-model training, the Whisper architecture, an embedding pretraining method.
What the company patents, again and again, is how a product keeps state. A granted patent for stateful custom GPTs. A published application for ChatGPT Memory. Granted patents for prompt caching and server-side conversation state. Most of the brain goes on arXiv. The notebook goes to the patent office.
Position
Every architectural breakthrough of the last decade gave information a place to persist.
The identity path let it persist across depth. Registers let it persist across tokens. State-space memory let it persist across time. Nobody has given it a way to persist across runs.
Three axes solved. The fourth is unsolved, and it is where the money went.
- Depth
- Identity path, 2015
- Tokens
- Registers and sinks, 2023
- Time
- State-space hybrids, 2025
- Runs
- Open
Across depth
Measured · [10] [16]
Plain networks got worse as they got deeper: a 34-layer plain net trained to higher error than an 18-layer one. The identity path fixed it in 2015: Highway Networks in May, ResNet in December at 152 layers, a 1001-layer pre-activation ResNet on CIFAR the next year. The reason it worked was stranger than the fix.
Veit and colleagues showed a residual net behaves as an ensemble of short paths. Delete a block and half the paths survive. In a 110-layer network almost all of the gradient comes from paths ten to thirty-four layers deep. The long paths contribute almost nothing.
Delete a block and half the paths survive. In a 110-layer net most of the gradient comes from paths 10 to 34 layers deep.
Measured · [06] [07] [31]
The identity path then kept its form for nearly a decade, in DeepSeek's phrase [07]. It is being rewired now. Hyper-connections widened the stream into n parallel streams with a learned mixing matrix [06]. DeepSeek's manifold-constrained version pins that matrix to a doubly stochastic form, which tamed the loss surge and gradient-norm instability the first version showed at 27B scale [07]. Value residuals send first-layer values to every layer: the same loss with 16 percent fewer parameters. A variant that shares the first layer's values outright halves the KV cache, at a small cost in loss [31].
One stream
Identity path, 2015 to 2024. x + F(x). Nearly a decade in its original form.
n streams
Hyper-connections, 2024. mHC, 2025 [06] [07]. Every block reads, writes and mixes all streams. Schematic: the mixing matrix is learned and input-dependent.
Position
The one wire that kept its form for a decade is now a switchboard. Depth is no longer a line. It is a bus.
Depth as time
Measured · [09] [32] [33] [34]
The second rewiring treats depth as a dial. A looped transformer runs one block L times over the same tokens with the same weights. MIT's Hyperloop adds a hyper-connection per pass and matches or beats depth-matched baselines, and the stabilized mHC baseline, in its main runs at 240M, 1B and 2B scale, with roughly half the parameters. A wave of looped and recurrent-depth papers landed between April and June 2026 on the same idea: latent reasoning, fixed-point reasoners, looped world models [32] [33] [34].
Hyperloop: a looped middle block with per-pass hyper-connections. Half the parameters of a depth-matched baseline.
Position
If the ensemble paper was right, depth was never the thing. Paths were. A loop is a way to buy paths without buying layers.
The practical result is that a model can think longer on a hard token and shorter on an easy one. Earlier looped models already set the count per token [09]. Depth has a knob.
- ~50%
- Fewer parameters, Hyperloop vs depth-matched [09]
- 16%
- Fewer parameters at the same loss, value residuals [31]
- 9
- Years before the identity path was widened [06]
Across tokens
Measured · [11]
Large vision transformers were caught repurposing their own input. About two percent of patch tokens, mostly over blank background, carried norms roughly ten times normal and held little of their own pixels. Trained as classifiers, those stolen patches named the car. The model needed somewhere to hold the whole image and took the least useful tokens it had. Learned register tokens, discarded at output, made the artifact vanish; one was enough, and the authors used four.
A background patch that dropped its own pixels can name the car. It was holding the whole image.
Measured · [12] [13] [14] [15]
Language models do the same thing with the first token: attention sinks [12]. In 2025 OpenAI designed the sink in rather than leaving the model to improvise one, a learned per-head logit with no value row [13]. In 2026 MiniMax tested the same trick: it took over the sink in some heads but not others, gave no consistent gain, and was left out of the final model [14]. A mechanistic account of sinks in GPT-2 traced them to one circuit of three parts, each necessary in GPT-2, and each missing from some other model that still has a sink [15].
- 2023
- Artifact. StreamingLLM: the first token is a sink [12]
- 2023
- Diagnosis. Vision transformers need registers [11]
- 2025
- Designed in. gpt-oss: learned sink logits [13]
- 2026
- Dropped. MiniMax tests it, leaves it out [14]
- 2026
- One circuit. Not the only one [15]
Position
A model with nowhere to put a thought will steal a token. Give it one on purpose. Then check that it took it.
Across time
Measured · [17] [18] [19]
Persistence across time shipped as a ratio. Qwen3-Next runs 48 layers at three to one: 36 linear-attention layers carrying a delta-rule state, 12 full attention [17]. Kimi Linear cuts the KV cache by up to 75 percent against full attention, and at a million tokens the freed memory buys up to six times the decoding throughput [18]. Mamba-3 gives the state complex values, and parity, a state-tracking test where Mamba-2 sat near random, goes to about 100 percent; at 1.5B its MIMO variant scores 1.8 points of average accuracy above Gated DeltaNet, the next best model; the base variant, 0.6 [19].
- 36
- Gated DeltaNet, linear
- 12
- Full attention
Parity, a state-tracking test [19]: Mamba-2 near random. Mamba-3, complex-valued state: about 100 percent.
- 75%
- Smaller KV cache at most, Kimi Linear [18]
- 6x
- Decode throughput at most, 1M tokens [18]
- +1.8
- Points, Mamba-3 MIMO at 1.5B [19]
Measured · [20]
One lab walked it back. MiniMax-M2 runs full attention in every layer. Its technical report says no efficient-attention variant reliably matched full attention in production on reasoning, coding and agent tasks.
Position
Time persistence is solved for the benchmark and contested for the agent. The state is cheap to carry and expensive to trust.
They patented
the notepad
Across runs
Position
Weights are a photograph taken the day training stopped. Everything after is read off a caption. The fourth axis is whether the photograph can change without tearing.
Measured · [21] to [27], [35]
The attempts are serious and all partial. Titans adds a neural memory that learns at test time, gated by surprise; an independent reproduction found the memory helps but the full model does not always win [21] [22]. Google's HOPE treats its own weights as writable memory with blocks updating at different frequencies, with no public code and no forgetting curve: its continual-learning results are end-of-sequence accuracies, not retention across repeated updates [23]. SEAL lets a model write its own training data and apply it: on an ARC subset, 0 percent in context, 20 percent with untrained self-edits, 72.5 percent with SEAL. Repeat the edits and it forgets [24]. Retrievable Gradients store a gradient per document, apply it, then revert, so nothing drifts and nothing stays [26].
| Attempt | Writes weights | Survives the run | The catch |
|---|---|---|---|
| Titans [21] | Fast weights | Within context | Reproduction contested |
| HOPE [23] | Yes, at inference | Claimed | No release, no curve |
| SEAL [24] | Yes, by self-edit | Yes | Forgets on repeat |
| TTT, next token [25] | Fast weights | Within context | Resets |
| Retrievable Gradients [26] | Apply, then revert | As stored gradients | Not in the weights |
| Memory Layers [27] [35] | At training; sparse updates after deployment in a 2025 follow-up | Yes, in the follow-up | Still forgets: an 11 percent drop |
Position
Every row is partial. The partial is always in the same column. Nobody has a weight that learns on Tuesday and remembers Monday.
What shipped
Record · [01] to [05]
Persistence across runs shipped. It shipped around the weights, not in them. OpenAI's application for what reads as ChatGPT Memory describes a human-readable personalization notepad capped at a set number of notes; the model may be trained by reinforcement learning, or use heuristics, to decide what is saved, and an offline pass rewrites similar notes into one [02]. Prompt caching hashes a prompt's prefix so a repeat request lands on the same server and reuses its computed tokens, for minutes to an hour [03]. The Responses API keeps conversation state server-side and resumes from an identifier [04]. A stateful-GPT patent has a custom GPT change state mid-conversation, rewriting its system prompt when a user's message calls for it [01]. Groq's granted patent gives its tensor streaming processors a deterministic, globally addressed memory whose reads and writes the compiler schedules; priority 2021, this continuation filed 2024 [05].
| Layer | What lives there |
|---|---|
| Weights | Nothing shipped here |
| Fast weights | Research only. Titans, TTT, HOPE |
| Cache | Prompt caching patent [03]. NVIDIA context memory tier [29] |
| Notepad | ChatGPT Memory application: a human-readable notepad, capped [02] |
| Server state | Stateful GPT patent [01]. Responses API [04] |
| Silicon | Groq deterministic memory, compiler-scheduled [05] |
Where persistence across runs actually lives, October 2026. Deeper in the stack is lower on the list.
Measured · [28] [29]
The software mirror: a 2026 preprint argues that for streaming workloads, session state kept across requests is not an optimization but a requirement for constant query latency [28]. At CES 2026 NVIDIA announced a context-memory storage tier for KV cache that it says extends AI agents' long-term memory; its technical blog puts it on NVMe flash, and NVIDIA has a published application on moving multi-turn KV cache between memory tiers down to SSD [29].
Position
Nothing shipped in the weights. Everything shipped around them. That is the market telling you where the unsolved problem is.
The map
Position
What a lab publishes is what it is proud of. What it patents is what it sells.
Published. No patent found.
- Titans
- Google, 2024
- Memory Layers at Scale
- Meta, 2024
- Hyper-connections
- ByteDance, 2024
- Attention sinks
- MIT, Meta and others, 2023
- Registers
- Meta and Inria, 2023
- mHC
- DeepSeek, 2025
Patented. No paper.
- Stateful GPTs
- OpenAI, US12400074B1
- Memory
- OpenAI, US20250200361A1
- Prompt caching
- OpenAI, US12596764B1
- Responses API
- OpenAI, US12591766B1
- Deterministic memory
- Groq, US12561279B2
What a lab publishes is what it is proud of. What it patents is what it sells. [01] to [05], [08]
- 6
- Architectures published, no patent found
- 5
- Memory and state patents, no paper
- 18
- Months from priority to publication
Record · [30] [36]
Outside the frontier labs, papers get patented too. A Korean university patented the continual-learning method it had already posted to arXiv: prompts summed over time on a frozen backbone, filed in the US in September 2025, published April 2026. Reading the patent tells you what the university thinks is worth owning; reading arXiv would not.
Measured
No patent was located for Titans, Memory Layers, hyper-connections, attention sinks, registers or mHC. Applications publish eighteen months after their earliest priority date, unless the applicant asks for earlier publication, or for none. A Titans filing made before the paper posted would have published by mid-2026, and none has been located. One filed later in the US grace year, or under a non-publication request, could still be dark. Absence of a record is not absence of a filing.
Position
Read the filings. They are the only place a lab tells you, on the record, what it thinks is worth owning.
The clock
Record and measured
Three clocks run at different speeds. Papers post within weeks of a result. Filings surface eighteen months after priority. Products ship when the business is ready, which is sometimes before the paper exists.
| Year | Papers | Patents | Products |
|---|---|---|---|
| 2021 | Groq priority | ||
| 2022 | OpenAI earliest priority. Groq first filed | ||
| 2023 | Sinks, registers | Memory priority, December | |
| 2024 | TTT layers. Hyper-connections. Memory layers. Titans, 31 December | Groq continuation filed | ChatGPT Memory, test in February, every tier by September |
| 2025 | Chung Ang paper. HOPE | Memory published, June. Chung Ang filed | gpt-oss sinks |
| 2026 | Mamba-3. Hyperloop. TTT with next-token prediction | Chung Ang published | NVIDIA context memory. Qwen3.5 hybrid |
A memory feature reached users before Titans was posted, and its filing was public before HOPE. The notepad beat that brain to market.
- Feb 2024
- ChatGPT Memory, first test group
- Dec 2024
- Titans posted, 31 December [21]
- Jun 2025
- Memory filing published [02]
Position
ChatGPT Memory went to a test group in February 2024 and to every tier that September. Titans, Google's test-time memory, was posted on 31 December 2024, and test-time training goes back to 2019 [37] [38]. The notepad reached users before Titans did, and the gap has not closed.
If you want to know what persistence will look like in 2028, the filings of 2026 are the only forward-looking document that exists.
Where it is wrong
06 · The case against
- 0
- Production models learning across runs in the weights
- 4
- Serious partial answers: HOPE, SEAL, Retrievable Gradients, sparse memory finetuning
- 1
- Flat forgetting curve would end the argument
Sources
Primary sources · read in October 2026
- 01OpenAI. Stateful pretrained transformers in a generative response engine. US12400074B1, granted. USPTO record.
- 02OpenAI. Selective learning of information for the generation of personalized responses by a generative response engine. US20250200361A1, published 19 Jun 2025, pending / WO2025128397A1. Google Patents record.
- 03OpenAI. Prompt caching in generative response engines. US12596764B1, granted. Google Patents record.
- 04OpenAI. Application programming interface with generative response engine state management (the Responses API). US12591766B1, granted. Google Patents record.
- 05Abts, Groq Inc. Deterministic memory for tensor streaming processors. US12561279B2, a continuation filed 3 Jun 2024, priority 7 Jul 2021. USPTO record.
- 06Zhu et al., ByteDance. Hyper-connections. arXiv:2409.19606, ICLR 2025.
- 07Xie et al., DeepSeek-AI. mHC: manifold-constrained hyper-connections. arXiv:2512.24880, 31 Dec 2025.
- 08PatentVest. What is OpenAI actually patenting? Analysis of all 69 published families, 23 Jul 2026. Exceptions on models: US20240370779A1, US20260017861A1, US20260073164A1.
- 09Zeitoun, Torroba-Hennigen, Kim, MIT. Hyperloop transformers. arXiv:2604.21254, Apr 2026.
- 10Srivastava, Greff, Schmidhuber. Highway networks, May 2015. He et al. Deep residual learning, Dec 2015. He et al. Identity mappings in deep residual networks, 2016.
- 11Darcet, Oquab, Mairal, Bojanowski, Meta. Vision transformers need registers. arXiv:2309.16588, ICLR 2024.
- 12Xiao et al. Efficient streaming language models with attention sinks. arXiv:2309.17453, 2023; ICLR 2024.
- 13OpenAI. gpt-oss-120b and gpt-oss-20b model card, 5 Aug 2025, arXiv:2508.10925: learned per-head sink logits.
- 14MiniMax. MiniMax sparse attention. arXiv:2606.13392, 2026.
- 15Ran-Milo, Ofek, Mendel, Tel Aviv University. A mechanistic account of attention sinks in GPT-2: one circuit, broader implications for mitigation. arXiv:2604.14722, Apr 2026.
- 16Veit, Wilber, Belongie. Residual networks behave like ensembles of relatively shallow networks. NeurIPS 2016.
- 17Qwen team, Alibaba. Qwen3-Next-80B-A3B model card and blog, Sep 2025; Qwen3.5, blog and model card, Feb 2026.
- 18Moonshot AI. Kimi Linear. arXiv:2510.26692, 2025.
- 19Mamba-3: improved sequence modeling using state space principles. arXiv:2603.15569, ICLR 2026.
- 20MiniMax. MiniMax-M2 technical report. arXiv:2605.26494, 2026.
- 21Behrouz, Zhong, Mirrokni, Google. Titans: learning to memorize at test time. arXiv:2501.00663, posted 31 Dec 2024.
- 22Titans revisited: a lightweight reimplementation and critical analysis. arXiv:2510.09551, 2025.
- 23Behrouz, Razaviyayn, Zhong, Mirrokni, Google Research. Nested learning: the illusion of deep learning architectures (HOPE). NeurIPS 2025; arXiv:2512.24695, Dec 2025; blog 7 Nov 2025.
- 24Zweiger et al., MIT. Self-adapting language models (SEAL). arXiv:2506.10943, 2025.
- 25Test-time training with next-token prediction. arXiv:2606.21803, 2026.
- 26Retrievable gradients: continual post-training without cumulative weight drift. arXiv:2606.15734, 2026.
- 27Berges et al., Meta. Memory layers at scale. arXiv:2412.09764, Dec 2024.
- 28Attention once is all you need: efficient streaming inference with stateful transformers. arXiv:2605.13784, 2026.
- 29NVIDIA. BlueField-4 powers Inference Context Memory Storage Platform, press release, 5 Jan 2026; technical blog, 16 Mar 2026. Related filing: Dynamic key value pair cache scheduling, US20260133908A1, published 14 May 2026.
- 30Kim, Jin, Chung Ang University. Continual learning method and device for multimodality data. US20260093780A1, filed 29 Sep 2025, published 2 Apr 2026. Google Patents record.
- 31Zhou et al. Value residual learning (ResFormer, SVFormer). arXiv:2410.17897, ACL 2025.
- 32Stabilizing recurrent dynamics for test-time scalable latent reasoning in looped language models. arXiv:2605.26733, May 2026.
- 33Movahedi et al. Fixed-point reasoners: stable and adaptive deep looped transformers. arXiv:2606.18206, Jun 2026.
- 34Lu et al. Looped world models. arXiv:2606.18208, Jun 2026.
- 35Lin, Zettlemoyer, Ghosh, Yih, Markosyan, Berges, Oğuz, Meta. Continual learning via sparse memory finetuning. arXiv:2510.15103, Oct 2025.
- 36Jin, Kim, Chung-Ang University. Continual learning for multiple modalities. arXiv:2503.08064, Mar 2025.
- 37Sun et al. Test-time training with self-supervision for generalization under distribution shifts. arXiv:1909.13231, 2019.
- 38Sun et al. Learning to (learn at test time): RNNs with expressive hidden states (TTT layers). arXiv:2407.04620, Jul 2024.
Colophon
Set in Bricolage Grotesque and Martian Mono. The cover rose is computed from r = cos(kθ), frozen at k = 21/4, 37/5 and 19/6: a still from a film. Preprints dated 2026 are recent; open them before citing. Patent numbers from a patent analyst should be opened at the office record before they are repeated.
The axis
A decade of architecture can be read as one project with three finished parts. The identity path taught a network to carry a signal through a hundred layers without losing it. Registers taught it to hold a thought that belongs to no token. State-space memory taught it to carry that thought across a million positions.
Then the run ends, the state is discarded, and the next run starts from the photograph.
Three axes are architecture. The fourth is still a product feature. Until a weight can learn on Tuesday and keep Monday, that is the whole business.
The fourth is still a product feature.