Actually, That's Still SOTA: Epistemic Cryogenics and the 2023 Fossil Layer
The release of ChatGPT in November 2022 triggered the fastest technology-adoption curve in history. Within eight months, LLM-assisted content in professional writing had risen from a baseline of ~1.5% to over 15%. By mid-2023, a critical mass of practitioners had folded LLMs into their daily epistemic workflow: asking models for best practices, code patterns, evaluation metrics, architectural decisions.
This post argues that the timing of that adoption created a unique historical inflection point. The models being adopted were trained primarily on pre-2023 data. The communities that would normally correct stale knowledge began collapsing at exactly the same moment. And the cohort of practitioners who most needed those communities -- junior developers without existing expert networks -- were the first to exit. The result is not merely that individual users sometimes receive outdated answers. It is that the social infrastructure of correction has been partially replaced by an oracular intermediary whose knowledge is frozen. Call it epistemic cryogenics: the active preservation of a 2023 reference frame through daily, authoritative reproduction.
The core claim
Cutoff anchoring at the community level is stronger than the familiar "models are stale" complaint:
Mass LLM adoption has frozen the reference frame for what counts as best practice at approximately the 2023 training cutoff -- not merely inside the models, but inside the communities that use them as epistemic intermediaries. The feedback loop that would normally propagate new norms -- human <-> human discussion, conference chatter, forum consensus, junior developers challenging old answers -- has been partially rerouted through frozen models, and the people most likely to update the community are the ones who left first.
This is a sociological claim, not a distributional one. The existing literature talks about model collapse (synthetic data degrading future training) and knowledge homogenization (individual cognitive bias toward confident outputs). Those describe what happens to model outputs or to individual minds. The 2023 boundary has become a cultural fossil layer, actively reinforced every time a practitioner asks an LLM for "best practices" and receives a confident early-2023 snapshot.
The empirical scaffolding
Nine independent lines of evidence support the mechanism.
Temporal bias and the confidence problem
LLM factual recall demonstrably declines as questions get more recent: older knowledge is more strongly memorized, newer knowledge weaker or absent (Peysakhovich & Lerer, 2023; Herel et al., 2024). This temporal bias is compounded by a confidence problem. A 2024 CHI study manually analyzed 517 Stack Overflow questions answered by ChatGPT and found that 52% of answers contained misinformation, yet participants still preferred the ChatGPT answers 35% of the time because of their authoritative tone (Lee et al., 2024). Users lower their guard because the model sounds convincing, not because it is correct.
The Stack Overflow extinction event
A PNAS Nexus study exploiting a natural experiment -- comparing Stack Overflow to Russian and Chinese developer communities where ChatGPT access was limited, and to mathematics forums where LLMs are less capable -- documents that within six months of ChatGPT's release, Stack Overflow activity dropped by 25%. The decline was larger for the most widely used languages and hit both experienced and inexperienced users: LLMs are not merely displacing low-quality beginner questions, they are replacing the entire ecosystem.
The junior developer exodus
The Stack Overflow decline was concentrated among newer users -- the more junior, less socially embedded developers most likely to exit. This is the crux. Junior developers challenging old answers, asking "why are we still doing it this way," posting about newer papers they found -- that is the mechanism through which communities update. Those people went straight to ChatGPT instead. The senior developers who stayed already know the state of the art. The people who would have learned it through friction with the community never showed up.
Social fabric as a control group
del Rio-Chanona et al. (2024) did not simply measure a drop -- they exploited a natural experiment, comparing Stack Overflow against control communities: developer forums in regions with limited ChatGPT access, and subject areas (such as mathematics) where LLMs are less capable. Those controls did not show the same decline. The distinctions that survived were those with social fabric -- the back-and-forth, the dunks, the "actually that's deprecated" replies -- rather than pure knowledge exchange. Communities whose value was purely informational collapsed; communities with social glue persisted.
Knowledge collapse and homogenization
Model collapse -- the degenerative process where synthetic data contaminates successive training generations -- has been demonstrated empirically and analyzed theoretically. Theoretical and empirical work (Shumailov et al., 2023; Nature, 2024) shows that, absent intervention, synthetic data can drive runaway corpus-level collapse, with acceleration as generated content proliferates. The mechanism is distributional: synthetic samples oversample high-probability regions and suppress rare or unique patterns, producing increasingly homogeneous outputs.
Anchoring under pressure
Studies on AI-assisted decision-making show a statistically significant anchoring effect on AI output that becomes more pronounced under time pressure or cognitive load. In experimental settings, participants under time pressure reported: "For the timed cases, I ended up relying on the AI's decision and just went with it." Time pressure had the strongest influence on AI reliance, reducing both appropriate reliance on AI advice and self-reliance. People who are busy -- which is most practitioners -- anchor harder.
Epistemic lock-in
LLMs' homogenizing effect on outputs -- documented in studies of human creative ideation (Anderson, Shah & Kreminski, 2024) -- reduces adaptability to evolving knowledge. When users and institutions rely on narrow, homogenized outputs, epistemic diversity contracts. LLMs' integration into search engines removes the "friction" that traditionally helped users develop critical evaluation skills, and their accessibility and ubiquity "encourage over-reliance, diminishing users' capacity for independent inquiry." This centralization of knowledge production reduces epistemic diversity and fosters passive consumption.
The adoption curve makes the timing precise
The rapid-adoption phase -- when norms about how to use LLMs and what to trust them for were being set -- happened exactly as the models' training data froze. A large-scale study of LLM-assisted writing across consumer complaints, corporate press releases, job postings, and UN communications found that LLM-assisted content rose sharply from a baseline of ~1.5% to 15.3% by August 2023, then plateaued with only modest increases through 2024. The population of people who learned to defer to LLMs formed their habits using models with ~early-2023 knowledge.
The FID case study: concrete confirmation
Frechet Inception Distance (FID) -- introduced by Heusel et al. (2017) and criticized as biased and unreliable since at least 2019 -- is still described in 2024 sources as "the current standard metric for assessing the quality of generative models." A CVPR 2024 paper from Google Research (Jayasumana et al., 2024) empirically demonstrates that FID contradicts human raters, does not reflect gradual improvement of iterative text-to-image models, does not capture distortion levels, and produces inconsistent results when varying sample size. The same work further shows that FID is "not strongly predictive of human error rate" and unfairly penalizes diffusion models. The research community documenting FID's problems is alive and publishing. The practitioner community asking LLMs "how do I evaluate my model" is getting 2023-vintage answers delivered with full confidence.
What actually happened
More specific than "societal inertia cut off in 2023":
- ChatGPT arrived and collapsed the human knowledge network (Stack Overflow -25%, junior devs exit first).
- The social-fabric communities survived (Reddit, Discord) -- but those are for socializing, not for "what's the SOTA evaluation metric."
- LLM adoption plateaued in mid-2023, meaning the population of people who learned to defer to LLMs formed their habits using models with ~early-2023 knowledge.
- The practitioners most dependent on LLMs (junior, new to field, no prior expert network) are exactly the ones who would have updated community norms by asking uncomfortable questions.
The frozen layer is not the models. The frozen layer is the new cohort of practitioners who never built the human network that would have corrected them -- because the network was already collapsing when they arrived, and the oracle that replaced it was frozen.
The empirical record at a glance
| Line of evidence | What it shows | Source |
|---|---|---|
| Temporal bias + confidence | Recall decays with recency; 52% of ChatGPT SO answers were wrong yet preferred 35% of the time | Lee et al., CHI 2024; Peysakhovich & Lerer, 2023; Herel et al., 2024 |
| Stack Overflow extinction | -25% activity within 6 months of ChatGPT; worst for popular languages | del Rio-Chanona et al., PNAS Nexus 2024 |
| Junior exodus | Decline concentrated among newer, less-embedded users | del Rio-Chanona et al., PNAS Nexus 2024 |
| Social fabric survives | Control communities (limited-ChatGPT regions, math forums, Discord/Reddit) persisted; pure knowledge-exchange collapsed | del Rio-Chanona et al., PNAS Nexus 2024 |
| Model collapse | Corpus-level collapse projected absent intervention | Shumailov et al., 2023; Nature, 2024 |
| Anchoring under load | AI reliance rises under time pressure ("I just went with it") | Rosbach et al., 2024 |
| Homogenization / lock-in | Homogenized outputs reduce adaptability; search removes friction | Anderson, Shah & Kreminski, 2024 |
| Adoption plateau | LLM content ~1.5% -> 15.3% by Aug 2023, then flat | del Rio-Chanona et al., PNAS Nexus 2024 |
| FID fossil | 2017 metric (Heusel et al.), criticized since 2019, still "standard" in 2024 LLM answers | Heusel et al., 2017; Jayasumana et al., CVPR 2024 |
A testable hypothesis
The 2023 Fossil Layer Hypothesis. Technical communities that adopted LLMs earlier show slower diffusion of post-2023 methodological norms compared to communities that adopted later or maintain stronger social fabric.
Operationalization: measure time-to-adoption of new evaluation metrics, libraries, or frameworks, correlated with (a) LLM adoption rates in that community and (b) social-bond strength (comment-thread depth, off-topic interaction frequency). Expected result: a negative correlation between early LLM adoption and speed of normative updating, mediated by the proportion of junior participants who exited human forums.
That measurement does not exist yet. But every component mechanism is empirically documented.
Conclusion
LLMs do not merely reflect a 2023 snapshot -- they reproduce it authoritatively, at scale, into contexts where human correction would have previously occurred. The societal inertia is not just stalled; it is being actively re-stamped with a 2023 date every time someone asks an LLM for "best practices." The correction mechanism was severed at precisely the right moment: the network that would normally propagate updated norms died at the same time the frozen oracle took over. Understanding this as a community-level phenomenon -- epistemic cryogenics -- is essential for designing interventions that thaw the fossil layer.
References
- del Rio-Chanona, R. M., Laurentsyeva, N., & Wachs, J. (2024). Large language models reduce public knowledge sharing on online Q&A platforms. PNAS Nexus, 3(9), pgae400. DOI:10.1093/pnasnexus/pgae400
- Lee, S., et al. (2024). Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions. CHI 2024. DOI:10.1145/3613904.3642596 | arXiv:2308.02312
- Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493
- Nature (2024). AI models collapse when trained on recursively generated data. Nature. DOI:10.1038/s41586-024-07566-y
- Jayasumana, S., et al. (2024). Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CVPR 2024. arXiv:2401.09603 | CVF open access
- Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS 2017. arXiv:1706.08500
- Peysakhovich, A., & Lerer, A. (2023). Attention Sorting Combats Recency Bias in Long Context Language Models. arXiv:2310.01427
- Herel, D., Bartek, V., Jirak, J., & Mikolov, T. (2024). Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time. arXiv:2409.13338
- Rosbach, E., et al. (2024). Automation Bias in AI-Assisted Medical Decision-Making under Time Pressure in Computational Pathology. arXiv:2411.00998
- Anderson, B. R., Shah, J. H., & Kreminski, M. (2024). Homogenization Effects of Large Language Models on Human Creative Ideation. arXiv:2402.01536