Commodity Intelligence
The seductiveness of “general intelligence” is rooted in a costly category error
In a doh moment last week, I realized I was missing a key dynamic in my thinking about AI: commodification.
The specific problem was that vgr_zirp, the RAG bot I’ve been training and tuning on my older writing, was acting boringly omniscient and tasteless, engaging deeply on topics I know nothing about, and more importantly, don’t care about. Conversations the real me would walk away from were playing out in dull ways. Claude Sonnet’s far greater knowledge and far larger circle of care (the union of all human cares ever rendered textually) were seeping in too much. I had to add filters and guardrails modeled on my own ignorance, indifference, and blindspot areas to get it to behave more interestingly and tastefully, and not sully my good name.
Too much commodity intelligence and indiscriminate caring were seeping into what I’m trying to design to be a differentiated and opinionated intelligence with a real-person personality (a stylized version of my own).
A lot of people, myself included have noted that LLMs offer a homogenized kind of intelligence that resembles index funds (see my LLMs as Index Funds, April 1, 2025, for one version of this argument). This view, I’m now convinced, does not go far enough. In advanced, innovation-based economies, index funds are collections of high-market-cap stocks that are still individually pretty differentiated and far from the commodity asymptote all economic goods and services tend towards. LLMs are much farther along the curve. The capabilities they manifest rest on vast corpuses of data that are not just public and with the equivalent of “high market cap,” but largely commodified. LLMs are not just index funds, they are dominantly commodity index funds.
LLMs are the informational equivalent of portfolios of coal, gold, and potatoes. The components may differ in intrinsic value and exist in varied quality grades, but are fundamentally fungible. Information embodied in LLMs is mostly high-paradigm and high-consensus common knowledge. LLMs know about fringe, crackpot, and low-consensus ideas in the same way markets know about emerging and penny stocks and junk bonds, but the center of gravity (or indexical perspective if you like) of both lies in commodified knowledge.
What is the informational equivalent of commodification? I pointed out one aspect of the answer 3 years ago, and dubbed it the Labatut-Lovecraft-Ballard (LBB) arc, inspired by reading Benjamin Labatut’s When We Cease to Undersrand the World, and the fiction of H. P. Lovecraft and J. G. Ballard.
In Disturbed Realities (Jan 20, 2023), I described the LBB arc as follows:
We might sketch a three-stage psychohistory of a disturbing new expanded reality, as more and more minds become stretched to accommodate it:
In the first, Labatutian stage, a handful of minds are forced to bear the brunt of the full, uncontrolled assault of a new idea on the human psyche.
In the second, Lovecraftian stage, a much larger group of somewhat inoculated minds willingly ventures forth to encounter a somewhat familiar, but still unsettling version of the idea, serving as an avant garde engaged in rebuilding social realities as required around it.
In the third, Ballardian stage, the construction of new social realities is (relatively) complete, but the costs and inherent contradictions have not yet been apprehended. The expanded reality has been civilized but not tamed. All minds are shaped by it, whether or not they are consciously equipped for it.
Benjamin Labatut’s book (one of the best of this century so far) explores the insanity-inducing effects of new-to-humanity knowledge, on the first minds that encounter it, via a series of quasi-fictional accounts of such encounters in the lives of famous scientists. My model is basically an account of how the human mind adapts fully and collectively, primarily through socialization. The larger the number of people who have experienced a piece of knowledge, the more domesticated it is, and the less able to cause madness. Labatutian psychosis leads to Lovecraftian cosmic horror leads to Ballardian banality.
In a talk shortly after that post, I argued that this partly explained crazed reactions to AI (remember Blake Lemoine?), but I didn’t complete the theory. Commodification effects complete the theory, but the mechanism is subtler than I anticipated at the time.
It is important to note that commodification is not the same as universal accessibility. Gold is a commodity, but most people in the world possess little to none. Classical mechanics is a fully commodified body of knowledge, but only a small fraction of humanity has the aptitude and educational preparation to understand and use it to the fullest extent widely available textbooks can teach. To the rest it can be the source of magic (eg. a double cone rolling “uphill” on a pair of slanted, diverging sticks).
The OpenAI proof of an 80-year-old math problem may have been beyond human mathematicians, but it rested on fully digested Ballardian priors, so to speak. The Labatutian era for that problem was circa 1946 when Erdos first posed it to himself and understood its significance. Human mathematicians have annealed it over 80 years into a familiar bit of mathematical territory, at least to mathematicians in the relevant subfields.
AIs trained on Labatutian data are highly differentiated, fragile, and unreliable. AIs trained on Ballardian data are highly commodified, robust, and reliable. To extend the analogy past AI to my favorite neck of the woods, protocolized knowledge has entered the utility stage past commodification, and is generally embodied by the “tool use” part of agentic AI. A very clear tell is that it runs on CPUs rather than GPUs.
To understand why it is a valid step to go from speaking of commodified knowledge to commodity intelligence, you have to understand a few features of AI of the sort we have today that justify such extrapolation:
Performance degrades outside the training set (though the training set is larger than the experiential base of many humans, so finding the actual boundaries, rather than simple errors or hallucinations, can be hard)
Performance degrades with time past the training epoch (a necessary consequence of what Emmett noted as the “overfitting without regularization” of constantly evolving internet data, which is a feature, not a bug)
Performance degrades if you try to train a model on its own output without additional new raw information entering the loop (“model collapse”)
These is reasonable phenomenology by the way, and visible in human intellligence too, despite the differences in architecture. We would be very surprised, like “is there phlogiston in there?” level surprise, if these phenomena didn’t manifest. They provide reassurance that AI does not appear to violate the known principles of information theory or thermodynamics. Megawatts worth of matrix multiplications don’t produce phlogiston in datacenters.
We don’t have a theory of how LLM-and-human style intelligence works, but we have strong evidence that there is no magic going on. The emergent phenomenology is like markets or weather, not theology.
A few things tend to confuse people into believing in magical properties:
Unexpected playability of domains. Many knowledge domains are turning out to be what I have started thinking of as unexpectedly playable (stronger subset: self-playable). Though a domain may not be technically a closed world like chess, and though there may be no obvious “physics” to it, capable of being abstracted into a “physics engine,” there is enough rule-like regularity that you can get farther with seemingly informationally impoverished data than you think. Code and protein folding are prototypes but more impressive examples are emerging. For example, recovering 3d geometry from 2d projection data (like photographs) is “unexpected playability of large corpuses of photos.” Egocentric video for training robots is another example. The various symmetries of many artificial and natural objects allows this.
Local entropy reduction. Agentic AI is exceptionally good at cleaning up messy local conditions and getting them into locally well-ordered states that are beyond normal human capabilities. This can seem magically negentropic, but is still local. Claude Code cleaning up your decades of downloads into a nicely organized library still requires wattage being expended entropically in a datacenter somewhere, mostly likely the backyards of people you don’t deal with socially.
New-for-you (secondary Labatutian) effects: This is the subtlety I was mentioning earlier. Normal knowledge commodification curves are limited by human aptitude and the patience of human teachers. So human physicists who understand advanced physics don’t have patience for humans who lack the aptitude to (say) earn a physics degree. They ignore crackpots. But an AI embodies commodified physics knowledge in a form that expands access to people previously priced of that commodified knowledge market. For these newly empowered people, a counterparty who engages with them triggers something similar to a Labatutian paranoia. The knowledge is not new, but they get it via a raw encounter rather being socialized into its Ballardian form, and embark on a solo LBB arc in a solipsistic reality tunnel.
Once you account for such wrinkles and clear away the red herrings created by worshippers, the idea that AIs today are commodity intelligences becomes intelligible and useful.
It also explains, at least to my satisfaction, the strange allure of the idea of “general” intelligence despite the obviously specific, training-context-adapted and contingent nature of all known biological and artificial intelligences. It’s the result of confusing two notions of “generality.” Generality as in “generally available in the market” is not the same as generality in the sense of totalizing universality.
Commodified knowledge is “general knowledge” in the sense tested by trivia/quiz contests. In grade school, we actually had a subject on the curriculum called “GK” and kids good at it (I was one of them) got put on quiz teams to represent their class or school. General intelligence of the sort we actually have today is simply AIs trained on general (ie commodified) knowledge.
But the theological motte-and-bailey move that conflates it with some totalizing-universal divine-omniscience idea of “Artificial General Intelligence” traps a great many of even the smartest people. A category error motivated by theological yearnings, validated by second-order Labatutian psychoses, sustained by epistemic bubbles, and encouraged by sketchy business roadmaps that need a story to justify trillion-dollar investments.
This widespread category error has consequences beyond the annoyance of the future getting hamstrung by getting “AGI” branded. My simple example of a bot being rendered boring by the seepage of commodity intelligence is a small example. A general intelligence in the strong sense could only have improved the bot (a God making the bot a more fully realized ideal version of me say). It would not have injected boring tastelessness.
There are bigger, costlier mistakes you can make if you pretend commodity intelligence deployed at scale is the same thing convergence towards divine omniscience.
The biggest mistake is perhaps this: Instead of marveling at and exploiting the capabilities of the truly amazing AIs we have built, you end up worrying about the features and flaws of incoherent and ill-posed thought experiments that simply don’t matter.