Fooled by Non-Randomness

Fooled by Non-Randomness
(why LLMs won’t be a universal revolution)

Late 2007, Alan and Nick, the CEO and Head of Risk of a successful investment bank, get in the lift to leave the Lehman Brothers tower in New York. “Those guys are on drugs, Nick, cut their credit”. One of the last things the CEO of said bank did before retiring was to forecast Lehman’s bankruptcy and move all the clearing lines and exposure to Deutsche Bank.

It wasn’t anything in Lehman’s numbers that caused Alan to lose his patience. Their Value-at-Risk model was rosy, after all. It assumed tomorrow would look like yesterday and that prices would continue to distribute log-normally. Their quarterly filings also looked fine. On the surface, that is, and without delving behind the “Repo 105” trades that hid up to $50 billion of debt. Rather, it was in Richard Fuld’s face, and it was there in his personalised initials on his pristine shirt cuffs, and in the way he had kept them waiting, unnecessarily so. On his desk there was also a pitch deck for what seemed like a flimsy commercial real estate deal in Florida. All things considered, it was in all “the stuff” not in the data.

What made Jim Carrey’s character, Truman Burbank, realise he was in a simulation (the “Truman Show”) was the human condition, the randomness of life, the enormity of the unknown, and man’s search for meaning. Again, “the stuff” not in the data.

Truman wanted love. He wanted friendship. He wanted therapy – for his guilt-ridden fear of water. He needed comfort. He had hallucinated his drowned father’s face on a stranger. He also needed adventure. He had grown restless. As a boy he had dreamed of Fiji, and however hard the people around him tried to talk him out of that, the longing lived on in his deep heart’s core. Given all these perturbations, Seahaven’s non-randomness became stark.[1]

An emotion-charged Truman tested the system and set about on a Hero’s journey.

“Yet all experience is an arch wherethrough
Gleams that untravelled world whose margin fades
For ever and for ever when I move.”

Alfred, Lord Tennyson, Ulysses (1842)

A large language model is like a scene from the Truman Show. A world has been constructed from a corpus into word tokens: self-contained, coherent, rehearsed. A world in which scenes can be inferred and acted out. A world that has painted backdrops all around it to make it appear far larger than what it really is. Sometimes, though, it is like the rain that falls only on Truman: staged on cue, just for him. Only when viewed within constraints, does the performance (inference) appear remarkable.

Ask an LLM to draft a contract, summarise a filing or explain an audit opinion, and it performs like Truman’s neighbours, perfectly on cue. The LLM has played out each scene a million times. Every next move is a Bayesian bet, and inside the dome the odds are excellent. Code, cyber and maths are also where the LLM performs. Again, these are closed worlds with hard rules and instant feedback: a program compiles or it doesn’t, a proof checks or it doesn’t, an equation holds or it doesn’t. Models can be run, then run again, and then run some more; in almost infinite loops, bashing through every path until one works. Pattern-matching by brute force. Now, the time that these loops run in is unfathomably fast for us. This speed alone makes the solution-finding appear as natural intelligence.

The human brain has something like 100 trillion synapses – far more connections than any model has parameters. It also only runs on about 20 watts – arguably a little less in Queensland but certainly orders of magnitude less than what Coreweave requires. We still don’t really know how the brain works. We have mapped the wiring of a fruit fly looking for a lunch, but we cannot yet explain a human holding a hunch. More importantly, the brain doesn’t just fathom; it wants. And, which is more, it seeks to cohere with other brains around these wants, severally and jointly held, severally and jointly evolving. The brain is tethered to the human condition: love, grief, hope fear, pride, and all those things that, when in excess, are described as the “Deadly Sins”. It was the human condition that sent Truman to the edge of the dome. No amount of compute would have ever devised a statistical rationale to even set sail.

“It’s easier to fool people than to convince them that they have been fooled.”

Mark Twain (maybe, though on this attribution we could have been fooled)

The shortcoming with LLMs is that the model does not know where the dome ends. Truman only found the wall by hitting it. A model trained on what has been written cannot tell us about what has not yet happened: a drone swarm flying a new pattern, a new type of grid fault, an adversary who goes around the playbook. In the open world, the events that matter most are the ones least represented in the data.

The Frontier Labs already know this about their LLMs. They know that selling raw tokens to everyone is a race to the bottom.[2] So they are moving up the stack, into “harnesses” (the agents, tools, memory and guardrails wrapped around a model) and into domain-specific knowledge sets where the value sits in the workflow rather than the next word. We called this out in July. In our “Revenge of the Nerds” edition of Around the Corner, we argued that Frontier LLMs would soon eke out only marginal gains versus open source, and that the real ground would be won in a specialised layer of proprietary weights, data ontologies, and new maths. Three months on, token prices have more than halved, and the labs are racing to build exactly that layer. Nowhere is that clearer than in biotech. In September, Reuters reported that Anthropic had quietly set up its own biology lab as it ramps up an in-house AI drug-discovery program.[3] On 20 October, OpenAI bought Orchestra Bio and folded it into its life-sciences toolkit.[4] New IP is not discovered with tokens and data: the bottleneck has always been in the lab, where a hypothesis has to survive contact with the physical world.[5]

Alex Karp, the OG of “I am tech bro and no-one gets to surf in Fiji”, has been at this game for years. He strategically “helped” businesses only if they enabled Palantir to develop an ever-changing ontology of the world, a sort of universal digital twin. He would then leverage that back to win new customers in the same fields. So, for example, he’d solicit business for construction companies with “unique” insights on projects and hedging: in the future construction materials A is likely to see a shortage due to X, Y, and Z. It was AI’s first form of recursiveness. That’s why he’s become a little bit shitty, lately.

What has made all the non-surfing CEOs of the frontier labs so nervous is the open-source models, and the tacit acknowledgement among the tech bros that the maths in an LLM isn’t enough, by itself, to sustain a business moat. Nervous people reach for big stories, like surfers paddling for the biggest wave on the horizon. We have seen this movie before with Level 5 autonomous driving: Musk has repeatedly predicted Level 4/5 autonomy as “something within the next 2 years” … for over 13 years now.

While legislatures need to be alarmed[6], we all need to keep some investment perspective: swarm of thousands of agents (that cost millions of dollars) brute-forcing their way through a corporate firewall is not proof that we are on a short hop to AGI.[7] We are conflating very different things: LLMs that respond quickly, hacking experiments that grind through the closed world of code, autonomous machines working in fenced-off environments. Each is impressive inside its own dome.

Kept inside their dome, with a human in the loop, LLMs make good people faster; they just won’t replace them.[8]

What agent swarms do change is security. Most enterprises currently only spend 4–10% of their IT budget on security; CIOs are currently rewarded for keeping costs low and systems up, and they used to spend their days recruiting rockstars into “Dev”. That’s all going to change, but in unimagined, probably positive, ways. Future CEOs won’t all just be lawyers or accountants. Quite a few will be promoted CIOs. Can you just imagine the Christmas parties?

Outside the dome is the open world: physical systems, contested environments, places where conditions change faster than any trained dataset can follow. Here, sleuthing is not enough. It takes machine-learning orchestration: the older, quieter discipline of sensing, data fusion, control and decision-making that runs factory floors, flies things, and balances grids. We need action on “the stuff” that no model has ever seen before, often at the edge and often in real time. Within the stuff there is different data, different states (like quantum states), different maths and different business models. They may never even run on the same chips or in the same neo-clouds: a transformer predicting tokens in a data centre probably has little in common with a control loop on an edge device that cannot phone home.

This is where we think the durable value sits. Measuring the world as it is and as it evolves, not as it once was described, and software that knows when it has reached the edge of its own knowledge. Defence, critical infrastructure and advanced manufacturing all need this.

Good morning, and in case I don’t see ya, good afternoon, good evening and good night.