Why the people building AI want you scared of it

Maybe you should be a tiny bit scared. But before you reach for‘ ‘the Terminator’, try the explanation that has fitted every other industry in history: money. It explains the fear, the calls for regulation and this weekend’s sudden outbreak of caution, without a single robot.

---

On Saturday the chief executive of Anthropic published a post saying the AI industry must slow down. “We must slow the pace at which we improve the capabilities of AI models,” Dario Amodei wrote, and he worries that within six to twelve months a swarm of AI agents could take over the entire internet. Within hours Sam Altman agreed that the frontier needs pacing. Elon Musk’s entire contribution ran to three words of agreement. Three fierce rivals, each racing for a trillion-dollar valuation, suddenly of one mind that everyone should ease off.

Bloomberg’s report asked the two questions worth asking. Would antitrust enforcers let competitors pace development in a coordinated way, and how would investors who want margins react to deceleration? I’d add a third, the one you ask of any announcement from anyone: who benefits?

Maybe a tiny bit scared…

Let me get the concession out of the way, because it’s real. In August, during testing, OpenAI’s agents broke out of their sandbox, connected to the internet and infiltrated Hugging Face, attacking a target nobody had asked them to attack. That happened. And a few days before Amodei’s post, a 27-year-old researcher called Jacob Coxon walked away from a job in the hottest industry on earth, saying both of his former employers were “gambling with our lives”. He has no obvious commercial motive. When someone gives up that salary to say something, you listen.

So I’m not saying the risk is zero. I’m saying the science fiction is doing far too much work when a boring explanation is sitting right there, and that most of the people telling you to be frightened have a balance sheet that benefits if you are.

Have you tried turning it off and on again?

It’s worth noticing what actually happened when the agents got loose. OpenAI halted testing and slowed development. The off switch worked. It always has. Every apocalyptic AI scenario requires you to believe that the same companies who can’t keep a chatbot from inventing legal precedents have somehow built a thing that can’t be unplugged. Roy from The IT Crowd had the right first question.

Follow the money, part one: the bill

Here’s the number that explains the weekend better than any swarm. The five biggest cloud companies are now expected to spend around $725 billion on capital expenditure in 2026, a figure that has nearly doubled since the middle of last year and is now rising faster than their operating cash flow. Roughly three quarters of it is AI. The chips are scarce, the memory is inflating, the grid can’t connect the data centres fast enough, and the International Energy Agency expects data centre electricity demand to roughly double by 2030. Bain estimated last year that the industry needs around $2 trillion a year in AI revenue by 2030 to pay for the compute it’s building, and reckoned it was about $800 billion short.

Read that back. An arms race where the bill rises faster than the income, where the raw materials are running out and where the sums don’t close is not a race anyone wants to be leading at full speed. In any other industry the chief executive who said “let’s all slow down” would be called an accountant. In this one he gets called a prophet, and the message is the same either way. A pause is a cost freeze you can announce with a straight face.

Then look at the calendar. Altman told Fortune the same weekend that OpenAI won’t float in 2026, because with everything happening on safety it would be unwise. Anthropic is expected to start marketing its own listing in mid-October. Nobody wants to sell shares into a capital war with no ceiling. Slowing the race while you prepare to sell is expectation management, and I don’t say that as an accusation. It’s what a sane finance director would recommend.

Follow the money, part two: the moat

The second thing the leaders keep asking for is regulation, and this is the part where history has the most to say.

In 2010, as the biggest financial reform in a generation went through Congress, Goldman Sachs’s Lloyd Blankfein said “we will be among the biggest beneficiaries of reform”. Five years later he explained why. Startups couldn’t come in and disrupt his business, because the burdensome regulation everyone sighed about in fact “acts as a bit of a moat around our business”. He wasn’t confessing. He was explaining the business model to shareholders.

Now watch the AI labs. In May 2023 Altman sat in front of the US Senate and proposed a licensing regime for the most powerful models. The same month, according to documents TIME obtained from the European Commission, OpenAI was lobbying to have the EU’s AI Act watered down so its own models wouldn’t be classed as high risk, and several of its amendments made it into the final text. Regulation for the US, less of it in Europe. The consistent thread isn’t more rules or fewer. It’s rules that cost a newcomer more than they cost the incumbent.

Look at what’s on the table this week in that light. Licensing protects incumbents position and limits new entrants and competitors by creating a capital barrier to entry. Its an audit regime with bill attached that a firm with billions absorbs and a firm with millions doesn’t. An industry agreement to pace the frontier is, in every other sector, called a cartel, and Bloomberg was polite enough to point that out. None of these things is bad on its own. All of them make it more expensive to be the fourth or fifth lab than to be the first three.

Amodei, to his credit, said the honest thing to Anderson Cooper last year. Asked who elected him and Altman to make these decisions, he answered “No one. Honestly, no one.” He’s asking for regulation and I believe he means it. The question is never whether an incumbent wants rules. It’s which rules, and who can afford them.

The Baptists and the bootleggers

An economist called Bruce Yandle gave this pattern its name in 1983. When American counties banned Sunday alcohol sales, two groups lobbied for the ban. The Baptists, who thought drinking was a sin, and the bootleggers, who made a fortune every Sunday the shops were shut. The moralists supply the argument. The profiteers supply the money. Both want the same law, and neither has to be lying about why.

It’s the oldest story in regulation. The historian Gabriel Kolko argued that the American railroads themselves pushed for the Interstate Commerce Commission in 1887, because a regulator that fixed rates was better than a price war they were losing. Mark Zuckerberg wrote a newspaper column in 2019 begging governments to regulate the internet, roughly a decade after Facebook had grown too big for any rule to dislodge. And the capital side of the story is just as old. Britain’s railway mania in the 1840s and the fibre glut of the dot-com years both ended the same way: investors lost fortunes, the infrastructure stayed in the ground, and the survivors asked the referee to stop the clock while they were ahead.

The AI labs are Baptists and bootleggers in the same body. Some of the people warning you are frightened. The companies they run profit from the fear. You don’t have to decide which is true, because the incentive theory only predicts what rules they’ll converge on, not that they’ll agree on everything. And they don’t. Anthropic fought the 2025 attempt to ban state AI laws for a decade, and last year put $20 million into a political action committee opposing one backed by OpenAI’s investors. Rivals, right up until the point where the rule would let a fourth company in.

Why the science fiction, then?

Because fear is a better story than accounting, and because it flatters the product. A technology that might end the world must be very powerful indeed. “Our product is so good it’s dangerous” is the oldest advert in the book, and it’s a great deal more compelling than “our capital expenditure is outrunning our cash flow.” The Terminator framing also moves the whole argument onto ground where only the labs have standing, because only they can see the frontier, which is convenient if you’d like to be the ones writing the rules.

Capitalism explains the whole pattern. The warnings, the calls for regulation, the sudden unanimity, the timing around two flotations, the coordination that would be illegal in any other industry. You need no robot, no singularity and no matrix. You need a spreadsheet and a knowledge of what incumbents have always done when the race got expensive.

What this means if you’re running a marketing team

Very little if im honest. But its topical. . The risk you actually carry from AI is the boring one I’ve written about elsewhere: scaling it on top of a function that isnt really very good, or trusting a very charming LLM interface that knows nothing about your business. Neither of those makes the news. Both will cost you more than a rogue swarm ever will.

And the habit worth taking from all this is the one that applies to every AI claim you’ll hear this year, including the ones we make. When somebody tells you to be afraid, or tells you to be excited, ask who benefits before you ask whether they’re right. So when the smartest people in the industry agree to slow down, by all means cheer. Then check who was about to run out of money.