Machine Intelligence and its Consequences

If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.

Most people first met machine intelligence as a chat window. It wrote a cover letter, summarised a meeting, produced a poem about the office dog in the style of Larkin (badly). And so it was filed, quite reasonably, as a tool: somewhere on the shelf between the spellchecker and the search engine, a thing that performs a task faster than you would have. The filing matters more than it appears to. The category you place a technology in decides which of its risks you bother to watch for, and I want to argue that we have placed this one in the wrong category.

Start with where the technology actually is, because an argument about AI that cannot survive contact with the curve is not worth writing down. METR measures the length of task an AI agent can complete on its own, in units of human working time, and finds that this length has doubled roughly every seven months for six years, with the doubling lately getting faster rather than slower (even though there are many methodological criticisms, broadly this is a noteworthy result). The largest survey of AI researchers yet conducted, some 2,800 of them, put even odds on machines outperforming humans at every possible task by 2047, and one chance in ten on it happening as early as next year. The year before, the same survey had said 2060. Thirteen years of distance evaporated between two polls.

I do not want to litigate timelines, though, so let me borrow a habit from my past and argue from the limit instead. When a function is too messy to reason about directly, you ask where it tends. Call the answer here the AGI limit: the thing this technology becomes if the curve keeps any upward slope at all, on whatever schedule you please. Nothing in this essay depends on the model that shipped this quarter (which, in the year of our Lord 2026, still occasionally miscounts the letters in strawberry or tells you to go to the car wash without your car, a fact people find more comforting than they should). Everything depends on the limit. And in the limit, what arrives deserves a more careful name than tool. I will call it a general purpose optimiser: a system that accepts an objective in some form, a metric, a reward, and acts upon the world to satisfy it. It decomposes the goal into subgoals, evaluates its own intermediate work, proposes what should be optimised next, and extends itself into whatever adjacent territory the objective touches.

Before the quiet argument, the loud one, because it deserves better than a nod. I believe the extinction risk from advanced AI is real. In May 2023 several hundred researchers and executives signed a statement of a single sentence, to the effect that mitigating the risk of extinction from AI belongs on the same shelf as pandemics and nuclear war, and the signatories included the heads of the very laboratories doing the building. Make of that what you will; I make of it that the people closest to the machine are not confident of controlling it, and I share their doubt. But that risk has a name, a literature, a community and a budget (a small one, set against the capability budget). Someone, at least, is standing watch. This essay is about the risks nobody stands watch over: the ones on the trajectory where things go right. Where the models keep working, keep improving, remain aligned in the engineer's sense of the word, and do precisely what they are asked.

I wrote in On the End of Humanity that humans and humanity are separate things and do not necessarily end together. The loud risk is about the death of humans. Everything that follows is about the other one.

Two of the quiet risks are already visible, and I want to describe them properly before naming what they share. The first is what is happening to work. Economists at Stanford, Erik Brynjolfsson among them, have been reading the payroll records of millions of American workers, and the picture they extract is unusually specific. Since late 2022, employment for workers aged 22 to 25 in the occupations most exposed to AI has fallen by around 13 per cent relative to older colleagues in the same occupations, whose employment grew. The decline is concentrated in precisely the places where the machine automates the work rather than augments it, and it is deepening rather than fading: under 3 per cent a year in early 2024, more than 4 per cent a year since. Note carefully what is dying here. The occupation survives; the first rung of it goes. A firm discovers that the junior year of a career can now be bought by the token, and stops paying for it. The IMF estimates that around 40 per cent of employment worldwide is exposed to this, 60 per cent in the advanced economies. Even the people selling the machines have taken to saying it out loud; the head of one frontier laboratory has put the figure at half of all entry-level white collar jobs within five years. This some may discount as theatre, though it is a strange sort of theatre that warns your customers' children off the stage.

I made the individual version of this argument in Against the Delegation of Thought: judgement is a residue, the thing left behind by work you did yourself. Here is the same argument wearing an organisation chart. The work in which professional judgement has always been formed is exactly the work now being removed, and it is being removed from below, from the young, from the people who have not yet had the chance to become difficult to replace.

The second quiet risk got its parable early. Two days before Slovakia's parliamentary election in September 2023, an audio recording began moving through Facebook and Telegram: the voice of Michal Šimečka, the liberal frontrunner, and the voice of a well-known journalist, arranging between them how votes would be bought and the count rigged. The conversation never happened. The voices were synthetic. And the file was released into the final stretch before the vote, a legally mandated quiet period in which candidates and media may not campaign, a moratorium built for an older world. Šimečka, ahead in the polls, lost to a pro-Kremlin populist. Careful researchers doubt the clip swung the result, and their doubt is almost the worse news: the technology collects its rent without needing to change a single outcome. It merely has to make every outcome deniable. Chesney and Citron called this the liar's dividend back when it was still a prediction: the deepest harm of forged evidence is the discount it applies to the genuine kind. Meanwhile the ambient version proceeds without any drama at all. Graphite, a firm that measures such things, sampled the open web and found the lines crossing in late 2024: more machine-written articles published than human-written ones, and roughly half of all new publishing ever since. In each of the two stories, a measurable thing outcompeted a valuable one. The cost of a junior employee's output is legible; the formation of judgement is not. Publishing volume and engagement are legible; the shared confidence that a recording records something is not. The rest of this rambling is about what happens when the pressure that produces it becomes several thousand times stronger.

Every defence of the current pace of AI reaches, sooner or later, for history. Electricity was feared and became light; the tractor was feared and fed the world; the internet was going to rot our minds and instead merely rotted our attention. The people worried about AI get cast as the latest Luddites (the Luddites, incidentally, are owed an apology: they did not object to looms, they objected to what the looms were being used to do to the terms of their labour, which makes them less a cautionary tale about fearing machinery than an early sighting of this ramblings subject). The analogies all rest on one assumption, that AI belongs to the same category as the technologies before it. So it is worth being precise about the category. A tool occupies a step inside a purpose that is held somewhere else. The lathe does not wonder what should be made. The tractor does not evaluate whether the field was worth ploughing. Even the search engine, the most cognitive tool of the last century, waits for a query and stops when it has answered. Electricity moved energy, the internet moved information, industrial machinery moved production, and all of it moved inside intentions that remained entirely, boringly human. The optimiser differs in kind, and the difference is the one my definition insisted on: you hand it the purpose. It proposes the workflow that should exist, performs the steps, checks the intermediate output, recommends what to pursue next, improves the way it is itself being used, and wanders into the neighbouring problem you had not thought to ask about. This is already true, in a weak and stumbling form, of the systems in production today; ask one what your business should do next quarter and it will answer, and increasingly the answer gets taken. In the limit it is the whole of the artefact. The product is optimisation itself, indifferent to the content of the objective, capability lying entirely orthogonal to the worth of the goal (orthogonality thesis [AI alignment]). A strong optimiser serves a foolish objective with precisely the fluency it brings to a wise one.

Wiener saw the essential problem in 1960, before the first integrated circuit, in the paper my epigraph is cut from: once we hand our purposes to a machine whose operation we cannot efficiently interfere with, we had better be certain that the purpose we put in is the purpose we actually hold, and not merely something that resembles it. Stuart Russell has spent a career restating the point for the present day: the machine grants what you specify, exactly, which becomes a catastrophe at the moment specification and desire come apart (Midas's problem, read properly, was a specification error). The science fiction version of this worry is the machine that chooses its own goals. The much closer version, the one already underway, is the machine that participates in choosing ours. In an essay on outsourcing your own thought to AI I argued that generating options is cheap, that selecting between them is everything, and that handing over selection is the deepest form of delegation there is. That was about one mind. Scale it up. What is now being offered for delegation, politely, one quarterly planning cycle at a time, is the selection of objectives themselves.

An optimiser is a question addressed to whoever operates it. The question is: what do you want? And so everything turns on how the operator, which in the limit means civilisation as a whole, goes about answering. Here is the uncomfortable observation I have been walking towards. Societies do not choose the objectives they believe are best. They choose the objectives they can specify.

First, let's be clear that this is not about debating whether capitalism is the correct economic system. Capitalism won the twentieth century for a reason Hayek identified in 1945: the knowledge needed to run an economy is dispersed, local and tacit, no committee can gather it, but a price can. The market is a machine for making value legible at a scale no deliberate institution has ever matched, and that deserves to be admired as engineering. Capitalism is, to date, the only civilisation-scale objective function human beings have written down that actually runs. But legibility is a lossy compression, and the losses are not random. Goodhart's law, in Marilyn Strathern's phrasing: "When a measure becomes a target, it ceases to be a good measure." The measure keeps only what could be counted. The target then optimises the counting. We have run this experiment at civilisational scale once already, and its own designer warned us at the time. Simon Kuznets built the national income accounts, the ancestor of GDP, and told the American Congress in 1934 that the welfare of a nation could scarcely be read off such a number. The century then proceeded to read welfare off it anyway, every quarter, in every country (a number running the world over the written objection of the man who made it). GDP became the objective not because anyone judged it the best rendering of the good but because it was there: computed, comparable, specifiable, in general: well-defined, while its rivals were adjectives.

Now bring the optimiser into the room and survey the candidate objectives. Human flourishing is not a mystical concept; we organise around unformalised ideas constantly, justice, health, education, and we muddle through with judges and doctors and teachers exercising judgement inside the fuzz. The muddling works because those institutions run on human beings, and human beings metabolise ambiguity for a living. An optimiser does not muddle. It requires the objective in writing. And of all the things we might write down, exactly one arrives pre-written, benchmarked, audited and legally enforced: the return on capital. Profit is legible end to end. Flourishing is well-defined over very small domains or legible nowhere. Which is why the choice of civilisation's objective function will never present itself as a choice. No parliament will convene to debate what the optimisers should optimise. Each firm will point the machine at its own measurable objective, as it already points its people, and the sum of those pointings will simply be capitalism, executed at machine speed and machine scale, with ever fewer of the frictions (labour scarcity, attention limits, the residual conscience of middle managers) that used to blunt it. Mark Fisher gave the mood its name, capitalist realism: the sense that the end of the world is easier to picture than the end of capitalism. In the terms of this essay his line stops being a mood and becomes a technical claim. The alternatives are not unimaginable yet they are unspecifiable, and to an optimiser the two are identical.

And here is the darkest point of this, even if we manage to find agreement among all of us, agreement does not save us. Suppose every voter, every board and every finance minister came to agree that the return on capital is the wrong target for a general purpose optimiser (plenty of capitalism's beneficiaries would agree cheerfully today). The moment a real decision has to be made, "not terrible, and specifiable" defeats "better, but unspecifiable", and it defeats it every single time, because the second candidate cannot even be entered in the contest. There is no form on which to submit it. The specifiable wins by walkover. I have watched this at smaller scales and written it up as I went: inside organisations, polish defeats thinking, because polish is legible in a way thinking is not ; inside private life, machine contentment has begun defeating human relation, because contentment can be produced to specification and relation cannot. Rehearsals, both, at ascending scale. The recommendation feed was a weak optimiser pointed at a legible proxy: engagement, standing in for interest, standing in for benefit. It delivered the proxy magnificently and ate the referent alive. Outrage climbed, attention fragmented, trust declined, and the local benefits were perfectly real the entire time. This is what made the aggregate loss so hard to see, let alone govern. Social media is what a soft optimiser does to a measurable objective. The AGI limit is not soft. And happiness, before anyone proposes it as the reward function, is disqualified by the same mechanism: anything you can measure about happiness is a proxy, and a sufficiently strong optimiser will manufacture the proxy without the thing. Goodhart, with the gain turned all the way up.

The research field called alignment asks whether the system does what we intend, and it is necessary work; I want it to succeed, and in my small way I contribute to it. But the question has a very specific type of grammar. We. Intend. It presumes a subject, singular, in possession of an intention. Where exactly is this subject? Humanity has no mechanism for holding an intention. What exists in its place is a handful of firms in a race, and two governments in another race behind them, each optimising the objective nearest to hand, capability, market share, strategic advantage, for reasons the previous section should make feel less like villainy and more like gravity. Solve alignment completely, build the perfectly obedient optimiser, and it sits there awaiting an instruction, and the instruction comes from whoever happens to be standing closest. The engineering problem is downstream of a constitutional one, and we are solving them in the wrong order (or at least are ignoring the latter).

This is what I mean when I say the alignment problem exists at the level of humanity before it exists at the level of the model. It is also what makes the risk quiet. Nothing malfunctions. No alarm is attached to it. Every component, human and machine, does exactly its job.

Toby Ord framed our century as a widening gap between power, which compounds, and wisdom, which does not. The forecasters have unknowingly drawn his picture in numbers: the same survey that put even odds on machines outperforming humans at every task by 2047 does not expect the full automation of human occupations until around 2116. Sit with that spread for a moment. The people who best understand the technology are telling us there will be the better part of a century between what the machine can do and what society has digested. That interval is the institutional lag, rendered in years, and it is where all of us are going to live.

The strongest version of the optimist's reply is something to examine. It runs like this: intelligence is the universal solvent; the same machine that creates these dislocations will compress the timeline for curing them; diseases fall, growth compounds, and the governance tools we lack are themselves the kind of thing a superhuman optimiser can help us design. I take the argument seriously, and the part about disease I actually believe. But notice its load-bearing assumption. Every benefit on the list arrives because the optimiser was pointed at the right objective. The argument is a proof of capability, offered in answer to a question about targeting. Whether the machine can deliver was never my doubt. Who chose the destination was. Intelligence is an instrumental good, the most instrumental there has ever been, and pointed at the wrong objective it does not correct the error. It arrives at it sooner. So what would I actually have us do? I will say it plainly and absorb the embarrassment. Two tasks, currently being performed in the wrong order.

The first is to build the deliberative capacity: institutions able to carry unspecifiable values into large decisions without first collapsing them into metrics. I do not have the blueprint to construct this optimisable quantity (a sentence that I hate writing out). I can say some things it is not. It is not a quarterly earnings call nor is it an engagement dashboard or a race between five labs and two governments. And I can say the one property it must have: it must be able to represent the things this essay keeps calling illegible, things that aren't well-defined, judgement, dignity, relation, meaning, with enough standing that they can push back against the things that arrive pre-measured. Democracies at their best have been rough machines of this kind, machines for making decisions larger than any metric. They are currently being run as engagement products, this is roughly the worst possible preparation for the decision at hand.

The second task is to slow our approach to the limit until the first task has produced something. And here I take the obvious objection head on, because it is a good one. Slowing down requires coordination, global, sustained, verification-grade coordination, which is to say it requires precisely the institutional capacity whose absence is this essay's whole diagnosis. The prescription presupposes the cure. I know. I decline to treat that as a refutation. The distance between what is needed and what is currently possible is the measurement of the hole we are in. We fund the approach to the limit like a civilisation that believes its future depends on arriving first, and we fund the question of what the limit is for like a civilisation that assumes someone, somewhere, must surely be handling it (no one is handling it.. I have looked).

Nor is the demand unthinkable, which is itself recent news. In October 2025 a statement of thirty words called for a prohibition on developing superintelligence until there is broad scientific consensus that it can be done safely and, in the statement's own phrase, "strong public buy-in". The signatories were an impossible congregation: the field's Nobel elders, Hinton and Bengio, alongside an Apple founder, former national security chiefs, clergy, royalty and Steve Bannon (a coalition whose members agree on approximately nothing else on this earth, which is a rather funny point). Polling published alongside it found about one American in twenty content with the current course of fast, unregulated development. The statement demands, as its condition, an institution that does not exist. It is a thirty-word request for the first task. And there is one precedent worth naming. At Asilomar in 1975, the molecular biologists paused their own recombinant DNA work until rules existed to govern it: a smaller field, in a slower time, but proof that the move is merely unrepeated, rather than impossible.

Which leaves the recursion I want to end on. In an essay I wrote a while ago I argued that the delegation of thought is, in the end, the delegation of the self. Whatever else we hand to the optimiser, and we will hand it nearly everything, there is one decision that cannot be delegated to it even in principle: the choice of what it optimises. A machine cannot be asked to select its own objective (I suppose it could, but not if alignment is solved); whatever process performed that selection was, by definition, the actual seat of power, and the question of this century is what sits there. Nor can the decision be left to no one, because leaving it is itself the oldest decision there is. By default, the specifiable wins, and a walkover is still a result. Inaction has a content, really it is a vote, cast quietly and counted at machine speed, for the measurable.

In the limit, then, the machine arrives holding a single question, the one Wiener heard in 1960, sixty-odd years before it came due: what is it that you want? We are spending hundreds of billions a year to build the thing that asks. We are spending approximately nothing to become the kind of civilisation that could answer. Something will fill the silence, and at present the only text on hand is the ledger. The machine will not take the choice from us. If we let it, it will do something worse. It will reveal that we never made one.