The Last Invention

In 1965, a mathematician named I. J. Good — who had worked alongside Alan Turing at Bletchley Park — wrote a sentence I keep coming back to: "the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control." Read it once and it sounds like science fiction. Read it twice and you notice he'd already spotted, in one breath, both the promise and the entire problem — the "provided that" is doing all the work in that sentence, and it still is, sixty years later.

I wrote a course this week on giving AI systems a working memory — a genuinely practical, hands-on thing, the kind of problem you solve on a Tuesday. But writing it, I kept bumping into the same larger question from underneath: memory is one of the ingredients a mind needs. What happens if you eventually have all of them? This essay is where I let myself actually ask that, properly, with the same rule I hold myself to everywhere else on this site — grounded first, speculative only where I say so out loud, and never mistaking a vivid story for a fact.

Three words people use loosely

Before going anywhere, the terms need to earn their keep, because "AGI" gets used to mean at least three different things depending on who's talking.

Narrow AI is settled — it's what the entire industry already runs on. AGI and ASI are not settled at all, and I mean that in a stronger sense than "the date is unclear." Even the definition is contested. Some serious researchers think we're closer to general intelligence than the public conversation assumes; others think the entire framing — that intelligence is one dial you turn up — quietly smuggles in an assumption that may simply be wrong about what "general" would even require. I'm not going to pretend that argument is settled here. I'm going to be honest that it isn't, and build the rest of this essay on ground that holds regardless of how it resolves.

We have been here before — just never all at once

The instinct to say "this changes everything" is not new, and it is not always wrong. Handled honestly, the right response isn't to dismiss the instinct — it's to go look at what actually happened the last several times something this large arrived.

Flip through those six and a pattern holds with uncomfortable consistency: every single one did change nearly everything, and every single one took far longer to actually land than the people living through its beginning expected. The printing press needed a century for literacy and institutions to catch up to the raw technology. Electrification needed forty years and a generation of factory managers who'd grown up with steam to retire before anyone figured out how to lay out a factory floor around a motor instead of a driveshaft. Even the internet — the fastest of the bunch — took fifteen to twenty years from "early web" to the platforms that actually captured its value.

I wrote about part of this pattern already, in More Light: every general-purpose technology looks, from the middle of its own installation phase, like either a bubble or a bust, and the real payoff — Carlota Perez's "golden age" — comes later than the hype and lasts longer than the fear. AI is not a special exception to that pattern. It may be the fastest instance of it in the historical record — the six-shift gallery above is itself accelerating, agriculture to writing to printing taking millennia-then-centuries, computing to AI taking a single human generation. That compounding speed is real and worth taking seriously. But "faster than before" and "instant, unprecedented, entirely unlike anything in history" are different claims, and only the first one is actually supported by looking.

Nobody agrees on the clock, and that disagreement is itself the data

Here is where I have to resist the temptation every writer on this topic faces: picking a date, attaching it to a name, and letting the piece live or die by whether that person turns out to be right. I won't do that, because it's usually not an honest representation of how hedged the actual experts are, and because — I've watched this happen — it ages spectacularly badly.

What is real and well-documented is the shape of the disagreement itself. Surveys of AI researchers — the AI Impacts project has run several of these over the years — consistently find enormous individual variance and a median that has moved earlier with each successive survey. That second fact is the more interesting one. It's not that everyone agrees and the date is soon; it's that the distribution itself keeps shifting, which is exactly what you'd expect if the honest answer is "we genuinely don't know, and the evidence keeps updating us." I wrote about the general shape of this kind of uncertainty in Seeing Early, Thinking Sideways — Roy Amara's law, that we overestimate the short run and underestimate the long run, applies here with unusual force, because both halves of that error are visible in the field's own history simultaneously, in different camps, right now.

Taking it all the way up — the universe-scale version of the question

Here is where I want to zoom out further than a single essay about AI usually goes, because there's a genuinely scientific frame — not a fictional one — for thinking about intelligence at the largest scale there is, and once I learned it I couldn't stop turning it over.

The physicist Enrico Fermi is said to have asked, over lunch in 1950, a deceptively simple question: given how old and vast the universe is, and how comparatively cheap interstellar expansion should be for any civilisation with a few thousand years of technological head start, where is everybody? The sky should be loud. It is, as far as we can tell, silent. That gap between "should be loud" and "is silent" is the Fermi paradox, and one serious candidate resolution is the idea of a Great Filter — some step between lifeless chemistry and a civilisation that reaches the stars that almost nothing gets past.

Sit with the actual structure of that idea for a second, because it's more unsettling than it first appears, in a specific and precise way. If the hardest filter is behind us — the origin of life itself, say, which does look staggeringly improbable — then we got lucky, and the sky ahead is genuinely, quietly ours. But if the hardest filter is still ahead of us, then something about the step we are approaching right now tends to end civilisations before they get any further. General intelligence — or more precisely, what a civilisation does in the narrow window after it invents something smarter than itself — is one of the candidates people have proposed for that filter. I want to be very careful here: this is a hypothesis among several, not a finding, and treating it as settled fact would be exactly the kind of overclaiming I'm trying to avoid in this piece. But as a reason to take the alignment problem seriously — to actually worry about the "provided that" in I. J. Good's sentence — it's a more sobering one than any op-ed I've read, precisely because it doesn't require believing anything about killer robots. It only requires believing that very capable optimisation processes, handled carelessly, are a plausible enough way for a technological civilisation to fail.

Two short scenes, clearly labelled as guesses

I promised a little fiction and hypothesis, so here it is — but marked, honestly, as exactly that: not a prediction, a way of making an abstract argument concrete enough to actually feel.

A decade out, grounded. A hospital in a mid-sized city runs a diagnostic assistant that has read more case histories than any doctor alive could in ten careers, and it defers, visibly and by design, to the human in the room for anything it flags as ambiguous — because the institution that bought it insisted on that architecture, the same way banks insisted on the four-eyes principle for large payments a century before software existed. Nothing about this scene requires new physics. It requires the memory systems from this week's course, wired up with the same restraint the healthcare row in that course's industry chapter already described.

Further out, and much more speculative. A research lab runs a system that can meaningfully improve its own training process — not "write code for a human to review," but shorten its own next iteration's cycle time in a way that compounds. Whether that loop runs for months or hours if it ever starts is, honestly, one of the more consequential unknowns in this entire field, and serious, sober researchers disagree about whether it's a near-term engineering question or a far-future one that current methods can't even approach. I'm not telling you which. I'm telling you it's the single scenario that turns "decades to think about this" into "maybe not," and that uncertainty alone is a reasonable thing to plan around, the way you'd wear a seatbelt without believing a crash is likely on any particular drive.

The genuinely open problems — not the ones from the movies

The public conversation about AI risk spends a strange amount of time on a monster that doesn't resemble the actual open research questions. Here are three that do, in language an engineer would recognise:

Alignment isn't "will it turn evil," it's a specification problem. Every engineer who has ever written a metric that got gamed already understands the shape of it — Goodhart's law, "when a measure becomes a target it ceases to be a good measure," at a much higher level of capability. Getting a system to reliably do what you actually meant, not what you literally, incompletely specified, is a hard, unglamorous, and unsolved engineering problem, not a movie plot.

Concentration of power is a nearer-term and more mundane danger than a rogue machine. A small number of labs, companies, or states holding a decisive capability advantage is a story humanity already has centuries of practice being nervous about, going back to nuclear weapons and before. It doesn't require the AI to want anything at all — only for the people who control it to be people, with the usual mix of good judgment and bad incentives.

Verification — trusting a system smarter than you at checking its own work — is a problem we do not have a general solution to. How do you audit a claim from a mind that reasons in ways you can't fully follow? This is, not coincidentally, the same underlying question I explored from the systems side in When AI holds the keys — every capability you hand a system is a door, and the more capable the system, the harder it becomes to be sure you've checked every door properly.

None of these three are solved. None of them require believing in doom to take seriously — they require the same instinct any good engineer already has for a system whose failure modes they haven't fully mapped yet.

Industry by industry, because the abstract version undersells it

Zoom back down from the universe to the ordinary, because this is also, still, an economics and a jobs story, and it deserves the same industry-by-industry honesty I gave the memory course.

Notice that every single row holds a real opportunity and a real risk at the same time, and that the technology itself doesn't pick which one wins — the institutions built around it do. That's not a hedge to dodge a hard question. I wrote the optimist's case for exactly this kind of abundance at length in More Light, and I stand by it — cheap intelligence genuinely does mean more of the problems the world has always had finally get addressed. This essay is that same coin's other, harder-to-look-at face: the size of the upside is not, on its own, evidence that we'll navigate the transition well. Both of those are true at once, and I don't think you get to skip either one.

What I actually believe, stripped of hedging

I said at the start I wouldn't dress up speculation as fact, so let me be equally direct about the parts I do think are solid.

I believe the historical pattern is real: general-purpose technologies reshape everything, take far longer than the hype implies, and eventually make the world genuinely better on average — while making specific years brutal for specific people, which is a cost worth naming plainly, not averaging away. I believe the honest epistemic state on AGI timing is wide disagreement trending earlier, not a hidden consensus anyone is sitting on. I believe the Great Filter is a serious enough hypothesis to justify real caution, without being anywhere close to a proven explanation for anything. And I believe the boring, unglamorous engineering problems — specification, concentration of power, verification — deserve far more of our attention than the cinematic ones currently absorbing it.

What I don't believe is that any of this is predetermined. I. J. Good's sentence has a conditional clause in it for a reason. The whole essay, if it's done its job, should have left that conditional feeling less like a throwaway and more like the entire point.

Closing the loop, back to memory

I started this piece by saying it grew out of a course about giving AI systems a working memory, and I want to end there too, because I think it's the most honest place to land. A person is not just a stack of intelligence. A huge part of what makes a person trustworthy is memory in the fullest sense — a continuous thread that holds them accountable to what they said last year, what they promised, who they've hurt and helped, and what they've learned from it. If we ever do build something that reasons better than we do, the question of whether it has anything resembling that — a memory that makes it answerable to its own past, not just capable in its present — might matter as much as how clever it is. I don't know the answer. I know it's a better question than "will it be evil," and I know it's the one I intend to keep asking.

We don't get to choose whether this century asks the question I. J. Good asked in 1965. We do, still, get to choose how carefully we build the answer — and, I'd argue, what we choose to remember while we do it.

Sources & further reading: I. J. Good, "Speculations Concerning the First Ultraintelligent Machine" (1965); on the historical pattern, Carlota Perez, Technological Revolutions and Financial Capital (2002), and Paul David's "The Dynamo and the Computer" (1990) on the decades-long lag in electrification's productivity payoff; on AGI timeline surveys, the AI Impacts project's repeated surveys of AI researchers; Roy Amara's law via Seeing Early, Thinking Sideways; Enrico Fermi's 1950 lunchtime question as recounted by Eric Jones (1985) and Robin Hanson's essay "The Great Filter — Are We Almost Past It?" (1998) for the Great Filter framing; on alignment as a specification problem, the long tradition running from Norbert Wiener's 1960 warning about machines whose goals diverge from ours, through Stuart Russell's Human Compatible (2019); and Goodhart's law, after economist Charles Goodhart. Estimates and interpretations of contested or unsettled questions are marked as such throughout; where this essay speculates, it says so.