When does aggregate self-interest cross into existential risk? How long can a system run with that risk present before the structure has to change or fail?
Those are the questions worth asking, and they are the questions almost no one is asking. They do not have clean, settled answers; they are diagnostic instruments, tools for steering. What they do is move our attention off the wrong target, the moral composition of the individual actors, and onto the right one: the structure that determines what those actors reliably produce under pressure.
A few weeks ago I released an open-source system that lets one person do the developmental work of a television writers' room for a few hundred dollars, along with a novel that system produced. The novel is real; you can read it. The novel is not the point. The point is what it makes visible: capability is advancing faster, getting cheaper, and diffusing more widely than the institutions responsible for governing it behave as though it is. If one person working alone, on a consumer budget, can do this, then the assumptions underneath our entire approach to managing powerful technology are aging badly: the assumption that dangerous capability stays rare, expensive, centralized, and easy to watch. That is the gradient. The novel just lets you see it.
This is where most of the public conversation goes wrong. It treats catastrophic outcomes as failures of individual morality: reckless founders, unethical executives, irresponsible deployments. Individual responsibility is real. Some people make better choices than others, and some are making good ones right now. A system capable of producing catastrophic outcomes, however, cannot be evaluated only by the intentions of the people inside it. Evaluating it that way is a category error. It has to be evaluated by the pressures it creates and the failure rates it reliably produces under competition.
Most people building advanced AI are not cartoon villains. Many are thoughtful. Many are cautious. Many are genuinely worried about the systems they are building. Most of them, though, operate inside an environment where slowing down does not merely shave profit; it can mean losing strategic position entirely. A system under that kind of pressure does not require universal recklessness to become dangerous. It requires only that enough actors eventually decide they cannot afford restraint.
Consider Boeing. I am not saying no one there made bad choices; people did. A structure that rewards share buybacks over safety engineering will eventually produce managers who choose to delay the 737 MAX fix. Many people inside the company resisted. The structure did not need every actor to fail. It needed only a fraction to fail, and the pressure made that fraction reliably non-zero. Blaming the specific managers who signed off lets the structure continue unchanged, because if the pressures remain, the structure simply produces new managers facing the same incentives. You can fire every person who made the wrong call and you will have changed nothing, because you did not change the thing that made the wrong call rational.
The history of industrial failure is full of systems like this: not systems where every participant was malicious, but systems where the incentive structure made eventual failure statistically likely, and where the public response afterward was to find an individual villain, punish him, and leave the structure that produced him fully intact. Boeing, the financial crises, regulatory capture: the same shape every time. We repeat the cycle because we keep asking the wrong question. Were the people involved good or bad? The question with an actual answer is different: what does this structure reward when ordinary self-interest gets applied, which it always will?
That last clause is the whole move, so I'll make it explicit. Whether any given actor reaches for the virtuous lens or the self-interested one is not a hard question made tractable by better analysis. It is the wrong question. In aggregate, a population applies both, reliably, in roughly stable proportion. To whatever extent that proportion is malleable, it is mostly downstream of the structure anyway. For the purposes of figuring out whether a system is safe, treat the distribution of individual morality as effectively fixed. Judging individual motives is not illegitimate. It is simply insufficient. Evaluate me all you like on the choices I made with my own tool; it won't be enough, because the next person will face the same pressures, and the structure, not their character, is what will tend to decide the outcome.
I want to slow down here, because this is the point where the argument gets misheard. When I say we cannot rely on individual virtue, I am not saying virtue is weak, or rare, or beside the point. I am saying something narrower, and I think harder to argue with. When a failure cannot be undone, no rate of human goodness is a safety margin, because the margin only has to be breached once. "Most people are good" is true. It is also not a number you can put under a load that is not allowed to fail.
That is not a verdict on anyone's character. We put guardrails on mountain roads, and nobody reads the guardrail as an insult to careful drivers. The guardrail is there for the predictable few who will go off anyway, on the one night that matters, and its presence makes no claim about how good the drivers are. You design the aircraft so that one exhausted mechanic's mistake cannot kill three hundred people, and you also train the mechanic well, and no one experiences the redundancy as contempt. Designing for failure and appealing to people's best work were never rivals. The structure is the guardrail. The appeal to our better nature is the training. You want both, and you build the floor out of the one that does not depend on every person clearing the bar every time.
So virtue is not idle in this picture. It has a job, and the job is hiding inside the second question I opened with: how long can a system run with the risk present before the structure has to change or fail? Part of the answer to that question is simply how much goodness the people inside it bring. Individual virtue does not decide whether a badly built structure eventually fails; the incentives decide that. What it does is move the failure rate at the margin, and moving the rate buys time. Every person who declines the extractive move when the structure rewards it lengthens the runway by a little. That is not nothing. It may be the whole difference between changing the structure in time and changing it too late, which is the difference the next essay turns on. What virtue cannot be is the floor. It can be the thing that keeps us on our feet long enough to build one, and that is worth being grateful for without ever being safe to depend on.
The core mismatch underneath all of this is between short-term and long-term incentives. In markets, quarterly earnings and share price dominate. In politics, the next election dominates. In both, we have optimized for present experience at the expense of future survival. A response that does not change that relationship does not do anything, however active it looks. It punishes the people the system produced and leaves the system free to produce more of them.
This is the standard I would carry into the next essay, and into any proposal from anyone, in any direction. The question is not whether it puts better people in the room. It is whether it changes what the structure rewards — whether it makes aggregate self-interest less likely to produce the outcome we can't undo, whether it lengthens the time we can safely run with the risk present. The question is not whether good people exist inside these systems. They do. The question is whether the system rewards caution strongly enough to make caution stable under pressure. That is an engineering problem, not a spiritual one, and the next essay is about the best mechanism I can find for it.