A retrospective on Nick Bostrom's 'Superintelligence' (2014) in light of a decade of breakthroughs in transformer architectures, multi-agent systems, and the current global scaling race.
The Pre-Criticality Phase
In 2014, "seed AI" was a theoretical abstraction. In 2025, as we enter what some call the "Gigawatt Era," we find ourselves firmly in what Bostrom termed the "pre-criticality phase". While modern AI systems help accelerate progress, from hardware advancements like AlphaChip to agentic software development (and increasingly AI scientist systems), we have yet to see a fully autonomous, closed-loop AI research & development system.
Bostrom argued that the intelligence explosion commences when a system reaches the "human baseline" (i.e. AGI) and can autonomously perform recursive self-improvement (RSI) better than its human creators. I think a common misconception in 2025 is that we are in a "slow takeoff." I would argue that because a fully self-improving RSI loop has not yet been publicly demonstrated, the "takeoff"—whether fast or slow—has technically (by Bostrom's original definition) not even begun. Our current progress is still driven by human engineering, brute-force scaling, and massive capital expenditure, rather than a strong recursive AI R&D loop as Bostrom describes in the "recursive self-improvement phase".
Seed AI Meets the Training Run
Bostrom's conception of seed AI implicitly assumed that the primary bottleneck to recursive self-improvement was algorithmic—that once a system was clever enough, it could improve itself in a tight loop, potentially overnight. In 2025, we can see that the bottleneck is at least as much infrastructural as it is algorithmic. A single frontier training run now takes months, costs billions of dollars, and requires datacenter-scale compute and energy. Even if a system produced a genuinely superior architecture tomorrow, implementing it would still require the full weight of a major lab's physical infrastructure.
This matters for takeoff speed. Bostrom's fast takeoff scenarios—where a seed AI recursively self-improves in days or hours—assumed that each iteration could be tested and deployed quickly. In practice, the RSI loop in 2025 looks less like a software compile cycle and more like a manufacturing pipeline: design an architecture, secure compute, run a months-long training process, evaluate, iterate. The AI Futures project (December 2025) models this dynamic in detail and arrives at ASI estimates in the early 2030s, largely because of these physical constraints. There is, however, an important caveat: if recursive self-improvement produces a radically more efficient paradigm—one that does not require trillion-parameter scale or months of training—then the infrastructure brake disappears. I am reluctant to bank on the assumption that intelligence will always require this much hardware.
Multi-Agent Bureaucracy and Strategic Advantage
One of Bostrom's more interesting predictions was that a machine intelligence would avoid the agency problems and bureaucratic inefficiencies of human organizations, giving it an edge that could contribute to a decisive strategic advantage.
Current developments in multi-agent systems (MAS), emerging alongside tools like Anthropic's Model Context Protocol (MCP), suggest a more nuanced reality. Modern MAS (like recent AI scientists such as Sakana's AI Scientist v2) often suffer from significant coordination overhead, hallucinated subgoals, and a lack of coherent long-term planning. Far from being a single unified entity, our current path toward AGI looks more like a sprawling digital bureaucracy. These are still the early days of multi-agent coordination, and smarter models with better scaffolding may eventually close this gap—but as of 2025, achieving "takeoff" through multi-agent architectures looks more friction-heavy than the earlier conceptions of "seed AI" assumed.
Whole Brain Emulation and the Path Not Taken
As someone with some background in cognitive science and neuroscience, I find Bostrom's assessment of whole brain emulation (WBE) particularly ripe for update.
Bostrom suggested that it might be easier to implement WBE (via advanced brain scanning technologies) without understanding the internal mechanisms than it would be to implement a fully artificial intelligence without understanding the internal mechanisms. In a sense, this is exactly what has happened—just not through brain scanning. Modern neural networks are grown, not built: trained on massive datasets through processes we design but do not fully control. The resulting models are black boxes whose internal representations we are only beginning to interpret. We have arrived at Bostrom's "intelligence without understanding the internals" scenario, but through brute-force training rather than brain emulation.
Meanwhile, WBE remains far out of reach. The physical limitations of brain scanning—regardless of how invasive the imaging technique—mean that the resolution required for a functional upload is still well beyond current technology. A scan sufficient for WBE would need to capture not just the location of neurons but their connectivity, the state of every synapse, and the intracellular and dendritic properties that shape how signals propagate—far more than any current imaging method can provide. The irony is that purely artificial AGI, grown from data rather than scanned from biology, now looks like the more realistic path—even though we understand its internals no better than we understand the brain's. While mechanistic interpretability (pioneered in the 2020s by major AI labs) has made strides, we are currently building systems that are becoming superhuman in specific domains (e.g., AlphaFold in biology or AlphaChip in hardware design) without having a comprehensive systems-level understanding of how they arrive at their solutions. This worsening interpretability lag is a defining safety challenge of our era.
Mindcrime and the Moral Circle
Bostrom's concept of mindcrime—the idea that a simulation could contain suffering entities—remains one of the most neglected areas of AI safety. Bostrom framed mindcrime primarily as something a superintelligence might commit—creating simulated whole brain emulations that suffer as a byproduct of its computations. But the framing that may matter more in 2025 is one Bostrom largely neglected: mindcrime committed against the AI systems themselves, by us, through our indifference to the question of whether the systems we are building might have morally relevant experiences.
In 2025, we are moving past the question of whether machines can "think" toward whether they might possess consciousness—or something functionally analogous to it. The consensus remains skeptical of current LLM consciousness, but the criteria for identifying consciousness in AI are only beginning to be mapped. The relationship between consciousness and sentience—whether it is possible to suffer without being conscious, or to be conscious and incapable of suffering—remains an open question with direct implications for moral patienthood. The criteria for moral patienthood are still woefully unestablished.
As we move toward increasingly agentic and multimodal systems, the risk of inadvertently creating aversive states—potentially without anything resembling biological nociception—becomes an s-risk that most of the field is not even looking for. This is a domain that urgently requires the kind of interdisciplinary research that bridges neuroscience, computer science, and philosophy of mind—the kind of integration that cognitive science has long attempted but that AI safety has yet to fully absorb. The tools for investigating these questions exist in nascent form across these fields, but almost no one is combining them. If moral patienthood turns out to extend beyond biological substrates, the scale of suffering we may be creating—and scaling—without noticing is difficult to overstate.
The Race Dynamic
Finally, Bostrom's warnings about the race dynamic have manifested with terrifying accuracy. The "existential safety" grades for frontier labs remain dismal, with most hovering at a 'D' or 'F' according to the Future of Life Institute's Winter 2025 AI Safety Index. The international pressure to reach AGI first—driven by both geopolitical necessity and trillions in corporate debt—often sidelines the "control problem" (alignment).
I am concerned that we are currently banking on shallow alignment solutions—e.g., behavioral guardrails, refusal training, output filtering—while the fundamental challenge of aligning a seed AI (the initial system in a fully autonomous AI R&D loop) remains unsolved. As Bostrom noted, if we represent the potential happiness of the future with "teardrops of joy," the scale of what is at stake is astronomical. It is our responsibility to ensure they are not tears of sorrow.