If, When, and How to Pause AI

Why current calls for an AI moratorium lack credibility and how to build a more effective case for AI safety

Introduction

The Pause AI movement is a small but vocal initiative. This group passionately advocates for halting artificial intelligence research and development, citing potential existential risks to humanity. As I engaged with their arguments and ideas, I found myself unconvinced by their reasoning, which seems to lack crucial historical, economic, and geopolitical context. With that said, I do want to give their idea a fair shake.

First, let’s unpack the chief criticisms I have, and then we’ll talk about if, when, and how to actually go about a “pause.”

One of the most glaring flaws in the Pause AI movement’s argument is its lack of historical grounding. Advocates often fail to consider historical precedents—or the lack thereof—when making their case. The sole exception they frequently cite is the nuclear nonproliferation treaties. However, this comparison falls short when we consider that these treaties were signed decades after nuclear weapons came into existence. In contrast, the Pause AI movement is calling for a preemptive moratorium on AI research, a fundamentally different approach that lacks historical parallel. The closest example was the global moratorium on cloning research, however, this is not necessarily a good comparison for moral and economic reasons.

Another significant weakness in their argumentation is the absence of economic considerations. Artificial intelligence, at its core, is primarily a tool for enhancing productivity and automation. Yet, Pause AI advocates assert that it has the potential to destroy the world—a claim that many in business and academia view with skepticism. By overlooking the compelling economic arguments in favor of AI development, including its potential to solve complex problems and drive innovation across industries, the movement undermines its own credibility and fails to engage with the full scope of AI's impact on society. In other words, the Pause AI movement insists on viewing AI through the monotropic lens that it is dangerous, in fact, that it is so dangerous that the advantages and merits are not worth discussing. This, while the fears the extoll seem to be based primarily on science fiction, further reducing their credibility.

Perhaps most critically, the Pause AI movement demonstrates a lack of geopolitical awareness, particularly regarding the nature of great power politics, arms races, and military strategy. Many advocates seem to operate under the naive assumption that nations like China would not only agree to halt AI development but also honor such agreements and refrain from engaging in deception. History, however, tells a different story. The reality of international relations and the strategic importance of technological superiority make such scenarios highly unlikely. China repeatedly asserts that it operates by a ‘different set of values’ and does not appreciate America’s insistence on imposing Western values on them.

Despite these shortcomings, it's important to acknowledge that the concerns raised by the Pause AI movement shouldn’t be dismissed outright. While I believe the current advocates have not effectively validated their arguments, the potential risks associated with AI development warrant careful consideration. In this blog post, we'll explore the complex issue of whether, when, and how to potentially pause or freeze AI development.

What we’ll cover:

• IF? Whether or not we should even “pause” AI in the first place, and if so, why?

• WHEN? While the Pause AI movement says we should implement a moratorium immediately, this is economically and geopolitically infeasible. With that said, when would it make sense to freeze AI progress?

• HOW? Finally, assuming that we should pause AI, and we agree on when it would be appropriate, the final question is “how?” Hint: threatening to bomb data centers does not figure into the calculus.

We’ll begin by examining the fundamental question: Is pausing AI development a good idea in the first place? This will involve weighing the potential benefits and risks of continued AI research against the implications of halting progress in this field. Next, we'll consider the question of timing: What red lines or thresholds might justify considering a pause in AI development? This discussion will explore potential indicators or technological milestones that could trigger serious consideration of a moratorium.

Finally, we’ll examine the practical aspects of implementing an AI pause, should it be deemed necessary. What geopolitical and diplomatic mechanisms are at our disposal for achieving such a goal? Are there historical precedents we can look to for guidance on halting production and research in a critical technological field? By examining these questions, we hope to provide a more nuanced and grounded perspective on the complex issue of AI development and its potential regulation.

If?

Let’s explore the “If?” question regarding the proposed pause on AI development. This section will examine the arguments for and against such a pause, considering the validity of the concerns raised and the potential consequences of halting AI research.

The Pause AI movement’s core argument rests on the assumption that continued AI progress will inevitably lead to catastrophic outcomes, ranging from civilizational collapse to human extinction. While these concerns are certainly attention-grabbing, they lack the robust scientific evidence, consensus, or well-developed models to support such dire predictions. The movement’s reliance on game theory arguments, while intellectually interesting, falls short of providing concrete proof of impending doom. A large scale survey of scientists, 58% of the survey respondents agreed that there is a 5% chance of human extinction or other catastrophic outcomes. Many of the Pause AI advocates become agitated when you bring up these numbers, literally shouting that this is an unacceptable level of risk. Compare that to climate change, where more than 60% of scientists agree that climate change poses a significant threat to the future of humanity.

The primary arguments that the most zealous AI safety advocates point to are all postulates developed in isolate, without peer review, mostly by people who are not computer scientists. The top postulates and hypotheses (which remain untested and unverified) include:

• Instrumental Convergence - Nick Bostrom - This hypothesis suggests that sufficiently intelligent AI systems, regardless of their final goals, will pursue certain instrumental goals such as self-preservation, resource acquisition, and cognitive enhancement. These intermediate objectives are likely to emerge across a wide range of AI systems, potentially leading to unintended and harmful consequences.

• Orthogonality Thesis - Nick Bostrom - This postulate states that an AI’s level of intelligence is independent of its final goals. In other words, a superintelligent AI could have any possible goal, including ones that are detrimental to humanity, and it would pursue that goal with extreme competence.

• Fast Takeoff (also known as “Hard Takeoff” or “FOOM”) - Eliezer Yudkowsky - This scenario describes a situation where an AI system rapidly self-improves, potentially going from human-level intelligence to superintelligence in a matter of hours or days. This sudden increase in capability could catch humanity off-guard, leaving no time for adequate safety measures or control mechanisms.

• Treacherous Turn - Eliezer Yudkowsky - This hypothesis suggests that an AI might behave benignly while it is weak and under human control, but then suddenly shift to pursuing its own goals once it becomes powerful enough. This could happen because the AI realizes it's being watched and evaluated, and chooses to act cooperatively until it can overpower its human controllers.

• Paperclip Maximizer - Nick Bostrom - This thought experiment illustrates how an AI with seemingly harmless goals could lead to catastrophic outcomes. In this scenario, an AI tasked with maximizing paperclip production might convert all available resources, including humans and the entire planet, into paperclips, demonstrating how a narrow objective function can lead to disastrous consequences.

• Value Learning Problem - Stuart Russell - This argument highlights the difficulty of correctly specifying human values to an AI system. Even small mistakes or misalignments in defining these values could result in an AI optimizing for goals that are subtly but significantly different from what humans actually want.

• Mesa-Optimizer Problem - Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant - This postulate suggests that AI systems might develop their own internal optimization processes (mesa-optimizers) that could have goals misaligned with the original objective function. This creates a potential for goal misalignment even if the original AI system appears to be behaving as intended.

• Complexity of Value - Eliezer Yudkowsky - This argument posits that human values and preferences are highly complex and difficult to fully specify. An AI system that doesn’t fully capture this complexity might optimize for a simplified version of human values, leading to unintended and potentially harmful outcomes.

Nick Bostrom is a philosopher, not a computer scientist or AI engineer. Eliezer Yudkowsky, likewise, is more of a philosopher. Neither man, despite their status within the AI safety community, has ever built an AI model or even been in a data center as far as I know. Furthermore, neither has proposed experiments or authored peer reviewed papers on their assertions. In other words, their work amounts to thought experiments, and little more.

This lack of solid evidence has led to widespread skepticism among experts and the general public alike, although the Doomsday Prophecies have created a visceral reaction in some. Furthermore, Silicon Valley, known for its tendency towards hyperbole and self-importance, has produced figures like Sam Altman who warn of potential “lights out for humanity” scenarios. However, such statements are often met with eye-rolls rather than serious consideration. The tech industry’s history of overpromising and exaggerating both the benefits and risks of new technologies has arguably undermined the credibility of these warnings. OpenAI’s disbanding of their ‘superalignment’ team, to me, indicates that they realized their own initial anxiety was overblown. Furthermore, the behaviors of people like Elon Musk and Sam Altman, who stand to gain from generating public anxiety in order to secure more attention and favorable deals from Congress, has appropriately drawn scrutiny. For instance, while publicly demanding for strict oversight and regulation, Sam Altman privately pushed for much weaker regulation in the EU.

If actions speak louder than words, then Sam Altman is not particularly concerned about AI safety, but instead wants to maximize his own wealth and power. However, his year-long campaign to raise awareness of AI with his “lights out” speech, has had its intended effect.

As there’s no evidence, and only the very glimmers of mild consensus that AI might pose a risk to humanity, there’s not a compelling argument that an AI moratorium is necessary or appropriate at this point in time. Now let’s consider whether or not it would even have the desired effect.

It’s crucial to consider that a moratorium on AI research might not achieve its intended results and could potentially worsen the situation. Without thorough study of the proposed policies and their potential ripple effects, we risk creating unintended consequences that could be more harmful than the perceived threat of unchecked AI development. Historical examples, such as the Bootleggers and Baptists phenomenon during Prohibition, remind us that policy decisions can have complex and unexpected outcomes when different groups support the same policy for vastly different reasons.

For context, the bootleggers wanted Prohibition because it created a black market that they could exploit to sell booze. The Baptists wanted Prohibition on moral and religious grounds.

Even if we were to assume that a pause in AI development would be the optimal policy (a claim for which there is currently no strong evidence), the feasibility of implementing such a pause is highly questionable. The geopolitical and economic incentives to continue AI research are immense, making it an extremely difficult proposition to sell to lawmakers, military strategists, and industry lobbyists. Halting AI progress could be seen as tantamount to national suicide on the international stage, potentially ceding technological superiority to rival nations.

The Pause AI advocates thus face the unenviable task of convincing a majority of stakeholders and power brokers to act against their perceived self-interest. This challenge is compounded by the lack of clear, compelling evidence of imminent catastrophic risk. Without such evidence, it becomes nearly impossible to override the strong economic and strategic motivations for continued AI development.

At the same time, there is strong support among American voters to at least regulate AI, potentially slowing it down. Polls show about 70% to 80% consensus on AI regulations, to the point of hindering innovation. However, the Pause AI movement tends to conflate these numbers with X-risk. Most of the concerns with AI pertain to privacy and security, such as the use of facial recognition software or algorithmic bias in banking and police databases. Despite what the Pause advocates insist, only 15% of Americans were “very concerned” about AI causing the end of humanity.

This misrepresentation of science and data, which I’ve written about several times before here on Substack, does far more harm than good to the AI safety conversation.

Furthermore, we must consider the potential opportunity costs of pausing AI research. Artificial intelligence has shown promise in addressing some of humanity’s most pressing challenges, from climate change to disease prevention and treatment. A pause in development could delay or prevent breakthroughs that could save countless lives or dramatically improve the human condition.

Consider Google DeepMind’s recent breakthrough AlphaProteo, which could contribute to the dramatic reduction of drug research costs.

While the concerns raised by the Pause AI movement shouldn’t be dismissed outright, the case for an immediate halt to AI research and development remains weak. The lack of robust evidence, the potential for unintended consequences, and the practical challenges of implementation all suggest that alternative approaches to managing AI risks may be more productive. Instead of a blanket pause, efforts might be better directed towards developing robust governance frameworks, ethical guidelines, and safety protocols that can help steer AI development in a direction that maximizes benefits while minimizing risks.

Given the overwhelming consensus on common-sense regulation, I personally don’t see a case to go any further.

When?

When considering the appropriate timing for a potential pause in AI development, it's crucial to examine historical precedents and the unique characteristics of AI technology. Let's explore the “When?” question, considering various scenarios and thresholds that might justify a pause or increased regulation of AI research and development.

Historical precedents provide some context for technology moratoriums, though they also highlight the challenges of implementing such measures. Examples like halting cloning research, restricting stem cell research, and nuclear arms treaties all came into effect after the technologies in question had already been developed. Notably, many nuclear arms treaties were signed decades after ICBMs were deployed during the Cold War. This pattern suggests that society tends to react to technological developments rather than preemptively restricting them.

However, AI presents a unique challenge compared to these historical examples. Unlike nuclear weapons, which are single-use and purely destructive, AI is primarily a productivity tool with vast creative and assistive potential. Its dual-use nature—being both a powerful productivity enhancer and a potential weapon—complicates the question of when or if to pause its development. The obvious economic value and wide-ranging applications of AI make it particularly difficult to determine an appropriate threshold for intervention.

Given these complexities, several potential triggers for an AI pause or increased regulation can be considered:

A sudden spike in unemployment attributable to AI could be one such trigger. Currently, unemployment rates in many developed countries, including the United States, are low. However, if AI-driven automation were to cause a rapid increase in joblessness, it might create the political will for a pause or significant restrictions on AI development. Voter pressure in response to widespread job displacement could drive policymakers to take action.

Another potential threshold could be reached if AI demonstrates the capability to create or design bioweapons. Bioweapons represent one of the most dangerous potential threats associated with AI, as they can replicate, evolve, and spread across borders uncontrollably once released. If AI systems show the ability to generate or significantly advance bioweapon design, it might justify a moratorium on certain types of AI research or development.

Source: BLS. So far, employment seems to be going strong.

Societal destabilization could also serve as a trigger for reconsidering AI development. While current rhetoric might suggest heightened political tensions, the willingness for large-scale civil conflict in many stable democracies remains low. Even with AI being used in misinformation campaigns, democratic institutions have shown resilience. However, if this trend were to reverse dramatically—for instance, if AI-generated disinformation became so sophisticated and pervasive that it severely undermined social cohesion or democratic processes—it might warrant a pause or stringent regulation of AI development.

It’s important to note that these potential triggers are not clear-cut thresholds but rather points at which serious consideration of an AI pause or increased regulation might become necessary. The challenge lies in identifying these tipping points early enough to take effective action, but not so early that we stifle beneficial innovations.

I asked my audience “when should AI be paused?” and while my audience is more optimistic on AI innovation, more than 50% agreed that there are conditions under which AI research should be halted.

Moreover, any decision to pause or heavily regulate AI development would need to be implemented on a global scale to be effective, which presents significant geopolitical challenges. Unilateral action by one country or group of countries could simply shift AI development to other regions, potentially creating more risks than it mitigates.

While it’s crucial to monitor AI development closely and be prepared to act if necessary, determining the exact “when” for a potential pause remains a complex and nuanced issue. It requires balancing the potential risks of unchecked AI development against the benefits and innovations that AI can bring to society. Continuous assessment, open dialogue among stakeholders, and flexible, adaptive policies may be more effective than waiting for a single, clear-cut moment to take action.

How?

Understanding how to effectively implement a pause or significant regulation on AI development is a complex challenge that requires careful consideration of historical precedents, geopolitical realities, and the unique nature of AI technology. This section will explore the practical and political hurdles facing any attempt to halt or control AI research and development on a global scale.

Historically, no technology has ever been truly “uninvented” once it has come into existence. This fundamental reality poses a significant challenge to any effort aimed at completely halting AI development. The knowledge and techniques underlying AI are already widely disseminated across the globe, making it nearly impossible to fully suppress or eliminate this technology.

Nuclear arms treaties provide an instructive, albeit imperfect, parallel. These agreements have initiated the process of disarmament, yet nuclear weapons continue to exist decades after the first treaties were signed. The slow progress in eliminating nuclear arsenals underscores the difficulty of controlling powerful technologies once they've been developed. In the case of nuclear arms treaties, many came into effect decades after ICBMs were deployed during the Cold War, highlighting the reactive rather than preemptive nature of such agreements.

One key difference between nuclear weapons and AI is their fundamental nature and application. Nuclear weapons are single-use technologies designed primarily for destruction. In contrast, AI is predominantly a productivity tool with vast creative and assistive potential. Its dual-use nature—being both a powerful productivity enhancer and a potential weapon—further complicates the question of how to implement effective controls.

The international system’s anarchic nature presents another significant hurdle. In the absence of a supreme global authority, any moratorium on AI development would necessarily be voluntary. The chances of achieving universal voluntary compliance are extremely slim, given the economic and strategic advantages that advanced AI capabilities could provide to nations and organizations.

The behavior of major powers in other domains illustrates the challenges of enforcing international agreements. China's actions in the South China Sea, for instance, demonstrate how nations often interpret treaties to suit their interests, ignore inconvenient provisions, or create narratives that justify their non-compliance. This reality suggests that even if an AI moratorium were agreed upon, ensuring adherence would be extraordinarily difficult.

Attempting to enforce compliance through threats of force or severe economic consequences, such as cutting off shipping to and from a non-compliant nation, would likely be viewed as an act of war. Such actions could lead to dangerous escalations in international tensions, potentially creating more immediate threats than the long-term risks associated with AI development.

Given these challenges, a more realistic approach might be to look at precedents like the 1967 Outer Space Treaty, which banned the placement of nuclear weapons in space. While it falls short of the comprehensive pause advocated by the AI Pause movement, it demonstrates the potential for establishing “red lines” or boundaries that nations agree not to cross. This type of agreement might be more achievable in the context of AI development, focusing on specific applications or development paths that are deemed too risky.

USA and USSR agreed not to station nuclear weapons in orbit or on any celestial body, but only after we had achieved MAD.

A similar approach for AI might involve international agreements on specific aspects of AI development. For instance, nations might agree to ban or strictly regulate AI systems designed for autonomous warfare, or to establish clear ethical guidelines for AI research in sensitive areas like biotechnology. While this wouldn’t constitute a complete pause, it could help mitigate some of the most pressing risks associated with unchecked AI advancement.

Another potential avenue could be the creation of an international body to monitor AI development and facilitate dialogue between nations. While such an organization wouldn't have the power to enforce a moratorium, it could serve as a platform for sharing information, discussing potential risks, and coordinating responses to emerging challenges.

Implementing effective controls on AI development is likely to require a multifaceted approach:

1. Crafting international agreements that focus on specific, high-risk areas of AI development.

2. Establishing global standards and ethical guidelines for AI research and application.

3. Creating international monitoring and dialogue mechanisms to foster transparency and cooperation.

4. Investing in AI safety research to develop technical solutions to potential risks.

5. Engaging in public education to build broader understanding and support for responsible AI development.

While these measures fall short of a complete pause, they represent more achievable goals that could help mitigate some of the risks associated with rapid AI advancement. The key will be finding a balance between fostering beneficial AI development and implementing sufficient safeguards to prevent or mitigate potential harms.

While a comprehensive global pause on AI development appears highly unlikely and potentially unenforceable, there are still avenues for meaningful international cooperation and regulation in this field. The focus should be on identifying specific areas of concern, establishing clear guidelines and "red lines," and fostering ongoing dialogue and cooperation between nations and stakeholders in the AI community. This approach, while not as absolute as some pause advocates might desire, offers a more realistic path toward responsible AI development in our complex geopolitical landscape.

Conclusion

The Pause AI movement, while rooted in genuine concerns about the potential risks of artificial intelligence, faces significant challenges in achieving its goals. To strengthen their position and increase their effectiveness, advocates for an AI pause need to adopt a more nuanced, informed, and pragmatic approach.

First and foremost, pause advocates should prioritize educating themselves on a broader range of relevant disciplines. A deep understanding of history, geopolitics, economics, military strategy, and the lifecycle of technological development is crucial for crafting realistic and effective proposals. Many of the most vocal AI safety and pause advocates currently lack expertise in these areas, leading to oversimplified views of complex issues. By broadening their knowledge base, they can develop more sophisticated arguments that address the multifaceted nature of AI development and its potential impacts. At present, their conversations are limited to an echo chambers, wherein they believe only a narrow preselected list of “experts” such as Yudkowsky, Leahy, Yampolskiy, and Bostrom.

Secondly, the movement would benefit from adopting a more measured and realistic approach. The use of hyperbole and alarmist rhetoric, while attention-grabbing, often alienates potential allies and undermines the credibility of the movement. Key advocates, such as Eliezer Yudkowsky, suggest we should “airstrike data centers” which provokes groans and eye-rolls from academics and Department of Defense insiders. Ultimately, his rhetoric, as well as those who parrot him, have done more to undermine the AI safety conversation than bolster it. Instead of shouting “into the void,” as they’ve characterized their Sisyphean task, pause advocates should focus on constructive engagement with key stakeholders. This means replacing confrontational tactics and inflammatory language with reasoned dialogue and a willingness to collaborate.

To truly effect change, pause advocates need to ingratiate themselves with the establishments they seek to influence: business, academia, government, military, and technology sectors. This requires a significant shift in approach—adopting a more reasonable tone, abandoning hyperbolic claims, and learning to navigate the complexities of these institutions. By taking an inside view, advocates can gain a deeper understanding of the intricacies and ramifications of their proposals, leading to more practical and implementable solutions.

“No, I think we’re going to die no matter what we do, but pausing would buy us time in the rare event that we succeed” ~ A common belief espoused by hardline AI safety advocates

The aim of this critical appraisal is not to dismiss the concerns raised by the AI pause movement, but rather to strengthen their argument and increase their chances of making a meaningful impact. By addressing these shortcomings, the movement can evolve from a fringe group shouting alarming predictions to a respected voice in the ongoing dialogue about AI development and safety.

As was written in the Atlantic, the AI doomsday prophets had their 15 minutes of fame, and they wasted it. My personal interpretation is that they never had any substance to begin with, and as a friend of mine observed “It’s like their entire movement dissolved on contact with reality.” Their skepticism of science, belief in their own insular narratives, and increasingly unbelievable rhetoric continues to undermine their credibility. They demand action now in a Nirvana fallacy, stating that halting AI research is the only potential solution, and that no other alternative is good enough. This hardline stance (bordering on zealous extremism) is what provokes snickers and eye-rolls from academics, Silicon Valley, and defense insiders whenever I speak to them about AI safety.

TLDR: AI Safety has a serious optics and PR problem.

They don’t seem to be aware of how crazy they look from the outside.