The Alignment Problem: Computer Science Meets Analytic Philosophy
TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong. Minimal Viable Safety The Task The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do. For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This is probably perfectly safe [1] , but also perfectly useless. We can decide on an arbitrary task such as "construct Lean proofs for given mathematical theorems" and declare victory once we have safe AI that does it with superhuman skill [2] . However, that wouldn't address the danger. First, there would be incentives to use AI for tasks to which our solution doesn't apply. Second, even if we had precise safety guidelines for any task that anyone cares about, there would always be actors that ignore the guidelines, and we probably need Artificial Superintelligence (ASI) to reliably stop them. If anyone, anywhere, creates an unrestricted unaligned ASI agent (more details about the word "agent" later), without an aligned ASI agent already in place to counteract it, then a catastrophic outcome is highly likely. I will not go into the reasons here: this has been discussed elsewhere at length, see for example the books I cited in the preface. Therefore, this event is something which we must never allow to happen. This is the main threat model that I set out to address. Imagine a world in which anyone with sufficient resources has the ability to destroy the world. Like today, many will foolishly believe that instead of destroying the world, this act will bring them power and glory. Moreover, as computing hardware improves and new algorithms are developed, the amount of resources for this act will go down from year to year. Even assuming best-case regulation and international agreements, how long until someone does it? Can mere humans build a society which will be foolproof against such a risk over the course of generations? Resilient to any political upheavals that might come, and undoubtedly will? I find it vanishingly unlikely. Eventually we will need the help of ASI itself. Therefore, the minimal task that safe AI must be able to solve is "create a reliable global defense system against dangerous AI" (in the following, GDS). The Problem Consider the consequences of this task for the capabilities that the AI needs to have. It needs to reason about the entire planet [3] . It needs to reason about all aspects of the problem, including scientific, technological, social, political and moral [4] . It needs to have preferences about the state of the world. It needs to make long-term plans (guided by these preferences) that would play out decades, if not centuries, into the future. And it needs to do all that on a superhuman level. In this light, here is how I (informally) define the "technical AI alignment problem". The problem is designing an AI system that can bring scalable (and ultimately superhuman) capability to bear on problems that require multi-domain reasoning and open-world long-term planning with far-reaching consequences, all the while staying aligned with the intentions and preferences of its human users. The Methodology What kind of methodology is appropriate for this problem? In the past, I argued that it requires emphasis on deep theoretical understanding, rather than the incremental trial-and-error approach that is employed in the AI industry today. I won't rehash the entire debate here, but let me recap the core argument. The nature of the incremental approach is as follows: we build a system, then we find problems [5] , then we make changes to fix these problems to the extent necessary for commercial viability in the current legal environment. We make some improvements; new problems arise (sometimes because of the improvements, sometimes because of new applications or other reasons); rinse and repeat. This approach utterly fails if the "problems" result in AI takeover and possibly human extinction: there is nobody left to fix them. In order for the incremental approach to succeed in preventing catastrophe, we need to reach a point where no new substantial problems arise, even inside the lab (before the system is deployed). Moreover, we need to reach this point before AI capabilities are sufficient for AI to escape our control. There are few reasons to expect such a miracle. On the contrary, sufficiently advanced AI might understand that it is in its interest to "lie low" (i.e. avoid visibly misaligned behaviors) until its capabilities cross the relevant threshold, such that the problems will only become apparent too late. In contrast, the sane approach is creating the the
TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong. Minimal Viable Safety The Task The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do. For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This is probably perfectly safe [1] , but also perfectly useless. We can decide on an arbitrary task such as "construct Lean proofs for given mathematical theorems" and declare victory once we have safe AI that does it with superhuman skill [2] . However, that wouldn't address the danger. First, there would be incentives to use AI for tasks to which our solution doesn't apply. Second, even if we had precise safety guidelines for any task that anyone cares about, there would always be actors that ignore the guidelines, and we probably need Artificial Superintelligence (ASI) to reliably stop them. If anyone, anywhere, creates an unrestricted unaligned ASI agent (more details about the word "agent" later), without an aligned ASI agent already in place to counteract it, then a catastrophic outcome is highly likely. I will not go into the reasons here: this has been discussed elsewhere at length, see for example the books I cited in the preface. Therefore, this event is something which we must never allow to happen. This is the main threat model that I set out to address. Imagine a world in which anyone with sufficient resources has the ability to destroy the world. Like today, many will foolishly believe that instead of destroying the world, this act will bring them power and glory. Moreover, as computing hardware improves and new algorithms are developed, the amount of resources for this act will go down from year to year. Even assuming best-case regulation and international agreements, how long until someone does it? Can mere humans build a society which will be foolproof against such a risk over the course of generations? Resilient to any political upheavals that might come, and undoubtedly will? I find it vanishingly unlikely. Eventually we will need the help of ASI itself. Therefore, the minimal task that safe AI must be able to solve is "create a reliable global defense system against dangerous AI" (in the following, GDS). The Problem Consider the consequences of this task for the capabilities that the AI needs to have. It needs to reason about the entire planet [3] . It needs to reason about all aspects of the problem, including scientific, technological, social, political and moral [4] . It needs to have preferences about the state of the world. It needs to make long-term plans (guided by these preferences) that would play out decades, if not centuries, into the future. And it needs to do all that on a superhuman level. In this light, here is how I (informally) define the "technical AI alignment problem". The problem is designing an AI system that can bring scalable (and ultimately superhuman) capability to bear on problems that require multi-domain reasoning and open-world long-term planning with far-reaching consequences, all the while staying aligned with the intentions and preferences of its human users. The Methodology What kind of methodology is appropriate for this problem? In the past, I argued that it requires emphasis on deep theoretical understanding, rather than the incremental trial-and-error approach that is employed in the AI industry today. I won't rehash the entire debate here, but let me recap the core argument. The nature of the incremental approach is as follows: we build a system, then we find problems [5] , then we make changes to fix these problems to the extent necessary for commercial viability in the current legal environment. We make some improvements; new problems arise (sometimes because of the improvements, sometimes because of new applications or other reasons); rinse and repeat. This approach utterly fails if the "problems" result in AI takeover and possibly human extinction: there is nobody left to fix them. In order for the incremental approach to succeed in preventing catastrophe, we need to reach a point where no new substantial problems arise, even inside the lab (before the system is deployed). Moreover, we need to reach this point before AI capabilities are sufficient for AI to escape our control. There are few reasons to expect such a miracle. On the contrary, sufficiently advanced AI might understand that it is in its interest to "lie low" (i.e. avoid visibly misaligned behaviors) until its capabilities cross the relevant threshold, such that the problems will only become apparent too late. In contrast, the sane approach is creating the the