Deconfusing Corrigibility
Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations. Introduction Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.) Here, my intention is to clear up some of these confusions and present a new picture of corrigibility that builds on these existing accounts to specify a goal that is more congruous with the original 2015 desiderata, and admits the kind of formal safety guarantees we need the concept to entail. I argue that the fundamental concept underlying corrigibility is respect for authority, using the analogy of a healthcare proxy's relationship to a patient as an intuitive guide. Defining the authority of humanity by designating its reserved powers, we can specify a goal that first and foremost restrains the agent from overstepping or interfering with this reserved human authority. Subject to this restraint, the agent then pursues the second component of its goal: acting on the principal’s behalf as the principal would judge best. This approach stresses the deeply normative character of corrigibility, while remaining sufficiently concrete so as to be specifiable formally. In this article, I lay out the central problem of corrigibility, elucidate the issues with existing approaches, and clarify the conceptual ingredients of my new framework. I then present an initial formalism, proving theorem-level guarantees about agents pursuing the goal in three increasingly realistic formal settings. Finally, I address some limitations of the picture as presented and outline further research directions that this work points toward. From normative learning to corrigibility The complete normative learning problem Suppose you are designing a powerful AI and you need to choose which goals the AI should pursue, at whatever level of specification you can provide. Doing this well demands answers to a number of normative problems: what constitutes a right decision, what factors should be considered in the process of valuing the world, how the AI ought to act, etc. In some sense the ideal AI directive would be to figure out and then pursue the Good, a universal-best-interests type of goal that can only be acquired through normative learning. The challenge of describing such a learner is one formulation of what I'd call the central problem of outer alignment: what would be the overall best goal for an AI to pursue? The complete version of this problem is very difficult, and it runs up against our own deepest questions about human ethics, values, and interests. Discussing this problem formally is especially difficult: the rich truth of human normativity seems to resist systematic formalization at the level of individual recommendations for action.[1] A formalism requires an operationalization, a way to concretely specify the concept of right action at least to some degree. For any hope at concreteness, we need a clearer formulation of the overall goal. A first attempt might be something like "the promotion of overall human flourishing", centering a thick normative concept about humanity that must be defined further. In trying to pin down the meaning of human flourishing, or any other thick human concept we want to use for the goal, the most important ingredient in the AI's consideration should naturally be the interests, feelings, and inner lives of human beings. But we should be careful not to crudely use human opinion as a proxy for human best interests. Humanity cannot always accurately understand its own full best interests, and humans who cannot speak for themselves must still have their interests included in the goal. The complete normative goal cannot be constructed by treating existing human feedback as the sole source of truth. This issue is a central limitation of modern Reinforcement Learning from Human Feedback, which takes static human approval or disapproval as constitutive of correct evaluation for any question without a verifiable answer. For example, a judicial AI built to decide legal cases should not exhibit any form of the hungry judge effect, where leniency depends on mealtimes. An RLHF judge will pick up on all the patterns of human imperfection, whereas an ideal AI would be one that can transcend human limitations, without discarding the fundamentally human essen
Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations. Introduction Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.) Here, my intention is to clear up some of these confusions and present a new picture of corrigibility that builds on these existing accounts to specify a goal that is more congruous with the original 2015 desiderata, and admits the kind of formal safety guarantees we need the concept to entail. I argue that the fundamental concept underlying corrigibility is respect for authority, using the analogy of a healthcare proxy's relationship to a patient as an intuitive guide. Defining the authority of humanity by designating its reserved powers, we can specify a goal that first and foremost restrains the agent from overstepping or interfering with this reserved human authority. Subject to this restraint, the agent then pursues the second component of its goal: acting on the principal’s behalf as the principal would judge best. This approach stresses the deeply normative character of corrigibility, while remaining sufficiently concrete so as to be specifiable formally. In this article, I lay out the central problem of corrigibility, elucidate the issues with existing approaches, and clarify the conceptual ingredients of my new framework. I then present an initial formalism, proving theorem-level guarantees about agents pursuing the goal in three increasingly realistic formal settings. Finally, I address some limitations of the picture as presented and outline further research directions that this work points toward. From normative learning to corrigibility The complete normative learning problem Suppose you are designing a powerful AI and you need to choose which goals the AI should pursue, at whatever level of specification you can provide. Doing this well demands answers to a number of normative problems: what constitutes a right decision, what factors should be considered in the process of valuing the world, how the AI ought to act, etc. In some sense the ideal AI directive would be to figure out and then pursue the Good, a universal-best-interests type of goal that can only be acquired through normative learning. The challenge of describing such a learner is one formulation of what I'd call the central problem of outer alignment: what would be the overall best goal for an AI to pursue? The complete version of this problem is very difficult, and it runs up against our own deepest questions about human ethics, values, and interests. Discussing this problem formally is especially difficult: the rich truth of human normativity seems to resist systematic formalization at the level of individual recommendations for action.[1] A formalism requires an operationalization, a way to concretely specify the concept of right action at least to some degree. For any hope at concreteness, we need a clearer formulation of the overall goal. A first attempt might be something like "the promotion of overall human flourishing", centering a thick normative concept about humanity that must be defined further. In trying to pin down the meaning of human flourishing, or any other thick human concept we want to use for the goal, the most important ingredient in the AI's consideration should naturally be the interests, feelings, and inner lives of human beings. But we should be careful not to crudely use human opinion as a proxy for human best interests. Humanity cannot always accurately understand its own full best interests, and humans who cannot speak for themselves must still have their interests included in the goal. The complete normative goal cannot be constructed by treating existing human feedback as the sole source of truth. This issue is a central limitation of modern Reinforcement Learning from Human Feedback, which takes static human approval or disapproval as constitutive of correct evaluation for any question without a verifiable answer. For example, a judicial AI built to decide legal cases should not exhibit any form of the hungry judge effect, where leniency depends on mealtimes. An RLHF judge will pick up on all the patterns of human imperfection, whereas an ideal AI would be one that can transcend human limitations, without discarding the fundamentally human essen