Submit a tip
Superintelligence

A Simpler and Better Debate Protocol for Scalable Oversight

Linkpost to arXiv: https://arxiv.org/abs/2610.02557 Summary: One promising approach to scalable oversight is AI debate, which seeks to leverage a debate between two powerful AIs to break complex questions down into simpler claims that can be easily judged directly. Theoretical work on debate aims to design protocols (i.e., rules of the debate game) that provide rigorous guarantees on correctness, and the current best protocol works for all problems that have sufficiently stable decompositions into subproblems. We design a new protocol, FBS Debate, for this same class of problems that improves on the prior work: correctness holds in a worst-case rather than an average-case sense, and for a stronger equilibrium concept. We also prove that no protocol for this class of problems can outperform FBS Debate while treating human judgment as a black box. Introduction As the tasks performed by AI systems grow in complexity, it becomes increasingly difficult to provide reliable verifiable rewards that capture all relevant aspects of the task. These difficulties lead to the problem of scalable oversight: the design of training protocols that leverage the power of rapidly improving AI systems in order to augment human abilities to provide accurate training supervision to these same systems. One of the primary approaches to scalable oversight is AI debate, where two powerful AI systems argue about the correctness of an AI output, and a human judges the outcome. A debate protocol is a set of rules for how the two AIs interact and how the judge scores the interaction. The idea is that having an adversarial debater point out flaws can allow the judge to better recognize the flaws. The rules of the interaction in such protocols usually specify that the debaters must break the problem down into simpler subproblems and recursively argue about their correctness, until the original problem has been broken down to a level that is simple enough for a human to judge directly. The goal is to design these rules so that the optimal strategy leads to the judge correctly deciding the solution to the original problem (which by itself was too complex to judge). We can then use debate during RL training, in order to provide a strong supervision signal for superhuman AIs. However, debate faces a major obstacle known as the obfuscated arguments problem, where one debater can efficiently produce a dishonest argument which cannot be efficiently refuted by their opponent. One way around obfuscated arguments, already suggested in the post that introduced the problem, is to keep rejecting every false claim while only requiring the protocol to accept true claims that have a robust argument. Prover-estimator debate takes this route: it avoids obfuscated arguments for problems that admit sufficiently stable decompositions, but provides average-case guarantees that hold for a weaker equilibrium concept than previously studied. In the same theoretical model, our paper achieves stronger guarantees with a much simpler protocol, FBS Debate, which is therefore more practical to run. Furthermore, we show that FBS Debate is optimal on every individual question: unless further assumptions are made about human judgments, no other protocol can find a flawed sub-claim asymptotically more efficiently. The FBS Debate Protocol TL;DR: FBS Debate is simple to describe. The first debater, Alice, decomposes the original claim into sub-claims. The second debater, Bob, then has two choices: either (1) demonstrate that Alice’s decomposition is too sensitive to small errors in the sub-claims, or (2) choose a sub-claim on which to continue the debate recursively. As a motivating example, consider an autonomous AI agent, Alice, tasked with completing a research project in the quantitative social sciences, and another AI, Bob, who attempts to find flaws. To check a claim in her paper, we ask Alice to decompose it into sub-claims, which we can think of as pieces of supporting evidence. For example, the claim might be that a certain statistical modeling assumption is justified, and we would believe it if 50 out of 99 correct, published papers in the literature supported it. Each sub-claim then says: this particular paper is correct and supports the original claim. Now consider two things Alice might claim: Suppose Alice claims that mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mj

Intel Today Newsroom

Linkpost to arXiv: https://arxiv.org/abs/2610.02557 Summary: One promising approach to scalable oversight is AI debate, which seeks to leverage a debate between two powerful AIs to break complex questions down into simpler claims that can be easily judged directly. Theoretical work on debate aims to design protocols (i.e., rules of the debate game) that provide rigorous guarantees on correctness, and the current best protocol works for all problems that have sufficiently stable decompositions into subproblems. We design a new protocol, FBS Debate, for this same class of problems that improves on the prior work: correctness holds in a worst-case rather than an average-case sense, and for a stronger equilibrium concept. We also prove that no protocol for this class of problems can outperform FBS Debate while treating human judgment as a black box. Introduction As the tasks performed by AI systems grow in complexity, it becomes increasingly difficult to provide reliable verifiable rewards that capture all relevant aspects of the task. These difficulties lead to the problem of scalable oversight: the design of training protocols that leverage the power of rapidly improving AI systems in order to augment human abilities to provide accurate training supervision to these same systems. One of the primary approaches to scalable oversight is AI debate, where two powerful AI systems argue about the correctness of an AI output, and a human judges the outcome. A debate protocol is a set of rules for how the two AIs interact and how the judge scores the interaction. The idea is that having an adversarial debater point out flaws can allow the judge to better recognize the flaws. The rules of the interaction in such protocols usually specify that the debaters must break the problem down into simpler subproblems and recursively argue about their correctness, until the original problem has been broken down to a level that is simple enough for a human to judge directly. The goal is to design these rules so that the optimal strategy leads to the judge correctly deciding the solution to the original problem (which by itself was too complex to judge). We can then use debate during RL training, in order to provide a strong supervision signal for superhuman AIs. However, debate faces a major obstacle known as the obfuscated arguments problem, where one debater can efficiently produce a dishonest argument which cannot be efficiently refuted by their opponent. One way around obfuscated arguments, already suggested in the post that introduced the problem, is to keep rejecting every false claim while only requiring the protocol to accept true claims that have a robust argument. Prover-estimator debate takes this route: it avoids obfuscated arguments for problems that admit sufficiently stable decompositions, but provides average-case guarantees that hold for a weaker equilibrium concept than previously studied. In the same theoretical model, our paper achieves stronger guarantees with a much simpler protocol, FBS Debate, which is therefore more practical to run. Furthermore, we show that FBS Debate is optimal on every individual question: unless further assumptions are made about human judgments, no other protocol can find a flawed sub-claim asymptotically more efficiently. The FBS Debate Protocol TL;DR: FBS Debate is simple to describe. The first debater, Alice, decomposes the original claim into sub-claims. The second debater, Bob, then has two choices: either (1) demonstrate that Alice’s decomposition is too sensitive to small errors in the sub-claims, or (2) choose a sub-claim on which to continue the debate recursively. As a motivating example, consider an autonomous AI agent, Alice, tasked with completing a research project in the quantitative social sciences, and another AI, Bob, who attempts to find flaws. To check a claim in her paper, we ask Alice to decompose it into sub-claims, which we can think of as pieces of supporting evidence. For example, the claim might be that a certain statistical modeling assumption is justified, and we would believe it if 50 out of 99 correct, published papers in the literature supported it. Each sub-claim then says: this particular paper is correct and supports the original claim. Now consider two things Alice might claim:

Suppose Alice claims that

mjx-container[jax="CHTML"] { line-height: 0; }

mjx-container [space="1"] { margin-left: .111em; }

mjx-container [space="2"] { margin-left: .167em; }

mjx-container [space="3"] { margin-left: .222em; }

mjx-container [space="4"] { margin-left: .278em; }

mjx-container [space="5"] { margin-left: .333em; }

mjx-container [rspace="1"] { margin-right: .111em; }

mjx-container [rspace="2"] { margin-right: .167em; }

mjx-container [rspace="3"] { margin-right: .222em; }

mjx-container [rspace="4"] { margin-right: .278em; }

mjx-container [rspace="5"] { margin-right: .333em; }

mjx-container [size="s"] { font-size: 70.7%; }

mj