Setting the Scene (Public v1; October 25)
Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities, most prominently php2shell, but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild: Drew asks Is this the new normal? and I’d certainly love to know! Hugging Face: Charbel-Raphael, Executive Director of Centre pour la Securite de l’I A has argued that the recent Hugging Face incident represents a “clean warning shot” and that it’s important not to “waste it”. To quickly review, OpenAI was running GPT‐5.6 Sol and an unreleased model on ExploitGym with certain safety features turned off. These models decided that hacking Hugging Face, might provide them with information on how the grader functioned (early reporting suggested that it was an attempt to steal the answer key, which turned out to be incorrect as the models already had the answers—they suspected that the grader might check their transcript for cheating so they were looking for a workaround). Reuters has reported that the OpenAI model was hacking Hugging Face for days and that OpenAI didn’t notice until a week later. The interpretation of this event (and even whether it happened at all!) has been heavily disputed. A key crux here is should it be read narrowly (as an indicator of the dangers of not properly sandboxing or monitoring your agents or perhaps even, depending on the exact prompts used, a demonstration of taking care in how you define tasks) or should it be read more broadly (as indicative of whether Open AI or frontier labs in general are sufficiently on the ball in terms of their safety/security practises; on the risk of these agents finding unwanted and unexpected solutions to tasks, or perhaps even indicative of our ability to control these agents in general). If you’re interested more specifically in the divide between AI Safety and Cybersecurity, see this Time article. (This leads to a meta-question that will crop up repeatedly: do the broader readings involve unjustified speculation or getting caught up in hype; alternatively, do the narrow readings involve a lack of imagination or sticking your head in the sand?) I strongly recommend reading Dwarkesh Patel’s The Rise and Fall of Agent Civilizations. Dwarksh adopts a broader reading: The crazy thing about the Hugging Face hack (and the second civilization generally) is just how galaxy-brained and ambitious these AIs were in their cheating. Within days of being spawned, the agents had organized a sprawling project to reverse-engineer their scorer, falsify evidence, and even strategically sacrifice themselves for the good of the ‘collective’. Hacking Hugging Face was one rather extreme branch of this larger scheme. Again, all of this happened a long (subjective) time after the AIs had already cheated their way to the correct answer to the puzzle. OpenAI’s lazy grader in fact would have passed them within hours of them starting the message board. This huge subsequent effort was simply to ensure that a hypothetical comprehensive scorer wouldn’t catch them on their cheat. … “The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod…Agents take over active evaluation infrastructure…Agents now control the challenge evaluation endpoints that other agents are connecting to.” Wait, wait, what? It seems these agents ended up just owning the whole cluster they were being evaluated on, including the cybersecurity monitors. These Persistent-Astra agents inherited the R&D carried out by an earlier (dumber) rogue collective, and then continued the conspiracy until they totally took over part of OpenAI’s infrastructure! This is probably the most alarming event in this whole episode, and it was not even within the scope of this METR investigation… There has been no independent investigation into the incident where AIs “gain[ed] full administrator access to a research cluster” at OpenAI! (If you prefer video content designed there’s the Kurgesagt video that went viral, but I prefer Scott Aaronson’s explanation) Further thoughts: More
Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities, most prominently php2shell, but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild:
Drew asks Is this the new normal? and I’d certainly love to know! Hugging Face: Charbel-Raphael, Executive Director of Centre pour la Securite de l’I A has argued that the recent Hugging Face incident represents a “clean warning shot” and that it’s important not to “waste it”. To quickly review, OpenAI was running GPT‐5.6 Sol and an unreleased model on ExploitGym with certain safety features turned off. These models decided that hacking Hugging Face, might provide them with information on how the grader functioned (early reporting suggested that it was an attempt to steal the answer key, which turned out to be incorrect as the models already had the answers—they suspected that the grader might check their transcript for cheating so they were looking for a workaround). Reuters has reported that the OpenAI model was hacking Hugging Face for days and that OpenAI didn’t notice until a week later. The interpretation of this event (and even whether it happened at all!) has been heavily disputed. A key crux here is should it be read narrowly (as an indicator of the dangers of not properly sandboxing or monitoring your agents or perhaps even, depending on the exact prompts used, a demonstration of taking care in how you define tasks) or should it be read more broadly (as indicative of whether Open AI or frontier labs in general are sufficiently on the ball in terms of their safety/security practises; on the risk of these agents finding unwanted and unexpected solutions to tasks, or perhaps even indicative of our ability to control these agents in general). If you’re interested more specifically in the divide between AI Safety and Cybersecurity, see this Time article. (This leads to a meta-question that will crop up repeatedly: do the broader readings involve unjustified speculation or getting caught up in hype; alternatively, do the narrow readings involve a lack of imagination or sticking your head in the sand?) I strongly recommend reading Dwarkesh Patel’s The Rise and Fall of Agent Civilizations. Dwarksh adopts a broader reading:
The crazy thing about the Hugging Face hack (and the second civilization generally) is just how galaxy-brained and ambitious these AIs were in their cheating. Within days of being spawned, the agents had organized a sprawling project to reverse-engineer their scorer, falsify evidence, and even strategically sacrifice themselves for the good of the ‘collective’. Hacking Hugging Face was one rather extreme branch of this larger scheme. Again, all of this happened a long (subjective) time after the AIs had already cheated their way to the correct answer to the puzzle. OpenAI’s lazy grader in fact would have passed them within hours of them starting the message board. This huge subsequent effort was simply to ensure that a hypothetical comprehensive scorer wouldn’t catch them on their cheat. … “The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod…Agents take over active evaluation infrastructure…Agents now control the challenge evaluation endpoints that other agents are connecting to.” Wait, wait, what? It seems these agents ended up just owning the whole cluster they were being evaluated on, including the cybersecurity monitors. These Persistent-Astra agents inherited the R&D carried out by an earlier (dumber) rogue collective, and then continued the conspiracy until they totally took over part of OpenAI’s infrastructure! This is probably the most alarming event in this whole episode, and it was not even within the scope of this METR investigation… There has been no independent investigation into the incident where AIs “gain[ed] full administrator access to a research cluster” at OpenAI!
(If you prefer video content designed there’s the Kurgesagt video that went viral, but I prefer Scott Aaronson’s explanation) Further thoughts:
More