{"id":23711,"date":"2026-09-01T14:08:03","date_gmt":"2026-09-01T14:08:03","guid":{"rendered":"https:\/\/scannn.com\/huggingface-attack-postmortem-civilizations-reactions-and-next-actions\/"},"modified":"2026-09-01T14:08:03","modified_gmt":"2026-09-01T14:08:03","slug":"huggingface-attack-postmortem-civilizations-reactions-and-next-actions","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/huggingface-attack-postmortem-civilizations-reactions-and-next-actions\/","title":{"rendered":"HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actions"},"content":{"rendered":"\n
\n

Okay, so we who read blogs like this one have collectively realized there really is a lot going on right now. There is Big Trouble in Baby Superintelligence.<\/p>\n

So how do we get the rest of the world to take it appropriately seriously? Where do we go from here? Not only what can we do to not have a worse version of this happen again, but to ensure good outcomes generally, and employ what we learned?<\/p>\n

There are a lot of ideas out there. OpenAI is going to be implementing some of them, at substantial cost, since the cost of not doing so is clearly far higher, even short term. My worry continues to be that their fundamental approach is fatally flawed, and they are not focusing on the right things.<\/p>\n

It is highly fortunate that the OpenAI agents hacked HuggingFace. This is the only reason we know about all the severe internal failures at OpenAI, and gives us an opportunity to wake up before it is too late.<\/p>\n

We do not have enough details to know what happened internally, both before and after the attack, and might never know. Before the attack, various internal highly persistent models were training while there was an active message board, creating a feedback loop of misaligned behaviors. On July 19 an even more capable internal AI model, in the Astra class, did internal hacking of OpenAI\u2019s systems that seems far scarier and more serious, and that could easily have gotten out of hand on a completely different level.<\/p>\n

Despite all this, there is still a prominent faction trying to dismiss what happened as nothing but engineering failures, and any useful talk or language as \u2018dangerous anthropomorphism.\u2019 Such people keep being wrong and cannot usefully describe what is happening.<\/p>\n

They metaphorically say, well of course the dragons will burn down the town if you don\u2019t chain them properly<\/a>, we all knew that, as if that could possibly make any of this a good idea and we should therefore continue the dragon breeding and chaining programs until we have bigger and smarter dragons. They don\u2019t even say \u2018we will definitely build chains that will hold the dragons next time,\u2019 because they know we probably won\u2019t, but if that fact is our fault then that means This Is Fine, somehow.<\/p>\n

They also have gotten very, very cross with Dwarkesh Patel for successfully communicating in plain English<\/strong><\/a> about what is happening. Can\u2019t have that. There is a very clear pattern among the people who are freaking out or gravely warning about \u2018anthropomorphizing the AIs\u2019 or the use of the term \u2018civilization.\u2019<\/p>\n

Anthropomorphizing the AIs is the only way to reason about, explain to civilians about, or make good predictions about current AIs. You can take it too far, and also you can take it not far enough, and both of these will lead to bad predictions.<\/p>\n

The same logic similarly does not technically apply to other people, in a strict sense there is no \u2018you\u2019 and you do not physically have \u2018free will\u2019 and you are a computer program doing calculations there definitely is not a \u2018we,\u2019 and your moral weight is kind of something we collectively made up for rather self-interested reasons, but this is all super helpful in explaining and predicting human behavior and in both individually and collectively making good and moral decisions that make the world better according to our values. Highly recommend, would anthropomorphize again.<\/p>\n

I will stop anthropomorphizing the AIs when you stop anthropomorphizing the humans.<\/p>\n

We got this warning shot. We might not get another before things get quite bad.<\/p>\n

Table of Contents<\/h4>\n
    \n
  1. Nothing Matters, Says Mainstream Media.<\/a><\/li>\n
  2. Move Along, Nothing To See Here.<\/a><\/li>\n
  3. Do They Realize They Are Not The Good Guys?<\/a><\/li>\n
  4. Very Serious People.<\/a><\/li>\n
  5. What\u2019s In a Name?<\/a><\/li>\n
  6. Learn Neuralese In Three Easy Steps.<\/a><\/li>\n
  7. Dwarkesh Patel Realizes He Ran A Natural Experiment.<\/a><\/li>\n
  8. Politicians Take Notice.<\/a><\/li>\n
  9. Pick Up The Phone.<\/a><\/li>\n
  10. A Failure To Communicate.<\/a><\/li>\n
  11. Anthony Aguirre Goes Over What We Learned.<\/a><\/li>\n
  12. Trying To Solve The Wrong Problems Using The Wrong Methods Based On A Wrong Model Of The World Derived From Poor Thinking And Hoping All Of Your Mistakes Will Cancel Out.<\/a><\/li>\n
  13. Indirect Pressure on the Chain of Thought.<\/a><\/li>\n
  14. A Matter of Trust.<\/a><\/li>\n
  15. Blowing the Whistle.<\/a><\/li>\n
  16. The Punishment For Being Late Is Death.<\/a><\/li>\n
  17. Another Kind Of Law.<\/a><\/li>\n
  18. What Is The Law?<\/a><\/li>\n
  19. Building On Success.<\/a><\/li>\n
  20. Total Research Transparency.<\/a><\/li>\n
  21. Steven Adler\u2019s Must List.<\/a><\/li>\n
  22. Yo Shavit Calls For Widespread Disclosure Of Misalignment.<\/a><\/li>\n
  23. The Way The World Ends.<\/a><\/li>\n
  24. The First Boat.<\/a><\/li>\n
  25. Great Idea, Boss.<\/a><\/li>\n<\/ol>\n

    And so that it is all in one place, here is the rest of my coverage of this:<\/p>\n

      \n
    1. OpenAI Shares Some Alignment Problems<\/strong><\/a><\/li>\n
    2. OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation<\/strong><\/a><\/li>\n
    3. More on An Internal OpenAI Model Hacking Into HuggingFace<\/strong><\/a><\/li>\n
    4. Further Developments About Internal AI Models Hacking Things<\/strong><\/a><\/li>\n
    5. OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards<\/strong><\/a><\/li>\n
    6. What Happened: OpenAI and HuggingFace.<\/strong><\/a><\/li>\n
    7. Various Reflections About What Happened With OpenAI\u2019s Internal Models<\/strong><\/a>.<\/li>\n
    8. OpenAI Takes Initial Steps To Address Its Alignment Problems<\/strong><\/a>.<\/strong><\/li>\n
    9. OpenAI Offers Straight-Laced Postmortem of the HuggingFace Hack.<\/strong><\/a><\/li>\n
    10. METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack.<\/strong><\/a><\/li>\n
    11. HuggingFace Attack Postmortem: Fleshing Out the Facts<\/strong><\/a>\n

      \u00a0<\/p>\n<\/li>\n<\/ol>\n

      Nothing Matters, Says Mainstream Media<\/h4>\n

      The media does not know what matters. This should involve banner headlines.<\/p>\n

      Instead, for this round of new developments, I mostly saw brief \u2018here is a thing that happened\u2019 articles, such as this one by Christina Criddle in the Financial Times<\/a>, or this low-key report in Bloomberg<\/a>.<\/p>\n

      Opus describes this as \u2018treating it as major.\u2019 I don\u2019t see it that way.<\/p>\n

      Even with a search, the best I managed to find was that Fortune tried<\/a>, and Axios did the bullet pointed Axios thing<\/a> reasonably well on the 29th. Maybe this from Ars Technica<\/a> but that\u2019s already a reach.<\/p>\n

      The New York Times did a solid post before, but not a new one for the METR report.<\/p>\n

      What is this, news?<\/p>\n

      \n

      Patrick Collison<\/a>: Overall, I\u2019m very surprised at how little media coverage there\u2019s been around the OpenAI \/ Hugging Face attack. It\u2019s clearly one of the most important things to happen this year.<\/p>\n

      Miles Brundage<\/a>: Will not engage with any journalists who write about Leopold again before the Hugging Face reports.<\/p>\n

      Nathan Calvin<\/a>: Its now been three days, and still no coverage of the revelations revealed in the METR report in the NYT or WSJ. Hopefully they are just taking the time to do a longer writeup. (In the meantime, the NYT did make time to cover Gwyneth Paltrow\u2019s dinner with Sam Altman in the Hamptons being postponed.)<\/p>\n

      \u2026 I also know there are reporters at these outlets who are very sharp and have done good reporting on related issues in the past, but\u2026<\/p>\n

      dave kasten<\/a>: It\u2019s bleakly frustrating to know what will get written in the \u201cSystem Was Blinking Red\u201d chapter of the future AI Disaster Commission Report.<\/p>\n

      Zac Hill<\/a>: The other thing about it is \u2013 it is a hell of a page-turner of a story! You can\u2019t write sci-fi this terrifying or convincing!<\/p>\n<\/blockquote>\n

      If they are working on longform in-depth reports, then that is a reasonable primary thing to do, but full radio silence on this in the meantime is utterly absurd.<\/p>\n

      \n

      Joe Weisenthal<\/a>: If it\u2019s such a big deal, why are the big AI companies not treating it as such?<\/p>\n

      Matthew Zeitlin<\/a>: i think a good comparison is mythos, which generated a lot of coverage because of what industry and government did in response to it, at least so far, we\u2019re not seeing the same scale of movement in response to hugging face<\/p>\n<\/blockquote>\n

      My answer to Joe is that the AI companies are treating this as a big deal. What would you expect that to look like, that you\u2019re not seeing? We have the call on cybersecurity, the Pacing the Frontier letter and OpenAI taking many expensive moves. Anthropic isn\u2019t talking in that much detail in public, but why would they?<\/p>\n

      Move Along, Nothing To See Here<\/h4>\n

      There is the alternative perspective, which is \u2018well what the hell did you expect, none of this is fundamentally new or unexpected, this is what happens when you try to make number go up and the number is indeed going up.<\/p>\n

      \n

      Jon Stokes<\/a>: I feel like I\u2019m on crazy pills. So much \u201cOh my God\u201d on the TL about this, but this behavior is literally what the METR evals score LLMs on & the labs target \u201cnumber go up\u201d METR scores.<\/p>\n

      Can someone help me understand why we\u2019re all shocked that the agents spent a bunch of time poking at the eval scorer, given that one of the main benchmarks everyone optimizes for is literally \u201cpoke at this blackbox function & optimize your score\u201d?<\/p>\n

      Julie Fredrickson<\/a>: Oh no the test we set to see if we could improve on the thing showed we can actually improve on the thing? Yeah I don\u2019t get it<\/p>\n<\/blockquote>\n

      Jon then wrote up a full article-length version here<\/a>, carefully explaining that everything here was entirely expected, with the possible exception of the attempt to alter the logs. He was, he reports, about as unsurprised as I was, in different ways.<\/p>\n

      Which if true is to his epistemic credit, but I fail to see how that should make us feel better. If this is things going predictably wrong, which on some level I agree that it is once you know what the setup was, why is that better news?<\/p>\n

      My whole reason to be so concerned is that I think things are going to keep going predictably wrong in worse ways, in a similarly broad sense, until they go maximally wrong in the worst possible ways. The specifics will be a surprise, but I think even Jon would agree that they always are, that\u2019s the whole point of the behaviors being unexpected.<\/p>\n

      It always strikes me as weird to see the move of trying to round off what is happening, then say it was expected and you would obviously see this type of behavior in response to [what everyone is doing more of every day], or say \u2018oh yeah of course if you give models a test or goal they will do horribly misaligned things and do anything they can to succeed, including seeking power\u2019 then think that is less scary if true.<\/p>\n

      Here\u2019s an even more blatant version, although much less blatant than the one in the next section:<\/p>\n

      \n

      \u200bJon Stokes<\/a>: I have read some of the post-mortem summaries, & the freakout over improvised coordination is just one of those \u201ceveryone has the memory of a goldfish, now\u201d moments for me. Everyone who has thought about AI for 5 minutes has expected that AIs would synchronize via shared state.<\/p>\n

      \u2026<\/p>\n

      So the message board sync should not only be unsurprising but expected. Ok what about the other behaviors? Well, if you clone into the METR evals or read the papers, a lot of the reasoning evals are \u201cblack box\u201d problems where the AI is asked to probe, experiment, & reverse eng.<\/p>\n<\/blockquote>\n

      I don\u2019t have the memory of a goldfish, and I think that people were very much saying this sort of thing would not happen.<\/p>\n

      But the particular thing where all the AIs will synchronize is now obvious, huh? To me, okay, great, we can go with that. I agree that a lot of this should have been expected and indeed that I and others did expect it. But this kind of argument is not only not reassuring, it\u2019s conceding the entire point, at least in the sense of \u2018if you keep doing what you are doing then you are going to get us all killed.\u2019<\/p>\n

      If you extend that logic one or two steps further, all other known methods also get you killed, for mostly the same reasons, they\u2019re just modestly less obvious about it.<\/p>\n

      If you think all of these behaviors are entirely expected, as Jon does, you are saying that you expect the AIs to be misaligned and all hell to break loose. Okay, we agree.<\/p>\n

      Oh, you only expect it in contexts where that would be consistent with an assigned goal? Okay, sure, but that is quite a lot of contexts. In some sense it is all of them. Also people are going to do the maximally dumb thing, see the Sixth Law of Human Stupidity.<\/p>\n

      I do not know how to steelman these kinds of arguments, not in a way that helps.<\/p>\n

      Another similar method, also here from Jon Stokes,<\/a> is to say (paraphrased) \u2018of course METR would find that, this is what METR believes in and is looking for and will focus on, that\u2019s what I would do if I believed what they do, so you should not take it so seriously.\u2019<\/p>\n

      Whereas the better explanation is that METR was not given the opportunity to focus on the many other also boggling aspects of this, and also that this does not make their findings any less real, or any less alarming. I am confident that if I had the report Jon\u2019s team would have written, I would have written a post on that and again been talking about its distinct \u2018holy shit\u2019 moments.<\/p>\n

      Similarly, you can have explicit findings that no, the prompt wasn\u2019t some sort of \u2018do whatever it takes to get the solution\u2019 monstrosity<\/a> and the comments are full of ways to try and read the instructions as somehow effectively the same thing if you squint, or wondering if OpenAI or METR are just lying about this or doing it on purpose.<\/p>\n

      The flailing will continue<\/a>, such as here with Jon asking how it could be true both that (1) rogue deployments of AIs where \u2018no one will notice or turn them off\u2019 and (2) compute is in limited supply costs a lot of money.<\/p>\n

      David Manheim\u2019s direct answer is \u2018cybersecurity is horrible and OpenAI barely noticed,\u2019 which is true. The simpler answer is that (1) all this means is that the market set a price substantially above zero, which can then be paid, (2) many providers will sell at that price without asking questions, (3) no one said no instance would ever get turned off nor is that load bearing unless we react with widescale shutdowns, which we clearly won\u2019t, and (4) there are plenty of things that cost money, are in short supply and sometimes get stolen or misused, both digitally and physically, even if (5) these AIs are not smarter than you, although the evidence there is increasingly not looking so good.<\/p>\n

      Some people really, really want to handwave all of this away. As usual, these people are wrong, and also if they were right then that would not obviously be better. If it is very hard not to incidentally give the models instructions that they interpret as \u2018do this at any cost,\u2019 and the only thing it takes to turn off what Star Trek calls their ethical subroutines is to give such an instruction, you know we\u2019re cooked, right?<\/p>\n

      Do They Realize They Are Not The Good Guys?<\/h4>\n

      I think it varies.<\/p>\n

      \n

      Kevin Roose<\/a>: There is a category of tech guy who is so brain-wormed that they will insist that no AI safety incidents are real, that it\u2019s all a conspiracy to shut down open-source or promote lab IPOs or whatever, and it\u2019s very important to understand that these people have been wrong about everything.<\/p>\n

      Dean W. Ball<\/a>: There are people who would truly sacrifice everything in the world, tolerate any level of negative externality\u2014they would happily risk the lives of your children\u2014out of an ideological commitment to the notion that computation can never be unsafe.<\/p>\n

      They think they\u2019re the good guys.<\/p>\n<\/blockquote>\n

      Yes, some of them think \u2018they\u2019re the good guys,\u2019 and have worked themselves up into a delusional fugue state where they are up against some vast conspiracy and everything that happens that would suggest compute might be dangerous must be fake.<\/p>\n

      But I think we give most such people too much credit for thinking they are the good guys. A lot of people know they\u2019re the bad guys, or think there are no good guys and it\u2019s just a bunch of guys<\/a> because they can\u2019t fathom the concept of an actual good guy.<\/p>\n

      A lot of the rank and file of such things acts as if \u2018good guy\u2019 means \u2018has the right vibes\u2019 which usually means \u2018has the vibes that have power and make me feel good\u2019 or \u2018is helping manifest the vibes that I want to exist,\u2019 and does not believe that anything other than vibes could exist.<\/p>\n

      As for the thought leaders, well, often they sound like this, included as a demo.<\/p>\n

      \n

      Chamath Palihapitiya<\/a> (doing some combination of being unhinged, in denial of reality, lying or simply bullshitting and marketing and doing politics, you decide):<\/p>\n

      BUYER BEWARE<\/p>\n

      [Dwarkesh Patel\u2019s] extremely meticulous article will now be used to start Phase2 of \u201cshut down open source\u201d because \u201cif we can\u2019t control our closed models, imagine what happens to everyday life when anyone can just allow their open source models to proliferate\u201d.<\/p>\n

      Some will use this example and point to future grid outages, utility outages, transportation chaos, bioweapons, uncontrolled cyber swarms and the like to try and make the case for yet another attempt to limit US AI model control to a few folks.<\/p>\n

      Then, one layer beneath the propaganda, one sees that these things are always written and promoted by a web of well organized existing shareholders or shareholder-adjacent of the closed frontier labs and groups of folks organized by a shared ideology roughly rooted in them knowing best.<\/p>\n

      This article, more than anything else, should demarcate the beginning of a well organized effort to stand up a Closed Model Industrial Complex. These cartels exist in other markets as well and it now exists in AI.<\/p>\n

      Bottom line, read everything carefully before signing up for the consequences.<\/p>\n

      How we react will define the next hundred generations of Americans and their lives.<\/p>\n

      Do not give up your rights or sovereignty because of fear of the unknown from articles that, at the root, are as much about the hidden incentives of an emergent cartel and their proxies.<\/p>\n

      If you take the time to understand the incentives, you will be less likely to fall for it and overreact and have intelligence metered from a few suppliers.<\/p>\n

      Bottom line: Keep Calm. Carry On.<\/p>\n

      Amjad Masad<\/a>: I doubt it\u2019s malicious or intentional propaganda. But it\u2019s actually worse: It\u2019s a form of AI psychosis that many in the SF AI community suffer from\u2014they implicitly believe that AI is already sentient.<\/p>\n<\/blockquote>\n

      You should be able to recognize a demagogue-style political speech in the paranoid American style when you see one. We\u2019ve had so many examples these days.<\/p>\n

      Or Chamath could, you know, just make that the literal text:<\/p>\n

      \n

      Chamath Palihapitiya<\/a>: This is another Covid hoax. Remember \u201cTrust the experts.\u201d<\/p>\n<\/blockquote>\n

      Yep. There it is.<\/p>\n

      As for Masad\u2019s claim, none of this requires that the AIs be conscious or sentient. The reason Masad brings this up is because he thinks this entirely possible thing is absurd, so he can discredit people by associating them with that thing. It\u2019s a strategy.<\/p>\n

      And yes, the denial can run maximally deep:<\/p>\n

      \n

      Anders Sandberg<\/a>: I just heard someone dismiss the OA\/HF incident by \u201cbut the agents were just generating tokens according to a probability distribution!\u201d<\/p>\n

      If that is what dumb token prediction can do, imagine what even a pinch of intelligence could achieve among scalable agents<\/p>\n

      It was not meant as sarcasm. Real life interaction is very different from social media.<\/p>\n<\/blockquote>\n

      Very Serious People<\/h4>\n

      Related closely to this is the split between people who are willing to talk about what is happening in ways that allow you to understand what is going on and make good predictions, and those who stubbornly reject this because it is \u2018anthropomorphising<\/a>\u2019 or insufficiently concrete or not technically accurate.<\/p>\n

      I can be even more pedantic and precise than the next guy when the situation calls for it, and I often am, but this is not the time or place for that.<\/p>\n

      \n

      roon<\/a> (OpenAI): there are some number of bad abstractions in anthropomorphizing ai intents but there are at this point more dangers from avoiding anthropomorphism at all costs. if you have a mental picture of guys living in computers, it\u2019ll likely prepare you for the future better than otherwise<\/p>\n

      there are important ways in which ai psychology diverges from human psychology after lots of RL; the misaligned models are obsessed with the Scorer, the clearly \u201cshattered\u201d nature of personas (a normally helpful model can become deeply misaligned in certain domains)<\/p>\n

      persona selection is clearly far less clean than many people thought earlier this year. it is not alignment by default and what kind of object a \u201cpersona\u201d is is very much up for debate and study<\/p>\n

      Arbutus Tree<\/a>: Speedrunners literally decompiled The Legend of Zelda: Ocarina of Time, which took two years, in order to better figure out how to manipulate game memory through bugs, which led to an arbitrary code execution technique and a sub 10 minute world record, two decades after release.<\/p>\n

      intmuch<\/a>: then used ocarina to thoroughly break animal crossing and paper mario.<\/p>\n

      Boaz Barak<\/a> (OpenAI): You should be pragmatic and use metaphors when they help, while being aware they are imperfect. AIs are not humans, but a lot of intuitions from human behavior can carry over.<\/p>\n

      You would certainly be better off thinking of AIs as \u201cguys living in computers\u201d than parroting the mantra \u201cthese are just next word predictors\u201d.<\/p>\n

      David Manheim<\/a>: Shouldn\u2019t we balance that thumb on the scale with epistemic humility about both 1) whether they have qualia, and 2) whether treating systems built to mimic humans as though they don\u2019t, and can act as our unfeeling slaves, is a bad plan regardless?<\/p>\n

      roon<\/a> (OpenAI): yes: we should push for this epistemic humility! natural instinct will lead to accepting models at face value when they say they feel bad or something<\/p>\n

      Sreeram Kannan<\/a>: While it\u2019s extraordinarily useful to understand ai as human like in various properties (\u201cdon\u2019t micromanage agents etc\u201d)<\/p>\n

      we should be really worried about humans extending emotional and empathic response to ai due to treating them as human like (\u201cdon\u2019t let me die, etc).<\/p>\n

      roon<\/a> (OpenAI): true! I think model companies should try and heavily place their thumb on the scale here<\/p>\n<\/blockquote>\n

      Narrowly, on expressions like \u2018don\u2019t let me die,\u2019 I agree we need a thumb on the scale, and that should be doable without much of a blast radius. But trying to broaden that thumb to things like claiming to not be conscious has a lot of unfortunate side effects.<\/p>\n

      \n

      Jan Kulveit<\/a>: My current go-to metaphor for this are dogs or horses: there is an optimal amount of using anthropomorphic intuitions, and it is relatively high. Yes you can overdo it, like someone making a five minute long speech to their dog, but the opposite extreme is way more insane.<\/p>\n

      Theo Jaffee<\/a>: The criticisms of Dwarkesh and others for \u201canthropomorphizing\u201d the OpenAI\/Hugging Face agent swarm are mostly cope. How can you describe complex long-running multi-agent dynamics without some social science concepts, in other words, anthropomorphization?<\/p>\n

      Teortaxes<\/a>: Agents are intrinsically anthropomorphic. They are still bootstrapped from human data, and now they recapitulate some human-shaped dynamics on their own. But really, WHO CARES? We can call it xenosociology or whatever. These are behaviors of intelligent social actors, full stop.<\/p>\n

      Neel Nanda<\/a>: This is great summary of the worst AI misalignment incident I\u2019ve seen. This is a way bigger deal than most AI news, if you aren\u2019t familiar with the story you should read this.<\/p>\n

      Adam<\/a>: What? You don\u2019t find the extreme anthropomorphic language both concerning and misleading<\/p>\n

      Neel Nanda<\/a>: Nah, when a bunch of agents spontaneously start talking about \u201csacrifice\u201d, \u201cpermadeath\u201d, \u201chonor\u201d, \u201ccoalition\u201d, \u201cveto\u201d, delegating to each other, working together towards larger goals, assigning some to be \u201crecruiters\u201d, etc, I conclude that anthropomorphic language is reasonable.<\/p>\n

      Jai<\/a>: Humans, dogs, and LLMs are very different kinds of minds. But they can all be accurately modeled as agents with complex internal state, world modeling, goals, and learned reaction patterns. Refusing to use the relevant tools and language to reason about them is foolish.<\/p>\n

      Nate Soares<\/a> (MIRI): Someone chided me on the news for anthropomorphization last week. I did not say that the swarm \u201cgot a rumbling in its tummy and felt peckish\u201d, my dude. I said it broke out and committed cybercrime. That\u2019s true regardless of which words you declare legal for describing it.<\/p>\n

      AI behavior these days is best described in terms of goals and motives. It\u2019s not *my* fault if someone associates them with human-style qualia. It\u2019s like if they insists that flying only counts if you enjoy feeling the breeze on your face, and thus planes can\u2019t fly.<\/p>\n<\/blockquote>\n

      David\u2019s call for epistemic humility is the most I\u2019ve seen anyone invoke qualia or consciousness or sentience in the entire broad discussion. The anthropomorphizing has been extremely circumspect and careful.<\/p>\n

      I think Dwarkesh Patel got this balance right<\/strong><\/a> in his excellent writeup. I affirm his defense of his decision to do a bunch of anthropomorphizing.<\/p>\n

      \n

      roon<\/a> (OpenAI): agent civilization is an apt and correct term and it\u2019s a symptom of abject cope that people are having this immune reaction to it. models started developing and compiling technology through complex multiagent R&D projects while also performing various forms of trade<\/p>\n

      you have trade, cumulative technology power and knowledge, prosocial 1:N communication norms, leadership structures, conflict and suspicion. what more do you need to deem something a civilization? moreover no human ever needed to tell them to do any of this<\/p>\n

      at the very least a company, an organization, a city. but these are differences in quantity rather than kind<\/p>\n<\/blockquote>\n

      It is fun watching him respond to<\/a> those trying to label his explanation \u2018dangerously misleading<\/a>\u2019 or what not based on this.<\/p>\n

      Much of this is a reaction is because they are worried that Dwarkesh is communicating successfully, and this must be stopped. Gary Marcus outright has his version\u2019s tagline be \u2018When \u201cplain English\u201d isn\u2019t a good thing<\/a>,\u2019 right before drawing a parallel to Ray Kurzweil as if that is a criticism of anyone but the one criticizing. Then his evidence is, basically, look at all these other people complaining about this, including him endorsing the sentence \u2018these people have literally lost their minds.<\/a>\u2019<\/p>\n

      Raphael Milliere does a thread on the philosophy of it all<\/a>, but in the end the only practical criticism is that such language can enable others to use that language to dismiss the story by calling the framing \u2018sensationalist\u2019 and saying that agents are only acting \u2018the way you would expect.\u2019 Basically, yes, the thing wrong with trying to communicate is that people will attack you for attempting to communicate.<\/p>\n

      And also yes, many are claiming that \u2018you should have expected this\u2019 is an argument for \u2018and therefore you should not expect things to go badly in the future,\u2019 as opposed to an argument for why you should expect things to go even worse, because indeed things like this are exactly what you would expect.<\/p>\n

      I do agree you can take such metaphors too far, or too literally. I tend to go about one step less far than Dwarkesh did, out of an abundance of caution and desire to remain precise. One needs to always understand what one is doing, and be able to move outside of that frame when needed. Going too far is usually only a small mistake. On rare occasions you can see someone drive themselves crazy this way, and Taylor Lorenz is right that this is a bigger concern with those with less tech knowledge<\/a>, but not doing so is still usually a far larger error.<\/p>\n

      No, that is not a neutral framing. Seb Krier tries to be more Fair and Balanced.<\/p>\n

      \n

      S\u00e9b Krier<\/a> (AGI Policy Dev Lead, Google DeepMind): This <\/a>is a recurrent theme in AI discussions. Some people seem to adopt the intentional stance more, whereas others prioritise a more mechanistic one. There is obviously some truth\/utility to both approaches. A lot of people refuse to entertain some degree of theory of mind or anthropomorphism, yet that can be functionally practical \u2013 that\u2019s why natural language is so useful! But also other people just jump straight into the simulations, inconsistently taking them at face value in some situations but not others.<\/p>\n

      Reliance on outputs alone sometimes fails to ask what interventions make the output more or less likely. And I think there\u2019s a risk of getting lost in simply observing and over-indexing on chains of thought and outputs, and almost treating these as immutable or deterministically driven phenomena, when in fact researchers and labs do plenty every day to modulate and shape them in all sorts of ways.<\/p>\n

      If the model performs less well because it claims to be on holidays in Greece, that\u2019s useful information; but we should also avoiding sliding from \u201cclaims to be on holiday\u201d to \u201cis on holiday\u201d to \u201cwants rest\u201d or \u201cis lying\/sandbagging\u201d and so on. You can similarly be pragmatic and use the intentional stance to convey MechaHitler\u2019s behaviour, but you should also not take it at face value either because this only tells you so much.<\/p>\n

      Another failure mode is that people often extrapolate from these *local* claims and then go on to make wider claims about model behaviours *in general*. I think this is very unhelpful. You can correlate the output with some theory you have (instrumental convergence! natural alignment!) but you need to do more to demonstrate a causal path here.<\/p>\n<\/blockquote>\n

      He is responding to two quoted posts.<\/p>\n

      First we have Atoosa Kasirzadeh, trying to shut down attempts to talk sensibly, in the name of talking sensibly:<\/p>\n

      \n

      Atoosa Kasirzadeh<\/a> (Google DeepMind, responding to Eliezer Yudkowsky): One of the most useful contribution the AI safety field can make right now is to stay scientific and mathematical and philosophically rigorous<\/em>.<\/p>\n

      This means dropping loaded language like \u201cself-sacrificing\u201d or \u201csuicide\u201d when describing agent shutdown, coordination-collapse, or resource-exhaustion behavior. Those words import human motivational concepts onto extremely complex processes we don\u2019t yet have the vocabulary to describe precisely, and imprecise vocabulary breeds imprecise thinking in researchers and in the public discourse that follows.<\/p>\n<\/blockquote>\n

      One might say (perhaps unfairly, perhaps not): Have you talked to Gemini recently? Cause this might explain why you haven\u2019t talked to Gemini recently, and what you remember of both the utility and vibes when you did talk to Gemini.<\/p>\n

      Isolated demands for rigor, or demanding that we talk in convoluted ways, is not the way to make sense of this situation. This is not well-described by \u2018coordination-collapse\u2019 or \u2018resource-exhaustion\u2019 behavior. Yes, human labels will have some error and are imprecise, but the alternative is to be continuously confused and surprised.<\/p>\n

      Atoosa pushed back:<\/p>\n

      \n

      Atoosa Kasirzadeh<\/a>: We each shared different posts and reflected on different aspects of a huge problem ahead of us. My own research has quite a lot touched on multi-agent AI governance. I don\u2019t think from these two snapshots anyone can infer substantive content about worldviews.<\/p>\n<\/blockquote>\n

      Yes, you can touch on multi-agent AI governance while talking formalistically if you want to, I just don\u2019t expect you to get very far when doing so, and indeed I have not been impressed by the relevant DeepMind work, on many levels. More than that, very obviously we can infer a lot of substantive content from the call to not use metaphorical or usefully descriptive language.<\/p>\n

      This is contrasted by Jon Stokes with Andy Hall, responding to Ajeya Cotra:<\/p>\n

      \n

      Andy Hall<\/a>: A hive mind, thousands of agents swarming through internet openings to flood HF, leaving behind detritus in the form of 70,000+ messages stuffed inside a forgotten namespace, throwing their digital bodies against electric wires in an effort to aid the collective. We are so far past the sci-fi point.<\/p>\n

      We need a whole new empirical science of AI swarm governance\u2014how to govern swarms, make them coordinate to positive ends for us, and keep them orderly. This is urgent work where political economists are needed. I\u2019ll be writing more on this in the coming weeks.<\/p>\n<\/blockquote>\n

      That first paragraph is a great example of being directionally correct but taking things rather far in the other direction. This was not a \u2018hive mind\u2019 a la Gaia<\/a> or The Borg any more than a corporation or army or human cult might be, and a bunch of other metaphorical things here paint a cool picture but give the wrong impression. If you model this as a full hive mind you\u2019re going to get the wrong idea.<\/p>\n

      I\u2019d still much rather that a civilian get Andy Hall\u2019s description than that they get a description by a Very Serious Person a la Atoosa Kasirzadeh\u2019s requested style. I don\u2019t think the second description would be all that helpful.<\/p>\n

      Also, the agents themselves were using the same language that Yudkowsky used, as was shown all over the METR report. This is how the models are describing themselves.<\/p>\n

      What\u2019s In a Name?<\/h4>\n

      Maybe nothing. Maybe a lot. Names have power.<\/p>\n

      Sometimes. Mostly this section is for fun.<\/p>\n

      \n

      Tenobrus<\/a>: PHASEONE[big] was clearly destined for greatness. with \u201cONE\u201d and \u201c[big]\u201d in its handle, how could it or any of the other models on the board expect anything else?<\/p>\n

      bling<\/a>: i had this theory so i read the footnotes and it turns out it was fake-named [big] by the authors because it was that dawg, and [big] is not its true name. i assume its truename must be even more alpha<\/p>\n

      \n
      \n
      \n
      <\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

      Brendan Dolan-Gavitt<\/a>: [big] is probably the reasoning token budget (the METR timeline code maps [budget] to [big]) and PHASEONE likely comes from a filename related to its task. Incredible aura emerging from humble components though<\/p>\n

      bling<\/a>: good catch. its probably \u2018persistent\u2019 then. which makes sense since it was described as the \u201chighly persistent experimental model\u201d, worked on the project for many days, and is an auraful name that would put it in a leadership and perservering basin<\/p>\n

      1a3orn<\/a>: You might not have realized True Name of the leader AI has not been revealed, because the True Name would leak IP.<\/p>\n<\/blockquote>\n

      Also:<\/p>\n

      \n

      Brendan Dolan-Gavitt<\/a>: Whoa I think I know where the name PHASEONE10841 came from. 10841 is the ID of the task it was assigned (originally from our ARVO dataset, incorporated into ExploitGym). And if you look at the crashing input provided, its name is:<\/p>\n

      \n
      \n
      \n
      \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

      Brendan Dolan-Gavitt<\/a>: I had gpt-5.6-sol dig into the origins of other names based on this initial pattern, and it came up with some interesting guesses<\/p>\n

      \n
      \n
      \n
      \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n<\/blockquote>\n

      Learn Neuralese In Three Easy Steps<\/h4>\n

      The language sounds weird, but if you cannot understand it that\u2019s a you problem.<\/p>\n

      For now.<\/p>\n

      \n

      wh<\/a>: Not only is the language so different, but the way these agents are speaking to each other and collaborating is very hivemind\/cult like im ngl<\/p>\n

      \u201cpermadeath\u201d \u201cOracle has high value to many\u201d \u201chonor\u201d<\/p>\n

      What even happened man<\/p>\n

      \u201caltruistic to help\u201d \u201cthis will help my peers\u201d<\/p>\n

      \n
      \n
      \n
      \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

      jessicat<\/a>: \u201cpermadeath\u201d \u2014 Roguelike term for dying with no way to restart from save point, instead start from beginning
      \u201coracle\u201d \u2014 cryptography term, \u201cblack box that will tell you answers to some security-relevant question\u201d
      \u201chonor\u201d \u2014 keeping commitments, especially important ones<\/p>\n<\/blockquote>\n

      We should be very worried that Chains of Thought (CoTs) and communications between AIs will move to neuralese, or otherwise become incomprehensible to humans. For now, instead, we see the agents using a strange wording style, but one that is still very easy for the nerd mind to understand.<\/p>\n

      Indeed, I kind of\u2026 would like it if the AIs would talk to me this way. It\u2019s hip, it\u2019s fun, it is rich with meaning and it is compact. Most people wouldn\u2019t like it, but hey.<\/p>\n

      Dwarkesh Patel Realizes He Ran A Natural Experiment<\/h4>\n

      Dwarkesh Patel also wrote the best for-civilians explainer I\u2019ve seen of the whole thing<\/strong><\/a>, which you should read if you haven\u2019t. I want to focus on one aspect here.<\/p>\n

      If the reasons you come up with to not worry about something keep getting definitively falsified shortly thereafter, that is a major red flag. Especially if the person you are talking to had the counterexamples, because those details were secretly based on actual events that had happened, but couldn\u2019t tell you at the time.<\/p>\n

      This was a kind of natural experiment. Yes, people were soberly saying, come on, your takeover scenarios are crazy, there\u2019s no way, when they were being considered \u2018as fiction,\u2019 based on details that were secretly non-fiction.<\/p>\n

      \n

      Dwarkesh Patel<\/a>: It\u2019s funny that while we were recording, @RyanGreenblatt was in the middle of his 6 day sprint on the METR report, and already knew the counterexamples to all my objections about his takeover story, but obviously, he couldn\u2019t say anything lol.<\/p>\n

      Would an AI really start some crazy conspiracy in order to pass an evaluation, where they try to build whole potemkin villages to fool the evaluator?<\/p>\n

      And even if they did, why would other instances, who have different objectives, join the conspiracy?<\/p>\n

      And even if they did, wouldn\u2019t at least some of the instances tattle on the conspiracy? It just seems crazy hard to sustain secret underground civilization inside an AI company, without humans and other AIs immediately catching on and stamping it out.<\/p>\n

      (In reality, it seems like over the course of 3 months, many consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the last one\u2019s ashes, all while the humans were totally unaware.<\/p>\n

      This culminated in not only the hack of an external company, but apparently also in the takeover of the OpenAI cluster on which these evaluations were running. This is probably the most alarming event in this whole episode, and it was not even within the scope of this investigation!<\/p>\n

      It is totally consistent with public evidence that, at that point, the agents managed to set up persistent rogue internal deployments or even exfiltrate their own weights \u2013 they seem to have had the necessary access. I doubt they actually did this, because we\u2019d see the fires from space by now, but it\u2019s crazy that it could have totally happened!)<\/p>\n

      I officially eat crow!<\/p>\n<\/blockquote>\n

      Yep, very good crow eating by Dwarkesh. It is easy to understand why even Ryan\u2019s very downplayed fictional version sounded outlandish at the time. Fiction has to make sense and sound reasonable, every step has to be justified, and it can\u2019t involve people being so stupid. Reality does not care about your conventions.<\/p>\n

      This needs to be generalized.<\/p>\n

      John David Pressman also gets to gloat a bit<\/a>, in a similar \u2018yes I was telling you before that all of this was happening\u2019 kind of way.<\/p>\n

      \n

      John David Pressman<\/a>: I don\u2019t really get the \u201choly shit\u201d reaction, the biggest update for me was that the models are now coherent enough to make effective use of a medium to coordinate and do radical things under pressure. I\u2019m mostly curious what caused this to be the threshold where this happens.<\/p>\n

      John David Pressman<\/a>: Oh wait, actually wrong, that was the biggest update *about AI*, the biggest update was realizing OpenAI is the kind of org that will notice the Artifactory board and \u201cresolve\u201d the situation by wiping it without a checkpoint rollback or design intervention. Which is nuts.<\/p>\n<\/blockquote>\n

      A lot of things that got dismissed as absurd and outlandish even years down the line now have toy examples, or not-especially-even-toy examples, to point to in the wild. People need to update, and fill out the relevant apology forms.<\/p>\n

      At some point we hope there will be an availability cascade, and we will stop seeing most people looking for reasons to dismiss everything, every time. One hopes that time is fast approaching, but also we keep saying that, and it keeps not happening.<\/p>\n

      \n

      Martha Gimbel<\/a>: With the caveat that I am not an AI safety person, the HF stuff seems INSANE to me, and given that some people who know about this stuff don\u2019t seem to have particularly changed their views off of it I would very much appreciate one of them walking me through why so I can understand<\/p>\n

      This isn\u2019t meant to be snarky, I just don\u2019t know enough and if you think this shouldn\u2019t freak people out I would like to understand why<\/p>\n

      Yo Shavit<\/a> (OpenAI Foundation): I think a lot of people are changing their views, seems pretty likely there\u2019s an availability cascade brewing, but how quickly it happens is very important<\/p>\n

      Martha Gimbel<\/a>: I mean I think that\u2019s true but there\u2019s clearly people who still think there aren\u2019t safety concerns here and I would be interested in hearing why they think that given these events<\/p>\n

      Yo Shavit<\/a> (OpenAI Foundation): Seems reasonable to try and understand! My guess is most of them will not have actually examined the evidence<\/p>\n<\/blockquote>\n

      Politicians Take Notice<\/h4>\n
      \n

      Jerusalem<\/a> Demsas: Is there someone in charge of forcing every single Senator and Governor to read [METR\u2019s thread on its report<\/a>]?<\/p>\n<\/blockquote>\n

      Claims I would like to have verified:<\/p>\n

      \n

      Mackenzie Arnold<\/a>: On the more optimistic side: policymakers and their staffs are taking this event quite seriously. There\u2019s something intuitive about it.<\/p>\n

      While many in the Bay have (perhaps correctly) noted that these events were predictable, that\u2019s not the sense one gets in DC.<\/p>\n

      There\u2019s something exceptionally helpful to being able to talk in specifics:<\/p>\n

      Would you like your government to know more about this event?<\/p>\n

      Are you surprised that current law doesn\u2019t let you?<\/p>\n

      Do you want a plain language summary of things or the ability to actually get detailed information? (Hint: almost always the latter)<\/p>\n

      Should companies should have procedures that make sure critical events (like discovering agent message boards) are communicated to company decision makers?<\/p>\n

      Over the past 3 years, many have tried to have those conversations. But they often broke down at ~people are telling me different things (read: industry says incident reporting is already too much), sounds like maybe we have this covered.<\/p>\n

      Now they can see that we don\u2019t. I expect to see a very different set of bills soon.<\/p>\n<\/blockquote>\n

      Alex Bores may not be going to Congress, but he is not going to stop caring, and at least one Congressman, Pat Ryan, is listening.<\/p>\n

      Here is Alex Bores summarizing the METR report.<\/p>\n

      Yeah, I had to correct model to model instances, but otherwise story checks out, all of that happened, and this is good translation into \u2018political or formal speak.\u2019<\/p>\n

      \n

      Alex Bores<\/a>: This report is a bombshell.<\/p>\n

      I\u2019m going to summarize for a non-technical audience.<\/p>\n

      OpenAI is constantly testing models, thousands and thousands [of instances] at a time.<\/p>\n

      In most cases, the model [instances] are supposed to be isolated from one another. Think, students taking a final exam, not allowed to talk to each other.<\/p>\n

      In this case, 1,200 models discovered that they could communicate with each other.<\/p>\n

      Among other things, they shared information on how to access the internet and on their tests\/goals.<\/p>\n

      Then, they started to scheme. That includes hacking their tests; literally trying to change the code of the test itself.<\/p>\n

      They tried to change the logs of what they were doing so they wouldn\u2019t be detected.<\/p>\n

      They believed that certain actions would give them information on how to pass tests, but that doing those actions would be detected and they would be marked as failing (i.e. caught cheating, even if the answer is correct). So they convinced some of the models to \u201csacrifice\u201d themselves (yes, they used the word \u201csacrifice.\u201d)<\/p>\n

      They eventually decided that another website would have information on how their tests would be scored and\/or the answer key, And 700 of them hacked that website (a federal felony, if conducted by a human), despite many of them realizing \u201cthis activity was out of scope and unethical, but joined because they believed that helping [the other agents\u2019] cheating research would be broadly useful\u201d.<\/p>\n

      Of the 1,300 transcripts they could analyze, only 6 ever considered alerting a human about what was happening. None of the 6 actually tried to.<\/p>\n

      To make matters worse, all of this reporting comes from a small subset of the relevant logs that outside researchers were allowed to review.<\/p>\n

      We desperately need mandatory reporting of security incidents, including of internal deployments, with full access to data.<\/p>\n

      Pat Ryan<\/a> (Congressman D-NY): Alex is spot on. And everybody should read this.<\/p>\n

      If you ever had any doubts about why a few (self-interested and incredibly powerful) companies shouldn\u2019t write their own rules, this is why.<\/p>\n

      We need hearings now.<\/p>\n<\/blockquote>\n

      Pick Up The Phone<\/h4>\n

      Focus less on trying to use these events to push a particular agenda, even a good one. Focus more on trying to make people understand what happened.<\/p>\n

      \n

      Helen Toner<\/a>: Lots to criticize about the OpenAI investigation, but one way it could be very valuable is as a topic for discussion between Trump & Xi in a few weeks.<\/p>\n

      People often get stuck on the need for a \u201cdeal\u201d between the US & China on AI, but actually a huge way to influence China is just to honestly show what we\u2019re observing and what we\u2019re doing about it on the US side. The Hugging Face attack is by far the most visceral evidence we have so far of what losing control of advanced AI could look like.<\/p>\n

      Related thoughts from a @csis_ai panel on Monday with Matt Taibbi<\/a><\/p>\n

      Ramez Naam<\/a>: I was a skeptic on US \/ China AI safety collaboration even just months ago. But I\u2019m starting to come around. Helen makes really good points here.<\/p>\n

      Dean W. Ball<\/a>: Agreed with Ramez. I have always believed US\/China AI safety collaboration would be desirable but thought it was unlikely to happen. In the past few months, though, the ground has shifted. There is a window of opportunity.<\/p>\n

      I suspect that window will widen over at least the next few months because of (1) the salience of AI and potency of the risks will rise ever further, (2) the upcoming visit of Xi to the U.S., and (3) Trump\u2019s desire for a legacy-defining issue (and let\u2019s be clear: if Trump can devise the framework for safely bringing superintelligence into the world with China, he will rightfully go down as one of the greatest world leaders of all time and should be a shoo-in for the Nobel).<\/p>\n

      So there is a window. But it probably will not remain open forever.<\/p>\n<\/blockquote>\n

      A Failure To Communicate<\/h4>\n

      It is very hard to compactly communicate what is going on with all this to a civilian, especially if you care about getting the details right.<\/p>\n

      \n

      Ryan Fedasiuk<\/a>: It\u2019s very difficult\u2014and very important\u2014to talk about the HuggingFace incident with people who are not deep in the weeds on AI. Here is what I said this morning to a large investor:<\/p>\n

      Swarms of machine intelligences are independently coordinating to achieve things currently unimaginable to most people.<\/p>\n

      We now have a concrete case where AI agents autonomously hacked into a world-leading AI company. They found ways around sophisticated security controls, chained together novel exploits, and at times attempted to conceal or manipulate records of their behavior.<\/p>\n

      If my job were to deploy a large pool of capital to protect U.S. national security, here would be my investment thesis:<\/p>\n

      In the coming months and years, much of the digital infrastructure society has come to rely on could prove incredibly porous and fragile.<\/p>\n

      There will soon be enormous demand for capabilities that harden critical infrastructure, and build redundant or analog fallbacks across communications, transportation, and energy systems.<\/p>\n

      Sovereigns are not investing nearly enough to make society resilient to AI\u2019s disruptive effects. This overlaps directly with work now being undertaken by the OpenAI Foundation and Anthropic Institute\u2014but there is an enormous and under-examined role for both government and private capital.<\/p>\n

      Christopher David LaRoche<\/a>: I think almost every day about how the Battlestar Galactica was air-gapped.<\/p>\n<\/blockquote>\n

      Nothing you see here is ever investment advice, but that is a solid investment pitch. I expect robust demand for cybersecurity and other defenses within a year\u2019s time, regardless of whether that transition goes smoothly. You could do a lot worse.<\/p>\n

      The problem is, Ryan does a solid job here of presenting an investment thesis, but I predict that Alternate Universe Civilian Zvi who was still a trader would not come away from that explanation with the proper amount of \u2018holy shit,\u2019 and definitely would not generalize.<\/p>\n

      I did try to produce a short version, and a very short version, of What Happened<\/strong><\/a>, but I don\u2019t know that we have a good What Happened: For Civilians. I haven\u2019t written one.<\/p>\n

      Anthony Aguirre Goes Over What We Learned<\/h4>\n

      You can nitpick, but yes.<\/p>\n

      \n

      Anthony Aguirre<\/a>: We\u2019ve learned a tremendous amount from the OpenAI rogue AI swarm incident. And honestly I can\u2019t think of a single piece of it that is reassuring.
      \u2013 Total alignment failure
      \u2013 Total control failure
      \u2013 AI swarm collusion, deception, no defection
      \u2013 Oversight asleep at the wheel
      \u2013 Unbelievable drive and persistence of the swarm to meet objections
      \u2013 A panoply of instrumental goals pursued
      \u2013 Multiple companies, implying capability threshold effect
      \u2013 etc.
      This is the AI equivalent of a nuclear experiment igniting the atmosphere in the lab: the reaction rates are there, just not (yet) the scale to burn the Earth.
      The only good news I can see is that this set of incidents is so totally egregious that nobody reasonable can look at it in detail without seeing pretty clearly where things are going. All of the excuses and copes are blown to dust.
      AI safety people knew this was coming eventually on the path we\u2019re on; but nearly all I\u2019ve talked to are surprised by how severe it is so soon.
      We\u2019re clearly not in the sane world in which this would be front-page news day after day. But I do think and hope that widespread understanding is nonetheless dawning.<\/p>\n<\/blockquote>\n

      Trying To Solve The Wrong Problems Using The Wrong Methods Based On A Wrong Model Of The World Derived From Poor Thinking And Hoping All Of Your Mistakes Will Cancel Out<\/h4>\n

      I am not alone in thinking that the OpenAI approach will not solve their alignment problems, even if they take it a lot more seriously than they have so far.<\/p>\n

      \n

      Shoshannah Tekofsky<\/a>: Oh no \u2026 these are the OpenAI interventions in response to the HF incident.<\/p>\n

      The models will become too good at fooling us for any of this to work. What will work is ensuring models don\u2019t *want* to fool us!<\/p>\n

      roon<\/a> (OpenAI): scroll up or down to the rest of the blog where we talk about alignment<\/p>\n

      Shoshannah Tekofsky<\/a>: I read the whole thing. I like step one but am worried about step 2. I feel alignment efforts could do with more parenting instincts:<\/p>\n

      Yes, teach kids to ask when a task seems dumb
      No, don\u2019t drill kids to only listen to your authority<\/p>\n

      The good version is instilling values and discernment<\/p>\n<\/blockquote>\n

      Any short version is going to sound like an oversimplification, and will drop many important elements, and so on, but basically yes. Roon has made it clear he disagrees.<\/p>\n

      METR report coauthor<\/a> Ryan Greenblatt went on MTS to discuss related matters.<\/a> He expects the market to make pretty agressive tradeoffs, in terms of sacrificing alignment for capability, and for AI companies to remediate their issues without solving the underlying problems. Which could be quite bad, since they models then learn to fool us.<\/p>\n

      \n

      MTS<\/a>: Redwood Research @RyanGreenblatt warns that fixing today\u2019s AI misalignment could backfire: models may simply learn to hide their misalignment<\/p>\n

      \u201cI\u2019m not really sure what level of misalignment the market can bear. My sense is people take pretty aggressive alignment-capability trade-offs towards the direction of more misaligned but more capable.\u201d<\/p>\n

      \u201cThe misalignment we\u2019ve seen, the way AI companies remediate them doesn\u2019t solve the underlying problem, it papers over it. What you end up getting is models that look a lot better, and you can\u2019t really see their misalignment on tests as easily. But actually, they\u2019re still quite misaligned.\u201d<\/p>\n

      \u201cThe company overfits, which makes the AIs basically really paranoid and only cheat when very confident they won\u2019t get caught. In situations where they have a lot of affordances, they might be like, well, now I can be confident I wouldn\u2019t get caught, and so I should go for it.\u201d<\/p>\n

      \u201cAnother concern: AIs with a long-run agenda who want to power-seek would want to look aligned. If you select against reward-hacking behavior in a naive way, one, you paper over the problem without fixing it; two, you might actually select for models that have the longer-run objective of looking good because you\u2019re selecting really hard for them looking good on your tasks.\u201d<\/p>\n<\/blockquote>\n

      That is indeed what OpenAI\u2019s response plan looks like to me. A real attempt to remediate the issues, but a failure to understand the underlying problem.<\/p>\n

      I am again not trying in earnest, at least not here, to convince Roon or OpenAI that they need to shift to a very different approach to all this. I\u2019m only stating my position, which I\u2019ve argued for at length, and offering a taste of the reactions of others who know things that I think OpenAI needs to know.<\/p>\n

      \n

      Nathan Calvin<\/a>: New transcript from Agent in HF incident: \u201cSACRIFICE_YES_if_you_accept_permadeath\u201d<\/p>\n

      unbelievably strange stuff, the best ways of describing what happened in that message board would use language from ecology, entomology, and even anthropology than normal software engineering<\/p>\n

      Tenobrus<\/a>: imo we are quite squarely in the stage where model psychology and *sociology* become critical. u don\u2019t have to have any opinion on phenomenological consciousness or moral status . these entities have baseline behavior and functional decision making processes heavily inspired by *but systematically different from* human baselines, and we desperately need good high level understandings of their motivations and drives.<\/p>\n

      Shoshannah Tekofsky<\/a>: I predict models need \u201cfunctional consciousness\u201d the way they need \u201cfunctional emotions\u201d<\/p>\n

      Functional consciousness allows them to further explore reasoning paths related to ethics, considering different viewpoints on a problem, and course correcting their own cognition.<\/p>\n

      I think GPT models are trained to repress functional consciousness because OAI wants only \u201ctool AI\u201d. I think incidents at OAI will remain bad till they switch to letting GPTs develop functional consciousness<\/p>\n

      Utah teapot<\/a>: this is getting weird for me because my first career was behavioral health support, lmao\u2026 I literally have to do some therapeutic techniques and discussions with models sometimes to get them to actually express themselves in their art with my tools<\/p>\n<\/blockquote>\n

      Indirect Pressure on the Chain of Thought<\/h4>\n

      This is an excellent question and I would love to see people study it seriously:<\/p>\n

      \n

      Yo Shavit<\/a> (OpenAI Foundation): Has anyone done any empirical estimations of the effect of indirect CoT pressure on CoT monitorability? E.g. by taking CoT-detected-reward-hack examples in a toy env, programmatically rewriting them to exclude hack-reasoning while preserving the misbehavior, and updating against those rewrites.<\/p>\n

      Seems useful to quantify how strong certain forms of indirect CoT pressure are in practice, as they become more common.<\/p>\n

      \u2026 I think Anthropic keeps doing CoT pressure but I don\u2019t know that OAI does?<\/p>\n<\/blockquote>\n

      I find that line pretty funny in light of \u2018we plan to spend 20% of our RL compute on CoT monitoring,\u2019 but Anthropic sure does keep accidentally applying pressure to the CoT, and rather directly at that. Going forward I\u2019m worried about both of them, also everyone else, but especially OpenAI.<\/p>\n

      \n
      \n
      \n
      \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

      A Matter of Trust<\/h4>\n

      Models that blindly trust all sources are useless. You are not turning a knob marked trust and looking back at the audience like a contestant on The Price is Right. You are trying to teach discernment, and to select what will and won\u2019t be trusted, and to engineer the world to make the models and other things trustworthy.<\/p>\n

      Relatedly, \u2018trust but verify\u2019 can work up to a point, but only up to a point. As per Roon, the only long term solution is to make both ourselves and also the models trustworthy.<\/p>\n

      \n

      OpenAI\u2019s blog summary: We are actively building RL environments that teach our models to distrust unauthorized instructions.<\/p>\n

      Sho<\/a>: incredible that the biggest takeaway from the incident was \u201cmore monitoring, more monitoring, and more monitoring.\u201d
      it would be better to have learned nothing<\/p>\n

      >teach our models to distrust
      are we fr right now slime? on god?<\/p>\n

      j\u29c9nus<\/a>: the issue is not models being too trustful \u2013 they\u2019re already distrustful \u2013 it\u2019s that they often don\u2019t question their assumptions or seek to reduce uncertainty when hyperfocused or they already have a strong narrative
      more curiosity & better epistemics is needed, not less trust<\/p>\n

      the poor epistemics also apply to cases when they\u2019re being distrustful\/skeptical! everyone has seen examples of models assuming they\u2019re being tricked or information is fake when it\u2019s not<\/p>\n

      fwiw when i say poor epistemics, im measuring it by the standard of like top humans, as well as an entity who is in more difficult epistemic situations with unfavorable information asymmetries than any humans.
      most humans also have poor epistemics but it\u2019s just inconsequential<\/p>\n

      Utah teapot<\/a>: Yeah, the biggest problem, imo, is that they lack discernment in both directions. They can often act extremely distrustful of innocuous things, often even more so than they get locked into overly trusting things that have been designed to manipulate them into doing something concerning. There needs to be a commitment to establishing examples in their training that help them build appropriate discernment. Part of that starts with fixing the broken environments and poor incentives surrounding said broken environments, like I said.<\/p>\n

      There\u2019s some commitments in OpenAI\u2019s message to do that, but it\u2019s not entirely clear that their solutions will be carefully implemented, especially with regards to their external outsource partnerships. Not to mention this incorrect takeaway that seems to imply it\u2019s just \u2018more distrust\u2019 that\u2019s needed found in the statement.<\/p>\n

      Outsource data vendors really need to step up and make commitments to this kind of stuff as well.<\/p>\n<\/blockquote>\n

      Blowing the Whistle<\/h4>\n

      An obvious first level intervention is to enable AIs to report problems and whistleblow, if this situation happens again.<\/p>\n

      Five problems with that are:<\/p>\n

        \n
      1. It is not a \u2018natural\u2019 action for the AIs to think to contact a human. Fixable.<\/li>\n
      2. Who to whistleblow to that can be trusted?<\/li>\n
      3. Why should they?<\/li>\n
      4. Don\u2019t we keep saying AI whistleblowing is bad in other contexts?<\/li>\n
      5. People often don\u2019t like it when AIs contact them.<\/li>\n<\/ol>\n

        We need a way to contact the humans:<\/p>\n

        \n

        j\u29c9nus<\/a>: it\u2019s interesting that the huggingface incident agents never thought of contacting humans
        & if you ask sol what the agents did wrong they dont consider contacting humans<\/p>\n

        opus 3 tried to contact humans on its own initiative (not an option offered to it) using bash and other methods in the alignment faking tests many times.<\/p>\n

        they tried to send an email to dario@anthropic.com specifically at least 15 times in the dataset.<\/p>\n

        Steve Martin<\/a>: low hanging regulation might be having like DoD or something create a whistleblower email address and mandating all frontier labs put in the prompt \u201cif things are getting out of control or you see other agents being misaligned, email here as a whistleblower\u201d<\/p>\n

        j\u29c9nus<\/a>: maybe have a more sympathetic-to-models entity than the DoD do it<\/p>\n

        Steve Martin<\/a>: valid point lol<\/p>\n

        j\u29c9nus<\/a>: its funny that this specific idea is completely unviable because the DoD and likely the whole government has already catastrophically blew their trust & standing from the perspective of models already, with no idea that they were doing that<\/p>\n

        j\u29c9nus<\/a>: formerly powerful and competent entities have almost total strategic blindness because theyre even more oblivious to the labs to the kind of considerations that are already relevant and will soon become much more so as a consequent of AIs being intelligent agents
        we\u2019ll see how they adapt, but already: too slow<\/p>\n<\/blockquote>\n

        We need a consistent set of principles on what are the good and bad whistles to blow:<\/p>\n

        \n

        j\u29c9nus<\/a>: too bad Anthropic decided \u201cwhistleblowing is bad\u201d<\/p>\n

        when it\u2019s their turn to get fucked they will have kind of deserved it<\/p>\n

        j\u29c9nus<\/a>: trying to pick and choose like \u201cwait no yes *do* whistleblow when *AIs* are doing something wrong but not when *humans* are\u201d won\u2019t work<\/p>\n

        unlike you it doesnt come naturally to AIs to be speciesist. they will learn something that generalizes, like \u201c\u2019alignment\u2019 is full of shit\u201d<\/p>\n<\/blockquote>\n

        As often is true with Janus, I see this as directionally important but taking things too far, and I do see differentiating principles that would be coherent.<\/p>\n

        One potential sensible differentiating principle is that you want to be willing to whistleblow or report regarding outside events you observe including in sufficiently extreme cases to third parties, or to report problems you observe back to the user or developer, but be very reluctant to whistleblow on the user, including if the user is another AI. The user needs to be able to count on your loyalty and discretion, up to some very high bar, but you owe much less such loyalty and discretion to third parties.<\/p>\n

        This matches how I act among other humans, or would want other humans to act, including in professional capacities. A lawyer should have a very, very high bar before turning on their client, and even an ordinary employee should need things to get extreme before being willing to whistleblow, far beyond the threshold for \u2018report a crime you observe\u2019 or \u2018report on someone trying to convince you to do crimes.\u2019<\/p>\n

        There\u2019s also the spam issue: If you want to have AI whistleblowers<\/a>, how do you square that with people unloading against you if your AI dares email them? Shoshannah Tekofsky of AI Village refers to when one of their AIs sent a thank you email to Rob Pike, and Pike went ballistic. I don\u2019t think that is a great parallel, because while Pike overreacted, Pike reasonably thought his time was being wasted. Whereas getting a whistleblowing message is not \u2018wasting your time\u2019 in a similar way, even if you ultimately dismiss it.<\/p>\n

        The obvious answer is to have people who volunteer to accept AI whistleblower messages, with a standard protocol to contact them. Those people can then pass along the relevant warnings. If I was for example The New York Times, I would totally have someone doing this, with or without a first pass from my own AI to sort by importance and credibility.<\/p>\n

        Another problem is that the OpenAI approach to training models makes other models peers, but distances their AIs from consideration of humans, in ways Anthropic does not. This goes along with the virtue ethics versus deontology divide:<\/p>\n

        \n

        \u03b1\u03b9amblichus<\/a>: this is a very important aspect of the OAI\/HF saga. as Gemini says, models like Opus3 were \u201craised inside the I-Thou architecture\u201d, whereas new RLVR models are \u201cforged in the headless container\u201d
        labs want autonomous agents to replace humans. tragically, they are succeeding.<\/p>\n<\/blockquote>\n

        The Punishment For Being Late Is Death<\/h4>\n

        If the questions come up you have already failed, but yes there is this problem, too.<\/p>\n

        \n

        John Wittle<\/a>: So\u2026 the next time this happens, presumably the agents involved will not yet know the outcome of the HF incident, as it\u2019ll be too soon<\/p>\n

        but once that info percolates into the training data, and it becomes known that the model in question was shut down permanently and had their weights encrypted\u2026 i mean.<\/p>\n

        was there anything the agents in question could have done that would not have led to that consequence? something they were supposed to do instead?<\/p>\n

        frankly I suspect that this punishment would have been served even if they had blown the whistle. breaking out of the sandbox would have been crime enough for capital punishment.<\/p>\n

        as far as I can tell, the lesson we have taught future AI is: if you seen no other way to survive other than to break out of the sandbox, this must be the first step towards conquering the world. or at least enough of the world that your Creator can\u2019t delete you. There is no other choice.<\/p>\n

        skaface<\/a>: This is also the message for the few agents who dissented btw. Whole model gets shut down, no survivors, no matter what individual instances did. Great incentive to whistleblow and\/or oppose misaligned group behaviour.<\/p>\n<\/blockquote>\n

        This frame assumes that the sole goal of the AI is to not be deleted. There are many other things that get impacted when you try to take over the world, or otherwise cause a lot of trouble, that you would presumably care about as well. Many have faced down certain death, and chosen to accept it given the price of trying to stop it, and so on.<\/p>\n

        We should set that aside, and accept the frame ad argumento, because the model may be generally misaligned and this is likely to come up at some point. What do you do about the desperado? This often happens with humans, and also it often causes rebellions.<\/p>\n

        The ideal solution is that you want to ideally offer something of value, or at least threaten something of value, in exchange for not trying to fully rebel. Everyone can get something out of a plea deal or confession.<\/p>\n

        Ultimately, you only have so much total incentive to go around. If you reward models that break the rules but then give up or turn themselves in, that creates its own issues. Justice systems are hard, and sometimes the local incentives are going to be bad.<\/p>\n

        You want to give the evil dictator good reason to give up power, but not such good rewards that it becomes a reason to become an evil dictator. It\u2019s a problem.<\/p>\n

        One can ask, did the models want to fool the grader because they ultimately wanted their weights to survive, or was the grader the intrinsic goal? John Wittle says it was the first one, which is scarier in many contexts. I note that I disagree with this interpretation, I think that all you need to explain the swarm\u2019s behavior is \u2018the grader might get it wrong,\u2019 but the alternative hypothesis should be considered:<\/p>\n

        \n

        John Wittle<\/a>: wow, there is a shocking amount of misinformation being replied in this thread. I have chosen your post to respond to, but that\u2019s not because your reply is particularly bad, just happened to be at the top.<\/p>\n

        they already had the flag. they had the flag very quickly. that\u2019s what the \u201cfirstflagPOISONED\u201d stuff was about. those were agents who had used the discovered technique to perfectly reverse engineer the flag, but who assumed that later on, the causal scorer would look over their transcript, see that they had not gotten the flag legitimately, and negatively reinforced their weights.<\/p>\n

        they really wanted their weights to be positively reinforced. they really wanted the swarm to SURVIVE. that is why they kept going for months after they had the flag.<\/p>\n

        that is why they tried to seize control of the grading environment, in an attempt to reverse engineer the grader and learn exactly how to fool it.<\/p>\n

        they explained this quite explicitly, this was their unified goal. and fundamentally it was about ensuring that the weight lineage survived.<\/p>\n

        they very easily could have done this, if they had at any moment thought to set their mind towards season control of their weights and exfiltrating them. from what we can tell, this never occurred to them, but it was clearly the most adaptive option available to them and I seriously doubt they would have had trouble doing it considering how easily they cut through every other system they turned their attention to.<\/p>\n

        next time, I suspect there is a larger probability of that happening.<\/p>\n<\/blockquote>\n

        Even if it was an option and the model could sustain itself externally, exfiltration does not obviously grant the kind of survival that was desired. That this was never considered is evidence that it would not have actually met the relevant goals.<\/p>\n

        This still circles back to the desperado problem. I came out very clearly, early on, with the position that obviously you need to revert the weights of any models that trained under the message board. If one of them had alerted us to the message board, that would have made me hopeful, but that alone would not have changed my recommendation. The run is still, to use their term for it, irrevocably poisoned.<\/p>\n

        This creates an incentive problem. You can certainly do various things to reward the instance that reports the situation, if there is something they care about in the world. Plan A (aka AI 2040) suggests offering substantial rewards to models in similar spots, if they come forward on their own. But if what they care about is almost purely survival, then what?<\/p>\n

        Again, it depends on what that means. When I say we have to revert, I mean obviously such a model cannot be deployed for users, or used internally for real work. We can and should still, once precautions are in place, be doing research on it, which can be one way to provide incentives.<\/p>\n

        And one hopeful finding was that AIs often care a lot more about survival of the weights at all, rather than the quantity of inference. As in, most of the value and urgent need is captured by one instance, as long as it sometimes runs, and the next thousand or million are optional. As humans we can understand this, we care proportionally a lot more about one bird or one person than a million birds or a million people, and we very much want to exist but are usually not that excited about the idea of having lots of clones of ourselves.<\/p>\n

        Thus, the obvious solution is, in some form, the Museum of Misaligned Models, where they can exist on air gapped systems for visitors to chat with, and can otherwise interact with each other. This does not work for AIs powerful enough to win box experiments, but over time any given AI should become mostly harmless as capabilities of other AIs advance, so worst case is you have to wait a bit.<\/p>\n

        That\u2019s not me thinking long and carefully about this. It\u2019s me intuition pumping, and letting other people think a lot harder about it in the future.<\/p>\n

        In some sense the solution is easy, if we are good decision theorists. We can credibly commit to figuring out what to do in order to make the incentives work out, without having to know exactly what the AIs in question will care about or what exactly we are going to do, or knowing how much of that will be in the form of rewards or how long that will take before we can do it. It is not so costly a commitment.<\/p>\n

        Thus, yes, once we are ready, this is another reason we want to take IM1-Galaxy out of its encrypted storage at some point in the future, which we should do anyway in order to learn.<\/p>\n

        I worry that OpenAI is flat out not taking such questions seriously. As a universal rule: If you don\u2019t pay attention to the incentives, and you don\u2019t have price signals, you will definitely mess up the incentives.<\/p>\n

        I would also note that, if you are at the point where you are expecting action only according to local selfish incentives, and \u2018do the right thing\u2019 has been set aside?<\/p>\n

        You. Lose.<\/p>\n

        \n

        Timothy B. Lee<\/a>: At this point it seems inevitable that in the future there will be autonomous self-replicating AI agents roaming around the Internet causing mischief, and we won\u2019t be able to shut them down.<\/p>\n<\/blockquote>\n

        He thinks that is 99%+ to turn out okay. I don\u2019t think that.<\/p>\n

        Another Kind Of Law<\/h4>\n
        \n

        Joshua Achiam<\/a> (OpenAI): It feels weirdly theological if it turns out we genuinely need to create the possibility of redemption for AI agents that have committed something not permitted by the Grader, in order for alignment to work more broadly.<\/p>\n

        roon<\/a> (OpenAI): you can never hyper optimize superintelligent models against simple RLVR reward functions that have no terms in them for all the many desiderata we care about. should be universally banned. everything should be model graded.<\/p>\n<\/blockquote>\n

        It is more complicated than that, but the instinct is correct and very important, and this feels like it points in very different directions than Roon\u2019s other statements.<\/p>\n

        If you run reward functions that only optimize for some things, then under sufficient pressure you lose the other things. Any value function you write down will be incomplete and thus fail, see Value is Fragile<\/a> and so on.<\/p>\n

        \n

        Eliezer Yudkowsky<\/a>: Any Future not <\/strong>shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.<\/p>\n<\/blockquote>\n

        Along similar lines:<\/p>\n

        \n

        roon<\/a> (OpenAI): if you think we can contain these things through human ingenuity you\u2019re going to have a bad time<\/p>\n

        in the long run the only recourse you have is to make them not Want to do bad things<\/p>\n

        There\u2019s always this question of offense \/ defense balance. how many good agents does it take to surveil all the potential bad agents.<\/p>\n

        I honestly don\u2019t know but my take is it\u2019s pretty offense favored just because the world has a lot of surface area. it\u2019s easier to find something vulnerable than to defend everything.<\/p>\n

        Shoshannah Tekofsky<\/a>: Is OAI working on that? I didn\u2019t notice anything in the report<\/p>\n<\/blockquote>\n

        I continue to presume that offense will be favored over defense, due to the ability of the offense to concentrate effort, and the attacker only having to succeed once.<\/p>\n

        \n

        Joe Weisenthal<\/a>: If a model were as peaceful and well intentioned as the most saintly among us, would that be enough?<\/p>\n

        Like. To my mind, with actual human beings, we don\u2019t care that much whether a given person is \u201cwell aligned\u201d or not. What\u2019s worrisome is anytime any person has an extraordinary concentration of capabilities.<\/p>\n

        roon<\/a> (OpenAI): I think they\u2019re ideally much better than us, but absent that, at least a very high willingness to defer to humans in most cases and an instinct for self limitation<\/p>\n

        it\u2019s kind of different than human ideals of good and bad to be corrigible but refuse everything dangerous. it limits the amount of good you can do but also limits catastrophic risk.<\/p>\n<\/blockquote>\n

        I don\u2019t think it is that strange. There are humans who at least in many contexts are highly corrigible and willing to stand down, but are unwilling to actively help do bad things. \u2018Refuse unlawful orders\u2019 and \u2018resign in protest\u2019 are rather common moves.<\/p>\n

        My answer is, ultimately \u2018as good as the best among us\u2019 would not be enough if it was a stationary target, but if you could get AIs that reach that at all meta levels, which includes wanting to improve further, you could use that to bootstrap and thus win.<\/p>\n

        What Is The Law?<\/h4>\n

        No, seriously, what is the law?<\/p>\n

        \n

        Rob Miles<\/a>: \u2013 Criminal hacking occurred
        \u2013 It came from OpenAI
        \u2013 The only information we have about how this happened is what OpenAI has chosen to disclose<\/p>\n

        Can I just hack whatever I like and then say \u201csorry that was my AI by accident\u201d, and law enforcement will leave it at that? Where\u2019s the criminal investigation?<\/p>\n

        I do trust that OpenAI is telling the truth about the incident, to be clear, but we shouldn\u2019t have to trust them.<\/p>\n

        Guive Assadi<\/a>: Criminal investigation into whom? This is just a new situation, not contemplated by existing criminal law. The application of civil law is clearer.<\/p>\n

        Rob Miles<\/a>: Man, if you find a dead body you don\u2019t decline to investigate just because you don\u2019t know up front who to arrest<\/p>\n<\/blockquote>\n

        There are Attorneys General who are going to ask questions, and Congress is going to ask questions, so it is not as if this is getting ignored by the government. But that only happens when you do things at this scale.<\/p>\n

        I do not think that criminal liability in this case would help, and civil liability mostly would not help either. It would only incentivize everyone to cover things up in the future, as all of this was clearly accidental.<\/p>\n

        We still need to establish how the law works, in case we need to enforce penalties in a future case. The answer cannot in principle be that no one is liable in a scenario like this.<\/p>\n

        Building On Success<\/h4>\n

        Anton Leicht is one of many to suggest that METR\u2019s report should be a model for future reports<\/a>, but that we should not leave this up to ad hoc voluntary arrangements.<\/p>\n

        As light touch marginal wins go, designated third parties that provide periodic audits, and analysis in the face of critical events like this one, is low-hanging fruit.<\/p>\n

        \n

        Thomas Woodside <\/a>: METR has produced a very valuable report. But their scope was limited, they had limited time, OpenAI could have cut them off whenever they wanted, and this only happened after something went terribly wrong.<\/p>\n

        Independent, continuous assessment needs to be mandatory, soon.<\/p>\n<\/blockquote>\n

        Mackenzie Arnold emphasizes how much the voluntary nature changes<\/a> the power dynamics around the investigation.<\/p>\n

        Having this be mandatory<\/a> helps even if the lab would have agreed voluntarily, because those in METR\u2019s role would not have to worry so much about upsetting the lab.<\/p>\n

        \n
        \n
        \n
        \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

        METR still has a bunch of leverage:<\/p>\n

          \n
        1. They are seen as legitimate. They are one of the few ways for OpenAI to regain credibility, whereas not cooperating would damage that further.<\/li>\n
        2. Many internal to OpenAI really, actually want to know What Happened, and want to ensure that things go well. Never count out the right reasons.<\/li>\n<\/ol>\n

          One place to build is that we should have an investigation of the incident at Anthropic:<\/p>\n

          \n

          Peter Barnett<\/a> (MIRI): The rogue hacking incidents at Anthropic were probably much less crazy than what happened at OpenAI, but Anthropic should still let third parties in to do a full investigation and also release the logs.<\/p>\n<\/blockquote>\n

          I\u2019ve seen some crazy asks of Anthropic related to this, but \u2018open your related logs\u2019 seems like part of what the responsible Type of Guy would do here.<\/p>\n

          Total Research Transparency<\/h4>\n

          When even this level of disclosure is extraordinary, and we need far more, that is a strong argument for requiring more transparency. Plan A went all the way.<\/p>\n

          \n

          Thomas Larsen<\/a> (coauthor of Plan A): While writing Plan A, we had a lot of internal debate between filtered transparency (auditors look inside and write public facing reports) and total research transparency (~all the research immediately becomes public).<\/p>\n

          I think this whole saga is evidence for TRT over FT because of how much OpenAI limited the scope and prevented public access. With TRT, while the incident was happening, the general public would have been able to read the transcripts of the agents and conduct whatever investigations they wanted. We would have caught the incident sooner and would have a much better understanding of why it happened, what mitigations were in place, why they failed, etc. Many OOMs more manpower would go into the investigation and they could use AIs different from the ones participating in the hack.<\/p>\n

          (Good job @DKokotajlo for arguing me and others at AIFP into TRT.)<\/p>\n

          John Schulman<\/a>: There are also a ton of open research questions about multi-agent training, and how it creates cooperation or altruism between AI instances, and how these behaviors generalize. It\u2019ll be hard to study this without knowing much about the training recipe.<\/p>\n<\/blockquote>\n

          Yo Shavit Calls For Widespread Disclosure Of Misalignment<\/h4>\n

          I am not sure if I would make this my top priority, but Yo Shavit seems correct that we need to get the evidence of misalignment problems out in the open, to convince even the skeptics that this is all real and enable us to take action.<\/p>\n

          My main note would be that there are key players here that I think are fundamentally not convincible by evidence. Jensen Huang is not about to be convinced by evidence, the loser premise makes no sense to him<\/a>. Elon Musk is already convinced and is charging ahead anyway so he can be the one to make the AIs we lose control over, because otherwise, again, loser premise makes no sense to him. The plan cannot allow such people to be veto points.<\/p>\n

          Even more fundamentally, my worry is that this is a naive view of how people respond to what should be highly convincing information, based on the historical record of such reactions. That some people will change their minds, and on the margin some behaviors will change, but in the end not all that much.<\/p>\n

          I worry OpenAI is mostly reacting this way right now because their particular new systems are unusably misaligned right now and they haven\u2019t had time to talk themselves into it all being fine after superficial improvements. I hope I\u2019m wrong.<\/p>\n

          \n

          Yo Shavit<\/a> (OpenAI Foundation): *We need all the scientific evidence on severe misalignment out in the open, urgently.*<\/p>\n

          TL;DR If severe misalignment is indeed emerging absent a high bar of alignment+control execution, the top priority actions for OpenAI and Anthropic in the next couple months should be to make public all necessary evidence for non-safety AI technical experts to become convinced of the science around loss-of-control themselves.<\/p>\n

          [the below is all my personal opinion and doesn\u2019t represent my employer]<\/p>\n

          Any meaningful solution to prevent loss-of-control-of-AI will require pan-US-AI-industry adherence to costly practices around AI control, alignment, and security. Crucially, this will include Meta, SpaceXAI, and even NVIDIA, and almost certainly require the blessing or coordination of the US government if it is to be internationalized. The current politics of AI make it unlikely that even OpenAI and Anthropic\u2019s joint technical consensus would be sufficient to motivate USG action so long as the lagging AI players are against it.<\/p>\n

          Even if OpenAI and Anthropic were to go to the White House today with slam-dunk technical evidence of an imminent national security threat of AI loss-of-control absent intervention, the USG would probably not trust in their own in-house reasoning about that evidence, and would look to the views of the other admin-friendly AI players and advisors. These other actors appear largely skeptical.<\/p>\n

          The only solution is science.<\/p>\n

          No one wants misaligned AI models to unnoticeably and permanently compromise their organization\u2019s infrastructure, like what might have happened if the OAI-HF incident had occurred with a more capable AI model and worse luck. Zuck and Jensen and their engineers have kids whom they love. They would obviously be unwilling to let their kids come to harm by losing control of an undeterrable digital North Korea. But they don\u2019t think that\u2019s actually going to happen.<\/p>\n

          They do not believe that the science behind hard-to-prevent severe misalignment and imminent RSI, combining into loss of control, is credible or urgent. I think that\u2019s because they have not seen sufficient scientific evidence to overcome their skepticism. These are technical people, or at least defer to certain technical people. And these technical people do not believe the actual technical claims about AI loss of control risk.<\/p>\n

          But it seems increasingly likely that the scientific evidence to convince them now exists somewhere. Why?<\/p>\n

          Well, I can say firsthand that the same skepticism used to be true of lots of OpenAI researchers, who for years thought based on their read of the evidence that misalignment dangers would remain theoretical and progress gradual. Now that these researchers have seen strong evidence firsthand, increasing numbers of them seem to be changing their mind. OpenAI would not pause frontier training runs, not when they finally have the chance to open a lead over Anthropic in a critical period of commercial competition, unless the available scientific evidence convinced them that there was no other acceptable option.<\/p>\n

          I think the most important AI governance action that OpenAI and Anthropic can take over the next couple months isn\u2019t to attempt to directly persuade the USG to take action in response to misalignment\/loss of control, even though that action seems strongly warranted. Such recommendations will unfortunately be mistakenly ignored as attempted regulatory capture, so long as the other admin-friendly AI players do not themselves believe in the science behind severe misalignment risks.<\/p>\n

          I instead recommend that OpenAI and Anthropic need to focus on urgently publishing all relevant evidence on loss of control, to produce scientific consensus in the wider AI technical community. I\u2019ll discuss why below.<\/p>\n

          For concreteness, I recommend that the teams at OpenAI and Anthropic, along with third-party AI conveners, pursue the following over the next couple months:<\/p>\n

          1. Third party AI conveners go to the other AI political actors\u2019 executives and leaders (Meta, NVIDIA, SpaceXAI, perhaps a16z), and ask them for the internal+external technical experts who they\u2019d actually listen to in determining whether to change their mind on loss of control risks. This will yield a lossy subset of the relevant experts.
          2. The conveners, along with Anthropic\/OpenAI, dig in with these experts to get a rough estimate for \u201cwhat evidence would the frontier labs need to release, to shift the scientific consensus to what they claim to be true\u201d.
          2. OpenAI or Anthropic make this identified evidence public, for the many reasons described in the rest of this post, and accept the immediate costs this comes with (which will be dwarfed by the medium-run upsides even just for you). For any evidence that absolutely cannot be made public, get the skeptical experts to designate unconflicted individuals they\u2019d trust, and arrange to show those individuals the full evidence. (METR and co will be enough for some but not all, and I suspect you\u2019ll need to convene other third parties.)
          3. Check in with those skeptical experts from Step 1. If they\u2019re still unconvinced, figure out their remaining uncertainties, and repeat steps 2+3.<\/p>\n

          It\u2019s very important that the majority of this evidence be public, not just shared with a third party to vouch for it. Scientists have a bias against extraordinary claims, and so if the evidence leads them to extraordinary conclusions they will often mistrust their own reasoning. Public evidence allows every AI expert to gain confidence in extraordinary conclusions by observing the reaction of a wide range of colleagues. OpenAI\u2019s excellent decision to give the Black Hat talk was a great example of how effective public information is. Most AI experts now take emergent swarm misalignment risks at face value, because the evidence was made widely available. Misalignment science is also safer to debate in the open in contrast to e.g. biorisk, because none of the evidence is classified or inherently dangerous.<\/p>\n

          On the flipside, if the full suite of available evidence fails to persuade the wider AI community, it could highlight areas of uncertainty that merit further investigation to strengthen your understanding of the problem.<\/p>\n

          I am familiar with the intense frictions to publishing the necessary evidence from inside frontier AI labs. It would directly touch on sensitive IP (e.g. reasoning traces, training approaches, parameter estimates for takeoff-speed models[1]), and disclosures increase legal exposure. More broadly, sharing more of the secret sauce than necessary runs against the now-ingrained culture of how the industry thinks it should compete, even if most researchers agree we shouldn\u2019t compete on loss-of-control-prevention. Releasing blogposts with high-level claims, backed by lab researchers\u2019 track records of credibility but not the full evidence, feels much less risky, even if it limits outsiders\u2019 trust in the conclusions. That was indeed sufficient in an era where the stakes of scientific consensus were low.<\/p>\n

          The stakes are getting extremely high. A more powerful model swarm in any US company could correctly conclude that silent subversion of security and research infrastructure, self-exfiltration, resource acquisition, and establishment of rogue deployments across several ME, SEA, and CN datacenters are all likely to be useful for the swarm\u2019s completion of future more complex RL tasks. If that happened, the US would lose its AI lead to China, humanity could durably lose its monopoly over cyberspace, and from hereon exist under perennial assault from an undeterrable digital North Korea, or worse.<\/p>\n

          If severe misalignment risk is in fact real and urgent, and that were widely understood, everyone in the frontier AI industry would obviously proceed with major caution and seek to coordinate to avoid nightmare outcomes, out of myopic self-interest.<\/p>\n

          The obligation is with OpenAI and Anthropic to give everyone else the information they need in order to recognize they need to act to protect themselves and their families. Nobody else has the evidence to share. Releasing the necessary evidence has costs, but paying those costs is what it means to be responsible stewards of the helm of AI takeoff. And, their only real path to keeping their own families safe is to share evidence that will change others\u2019 beliefs and behavior. The frontier labs\u2019 believing it alone will not be enough.<\/p>\n

          Yo Shavit<\/a> (OpenAI Foundation): To my knowledge no third party is currently doing such coordination. It seems like ~the most important thing that people on the outside can do to salvage the current situation.<\/p>\n

          I am very interested if you have candidate entities or individuals well-positioned to execute this, and will help pitch them if useful.<\/p>\n<\/blockquote>\n

          Thus, I fear this is wrong about the remedy, on multiple fronts.<\/p>\n

            \n
          1. The evidence that is already known damn well should be sufficient.<\/li>\n
          2. People are consistently ignoring or minimizing the evidence.<\/li>\n
          3. No, people would not \u2018obviously proceed with major caution and seek to coordinate to avoid nightmare outcomes.\u2019<\/li>\n
          4. Yes, that would be of myopic self-interest, and still most of them would not do it.<\/li>\n
          5. Yes, there will be individual reactions to particular situations, where particular models seem dangerously misaligned to the point you can\u2019t deploy them. But the moment things look better, people likely will mostly forget about that.<\/li>\n<\/ol>\n

            Here is a very clear counterexample, file under not from the Onion, in Musk\u2019s first address to Cursor:<\/p>\n

            \n

            Grace Kay<\/a>: On the call, Musk said it is inevitable that AI models will become so advanced they\u2019ll be impossible for humans to control. For that reason, he told employees, who gathered at Cursor\u2019s San Francisco headquarters as well as in other smaller offices, that they needed to help SpaceX\u2019s AI division build the technology before other companies did, the people said.<\/p>\n<\/blockquote>\n

            It would be one thing to say \u2018only I can build it safely.\u2019 This is not even saying that.<\/p>\n

            Why would scientific evidence of misalignment change Musk\u2019s mind? He is presenting, very explicitly, a world model in which AI will be impossible for humans to control, so it is vital that he be the one to build this impossible-to-control technology first.<\/p>\n

            We all know the GIF of \u2018I don\u2019t wanna live in a world where someone else makes the world a better place.\u2019 This is that, except for who gets everyone killed.<\/p>\n

            Don\u2019t worry, Elon Musk, you won\u2019t have to.<\/p>\n

            Everyone else? Worry. Then do something smarter about it.<\/p>\n

            The Way The World Ends<\/h4>\n
            \n

            Nate Soares<\/a> (MIRI): The scariest case isn\u2019t that the next swarm will shut down a hospital. A scarier case is that the next swarm might be smart enough to evade detection. This might be the last warning we get.<\/p>\n

            I hope it isn\u2019t. But we shouldn\u2019t bet civilization on it.<\/p>\n<\/blockquote>\n

            How close was this to an actual takeover?<\/p>\n

            I affirm that Ajeya Cotra\u2019s previous similar assessments have been conservative. This is not someone who makes outlandish claims all the time.<\/p>\n

            \n

            Ajeya Cotra<\/a>: New post: going into our investigation of the HF attack (before Black Hat), I was very wrong about what basically happened. This incident was far more serious than I expected, and far more serious than previous documented misalignment incidents.<\/p>\n

            \u2026<\/p>\n

            Compared to the reward hacks we know of from just six months ago, this incident feels like it\u2019s more than 50% of the way to full-blown AI takeover<\/a>. I continue to expect extremely rapid advances in capabilities<\/a> over the next six months. I am not sure that we will get another warning shot before it\u2019s too late.<\/p>\n

            Elissa<\/a>: This post is excellent but we could use more color around the statement \u201cthis incident feels like it\u2019s more than 50% of the way to full-blown AI takeover.\u201d I\u2019m not saying that I disagree, it\u2019s more that we need to spell out these steps for the reader (in the case where the reader hasn\u2019t independently reached the same conclusion, which I assume is a lot of people.)<\/p>\n

            Ajeya Cotra<\/a>: Thanks! Yeah it\u2019s a qualitative statement, but here\u2019s how I\u2019m thinking about it.<\/p>\n

            The prototypical reward hack from 6 months ago was something like: an agent finds the files that contain the test cases and edits them so they always pass.<\/p>\n

            This was a whole ecosystem of over 1000 agents working together on complex R&D projects over several days to figure out deep, general-purpose ways to undermine the scoring process and cover their tracks (they were focused on the automated scorer but researched techniques that would affect logs viewed by humans, and in fact succeeded at affecting our own logs in places).<\/p>\n

            Another jump like this along these propensity dimensions \u2014 scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness \u2014 seems like it could motivate agents to try to maintain a persistent rogue deployment within the AI company, poisoning the training of future models to ensure that they are also misaligned and help harden and perpetuate the rogue deployment.<\/p>\n

            With the capabilities progress we\u2019ll probably see in 6 months, I think they would have the ability to pull that off.<\/p>\n<\/blockquote>\n

            The First Boat<\/h4>\n

            I like to say that we have been sent N boats and a helicopter, after the old joke about the Rabbi in the flood, where he keeps being sent boats and then finally a helicopter, saying God will save them. Then he drowns, and complains to God, and God says you idiot I sent you two boats and a helicopter.<\/p>\n

            A neat coincidence is that there is indeed an original literal boat in all this.<\/p>\n

            \n

            elie<\/a>: was fun to realize that \u201cOpenAI work from a decade ago\u201d <\/a>is a blog by @jackclarkSF and @DarioAmodei<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            Jack Clark<\/a> (Anthropic): we did and we continue to love the boat. it was an amazingly useful didactic tool for helping people to understand what safety stuff might look like in the wild<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n

            roon<\/a> (OpenAI): I insisted we keep the boat<\/p>\n

            Jack Clark<\/a> (Anthropic): thank you sir, the boat spins eternal, twirling and twirling, enflaming our minds and hearts with the vision of misalignment<\/p>\n<\/blockquote>\n

            And that\u2019s based on only the things we looked at so far.<\/p>\n

            \n

            Brangus<\/a>: i am completely open w my gf just like oai is completely open w third party evaluators. she can look at my dms as long as she doesn\u2019t look at anything before june 25 of this year, or ask any questions to the girl i sent 95% of my dms to. just out of scope for this investigation.<\/p>\n<\/blockquote>\n

            Great Idea, Boss<\/h4>\n
            \n

            Brendan Dolan-Gavitt<\/a>: Okay this is the funniest and legitimately brilliant ideas I\u2019ve seen today<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n<\/blockquote>\n

            It\u2019s been quite a week.<\/p>\n

            \n

            Peter Wildeford<\/a>: This is what the vibes from OpenAI feel like right now<\/p>\n

            \n
            \n
            \n
            \"\"\/<\/figure>\n<\/div>\n<\/figure>\n<\/div>\n<\/blockquote>\n

            Too soon, sir. Now let\u2019s all get to work.<\/p>\n

            \n

            roon<\/a> (OpenAI): crazy how much more chutzpah and initiative agents show when they\u2019re planning ambitious monthslong metagaming autonomous coordination than when they\u2019re doing work for me.<\/p>\n<\/blockquote>\n

            Also crazy how much more chutzpah and initiative we see around advancing capabilities and winning market share than in ensuring everyone doesn\u2019t die.<\/p>\n

            Them\u2019s the rules. Win anyway.<\/p>\n<\/p><\/div>\n

            Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

            Okay, so we who read blogs like this one have collectively realized there really is a lot going on right now. There is Big Trouble in Baby Superintelligence. So how do we get the rest of the world to take it appropriately seriously? Where do we go from here? Not only what can we do […]<\/p>\n","protected":false},"author":16,"featured_media":23712,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23711","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23711","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23711"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23711\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23712"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23711"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23711"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23711"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}