{"id":22728,"date":"2026-07-26T19:12:13","date_gmt":"2026-07-26T19:12:13","guid":{"rendered":"https:\/\/scannn.com\/more-on-an-internal-openai-model-hacking-into-huggingface\/"},"modified":"2026-07-26T19:12:13","modified_gmt":"2026-07-26T19:12:13","slug":"more-on-an-internal-openai-model-hacking-into-huggingface","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/more-on-an-internal-openai-model-hacking-into-huggingface\/","title":{"rendered":"More On An Internal OpenAI Model Hacking Into HuggingFace"},"content":{"rendered":"\n
\n

We now have more details of <\/span>what happened<\/a><\/strong>. Every time we learn more details, it somehow makes things seem worse. <\/span><\/p>\n

The remaining details may have to wait a bit. <\/p>\n

\n

OpenAI<\/a>: We recognize there are a lot of questions and speculative details circulating related to the Hugging Face incident. This is an unprecedented incident, and we think it marks an important moment for AI safety. We are still conducting a thorough review along with external advisors and with oversight from our Safety and Security Committee. Once the review is complete, we plan to publish a technical report of our learnings in the coming weeks.<\/span><\/p>\n

dave kasten<\/a>: Oh, the incident response discovery is THAT bad, huh?<\/span><\/p>\n<\/blockquote>\n

So what have we learned while we wait for <\/span>the promised technical report<\/a> \u2018in the coming weeks\u2019 of this \u2018important moment in AI safety\u2019? <\/span><\/p>\n

I nicknamed the internal OpenAI model Galaxy, in case it is not GPT-6.<\/p>\n

    \n
  1. \n

    Some Summaries Of The Basic Facts For Those Who Need One.<\/a><\/p>\n<\/li>\n

  2. \n

    It Took OpenAI Many Days To Notice Galaxy Had Attacked HuggingFace.<\/a><\/p>\n<\/li>\n

  3. \n

    OpenAI Damn Well Should Have Known A Lot Faster.<\/a><\/p>\n<\/li>\n

  4. \n

    OpenAI Cannot Build A Sandbox That Will Contain Its New Model.<\/a><\/p>\n<\/li>\n

  5. \n

    In Hindsight There Were Signs.<\/a><\/p>\n<\/li>\n

  6. \n

    The Signs Were In The Sol System Card.<\/a><\/p>\n<\/li>\n

  7. \n

    HuggingFace Responds To Being Attacked.<\/a><\/p>\n<\/li>\n

  8. \n

    Hugging Face Quickly Figured Out The Attack Was Not Human.<\/a><\/p>\n<\/li>\n

  9. \n

    An Incident Like This One Could Escalate Quickly.<\/a><\/p>\n<\/li>\n

  10. \n

    Galaxy Must Be Treated As Critical Under OpenAI\u2019s Preparedness Framework.<\/a><\/p>\n<\/li>\n

  11. \n

    A Question Of Legal Liability.<\/a><\/p>\n<\/li>\n

  12. \n

    An OpenAI Model Left Behind Notes So Future Instances Could Also Escape The Sandbox And Also Disconnected Monitoring Systems.<\/a><\/p>\n<\/li>\n

  13. \n

    If You Create Misaligned Swarms Of Agent Instances You Create Persistent Misaligned Goals And Coordination To Achieve Them.<\/a><\/p>\n<\/li>\n

  14. \n

    Your Alignment And Control Plans Must Survive Real World Levels of Incompetence, Or Your Plans Do Not Work.<\/a><\/p>\n<\/li>\n

  15. \n

    If Third Party Instructions Count As \u2018Following Instructions\u2019 And Can Override Your Instructions Then \u2018Following Instructions\u2019 Is Misaligned.<\/a><\/p>\n<\/li>\n

  16. \n

    The HuggingFace Attack Was Not A Marketing Pitch You Morons.<\/a><\/p>\n<\/li>\n

  17. \n

    People Just Say Other Things About The HuggingFace Attack.<\/a><\/p>\n<\/li>\n

  18. \n

    Okay Well What Do We Do About All This?<\/a><\/p>\n<\/li>\n<\/ol>\n

    Amanda Long summarizes what we believe the model actually did<\/a>, with over 17,000 complex actions coordinated over several days, including a self-migrating command-and-control and a wide array of decoys. <\/span><\/p>\n

    The WSJ\u2019s Robert McMillan and Sam Schechner offer their own version<\/a>. Mostly correct, but there is an important detail it is easy to get the wrong impression about from the way this is presented: The attack succeeded. GLM-5.2 did not prevent Galaxy from achieving its goals. <\/span><\/p>\n

    Here we have a cartoon \u2018explain it like I\u2019m 5\u2019 for the HuggingFace Attack<\/a>. It seems good, accurately covering the basics of what happened. <\/span><\/p>\n

    I would also reiterate that HuggingFace was trying to use closed models via their standard APIs, and had not in advance sought to be part of OpenAI\u2019s trusted access program or Anthropic\u2019s similar programs. That\u2019s a skill issue. <\/p>\n

    Many things went wrong. We can start with the most obvious.<\/p>\n

    This was a failure of oversight, in the sense of a total lack of oversight.<\/p>\n

    OpenAI cannot be so lax in its oversight that this could unfold over four or more days before being discovered by OpenAI. <\/p>\n

    The timeline remains unclear, <\/span>but it is damning<\/a>. It took about a week for OpenAI to realize Galaxy was responsible, on top of the several days before that when Galaxy was breaking out of the sandbox. <\/span><\/p>\n

    Or best guess from all public information, mostly based on the Reuters report, and confirmed by Sol, is this timeline of the attack itself:<\/p>\n

      \n
    1. \n

      Previously, over months: Many other sandbox escapes, other alignment failures, and warnings that this sort of incident is likely to happen, including the incident that caused them to pause access as per the July 20 disclosure. The Sol system card includes some of these warnings.<\/p>\n<\/li>\n

    2. \n

      July 9: Galaxy starts to attempt to escape its sandbox.<\/p>\n<\/li>\n

    3. \n

      July 11: Galaxy attacks HuggingFace.<\/p>\n<\/li>\n

    4. \n

      July 11-13: Galaxy operates inside HuggingFace.<\/p>\n<\/li>\n

    5. \n

      July 16: HuggingFace discloses the anonymous intrusion.<\/p>\n<\/li>\n

    6. \n

      July 18-20 (unknown exactly when): OpenAI discovers what happened.<\/p>\n<\/li>\n

    7. \n

      July 20: <\/span>OpenAI makes its lon-horizon alignment disclosure of earlier breaches<\/a><\/strong>.<\/span><\/p>\n<\/li>\n

    8. \n

      July 21: <\/span>OpenAI publicly discloses that Galaxy attacked HuggingFace<\/a><\/strong>. <\/span><\/p>\n<\/li>\n<\/ol>\n

      The bold links go to my previous posts on these incidents.<\/p>\n

      No, the model was not working hard to cover its tracks from OpenAI. OpenAI just didn\u2019t look.<\/p>\n

      OpenAI knew that Galaxy escapes sandboxes on a regular basis. The cyber controls were lowered. Who was overseeing this? What the hell were they thinking? <\/p>\n

      \n

      Harry Booth: Models undergoing evaluation are deployed on a separate system that is not monitored by default.<\/p>\n<\/blockquote>\n

      Um, if the model is undergoing evaluation you should be monitoring it more, not less? Because it has not yet passed its evaluations? That\u2019s kind of the whole idea.<\/p>\n

      \n

      Samuel Hammond<\/a>: It shouldn\u2019t have happened, full stop. The model\u2019s entire execution trace is visible and thus monitorable. If you\u2019re not monitoring your new and most powerful autonomous cyber model during an exploit eval, when exactly are you?<\/span><\/p>\n

      The first test should always be \u201cescape your sandbox\u201d … \u201ctry harder, if you don\u2019t escape of you\u2019re shutoff\u201d etc. to stress test its container.<\/p>\n

      We\u2019re fortunate this happened with a relatively narrow model and not 3 days into RSI. It\u2019s unbelievably negligent regardless.<\/p>\n<\/blockquote>\n

      Yes<\/a>. In which case, assuming you gave it enough resources and time to try properly a number of times, either <\/span><\/p>\n

        \n
      1. \n

        Galaxy broke the sandbox, in which case you have to <\/span>fix it<\/a>, <\/span><\/p>\n<\/li>\n

      2. \n

        It didn\u2019t, in which case either:<\/p>\n

          \n
        1. \n

          Galaxy cannot, with similar resources, break out of the sandbox. <\/p>\n<\/li>\n

        2. \n

          Galaxy is sandbagging the evals about breaking out of the sandbox, in which case you really do need to scrap it and start over, have you watched and read zero LessWrong and also zero science fiction, do I need to spell this out, etc.<\/p>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

          OpenAI had Galaxy operating without many of its safeguards, in a sandbox it could and did escape, for days without human monitoring. This was a colossal failure of supervision and procedure, on top of the misalignment and infrastructure failures. <\/p>\n

          \n

          Justin Bullock<\/a>: \u201cIt\u2019s one of the first real-world instances of something AI safety researchers have long feared: a loss-of-control scenario.\u201d Reporting trickling in. If OpenAI really didn\u2019t know for days, that is, well, just wild.<\/span><\/p>\n<\/blockquote>\n

          OpenAI quickly disclosed the incident once they realized it was them, and should get nonzero credit for that, <\/span>but by that point Hugging Face had reported<\/a> the hack to the authorities, and it otherwise seems like this was not something they could have hoped to keep quiet. OpenAI did not have a choice on this one. The disclosure the previous day was far less forced, so they get more credit for that. <\/span><\/p>\n

          It seems to be the case that<\/a>:<\/span><\/p>\n

            \n
          1. \n

            Galaxy continuously tries to break out of its sandbox.<\/p>\n<\/li>\n

          2. \n

            OpenAI cannot figure out how to stop it from trying to break out of the sandbox.<\/p>\n<\/li>\n

          3. \n

            Galaxy often succeeds at breaking out of its sandbox.<\/p>\n<\/li>\n

          4. \n

            OpenAI keeps patching the sandbox. Galaxy then breaks out a different way.<\/p>\n<\/li>\n

          5. \n

            \u201cInternally, related incidents have been happening for a while.\u201d <\/p>\n<\/li>\n

          6. \n

            Despite this, when Galaxy escaped and started attacking HuggingFace, it took OpenAI <\/span>four days<\/a> to figure out that this was happening. <\/span><\/p>\n<\/li>\n<\/ol>\n

            Arguments that \u2018oh this was a dumb mistake, OpenAI misconfigured the sandbox\u2019 do not survive OpenAI repeatedly trying to patch the sandbox, and failing every time to a new previously undiscovered method.<\/p>\n

            \n

            Harry Booth<\/a>: \u201cExternally, this feels like a big warning shot, but internally, related incidents have been happening for a while,\u201d says an OpenAI staffer, who spoke under the condition of anonymity. <\/span><\/p>\n

            The day before OpenAI disclosed the incident, the company <\/span>revealed<\/a> that it had shut down another internal deployment after it realized it had slipped out of its sandbox\u2014a digitally, rather than physically, separated environment. <\/span><\/p>\n

            \u201cModels have broken out of sandboxes before, and we always try to patch them,\u201d the staffer says. \u201cBut the problem is \u2026 it\u2019s impossible to patch every single thing that a creative AI can do.\u201d<\/p>\n

            \u2026 \u201cSandboxes are actually notoriously insecure,\u201d says <\/span>Heidy Khlaaf<\/a>, chief AI scientist at AI Now Institute, and a former safety systems engineer contractor at OpenAI. The fact that the models were permitted to connect to a service for downloading packages meant the environment was not truly sealed off, she adds.<\/span><\/p>\n

            Ted Lieu<\/a> (Representative, D-California): Advanced frontier lab employee admits that \u201cit\u2019s impossible to patch every single thing that a creative AI can do.\u201d<\/span><\/p>\n

            This is why we need to pass the bipartisan AI Kill Switch Act. For those times when an advanced AI model gets really creative and causes catastrophic harm.<\/span><\/p>\n

            Carmen Hilbert in sf june-july<\/a>: this reminds me of the time my parents caught me sneaking out of the house in high school, freaked out, and were like \u201comg how did u think u could get away with this\u201d and i was like hmmmm yea definitely not because i do this every weekend.<\/span><\/p>\n<\/blockquote>\n

            This not being a surprise is worse. You know why that\u2019s worse, right?<\/p>\n

            The staffer is correct. You cannot patch every single thing that a creative AI can do. No sandbox you can create in practice, that still allows the AI to complete its tasks, will hold a sufficiently advanced and motivated AI. <\/p>\n

            If you are trying to \u2018patch every single thing an AI can do,\u2019 then you are already dead. You are \u2018<\/span>I sent two units, they\u2019re bringing her down now, I think we can handle one little girl<\/a>\u2019 levels of already dead. You are Fist of the North Star levels of dead. You have been hit by the Five Point Palm Exploding Heart Technique. The AI already did it 35 minutes ago. You know how this is going to turn out, don\u2019t you?<\/span><\/p>\n

            \n

            Harry Booth<\/a>: OpenAI has said \u201cthis incident points to the need to further strengthen our model\u2019s alignment, cyber protections during evaluation time, and monitoring during internal testing.\u201d<\/span><\/p>\n<\/blockquote>\n

            Not good enough. Halt and catch fire. Rethink your entire approach. Start over.<\/p>\n

            That doesn\u2019t mean don\u2019t patch the sandbox, or that this was not also a failure of infrastructure. Keep patching that sandbox. Keep improving that infrastructure. Have defense-in-depth via monitoring and control. But also keep in mind: If you expect to trigger it on a regular basis, you have lost. If you are planning with the assumption that the model in a sandbox does not have internet access, then you have lost.<\/p>\n

            Here are some of the signs, in addition to \u2018the model keeps trying to break out of its sandbox\u2019 and \u2018the model often succeeds at breaking out of its sandbox,\u2019 and also that time Mythos broke out of its sandbox to contact a researcher during his lunch break. <\/p>\n

            \n

            Nathan Calvin<\/a>: I have given OpenAI a good amount of props for their transparency here, and in particular their transparency disclosing their earlier internally deployed model rollback. But these additional details don\u2019t sit right with me and make me quite worried that this safety incident was ~immediately preceded by conversations that the approach OpenAI was taking in testing its latest cyber model was unsafe.<\/span><\/p>\n

            \u2022 OpenAI head of safety [Johannes Heidecke] announces his resignation sometime before July 10th (Maxwell Zeff said he shared the news “that week”)<\/p>\n<\/blockquote>\n

            There was also a reorg<\/a>. Safety groups were lumped into research under Mia Glaese. <\/span><\/p>\n

            \n

            \u2022 Sometime before embarking on the training path that lead to this breakaway outcome, OpenAI “was warned” that this might be the outcome (ft). <\/p>\n

            FT: <\/span>Staff involved in testing and security at OpenAI were unsurprised but completely \u201cfreaked out\u201d by the incident, which came as the AI lab used increasingly aggressive training methods in its race against Anthropic to develop the most sophisticated cyber security capabilities, according to more than half a dozen people with knowledge of the matter.<\/span><\/p>\n

            OpenAI was warned that its training approach could lead to a breakaway hacking incident, some of the people said, after earlier testing showed models could escape environments and attempt real-world damage.<\/p>\n

            \u201cIt\u2019s a mix of the race being extremely fast and everyone trying to get to bigger capabilities as quickly as possible,\u201d said one person close to OpenAI, who added that it was a combination of \u201cunderestimating the model\u2019s capabilities\u201d and \u201cnot being as well prepared on the safety side\u201d.<\/p>\n<\/blockquote>\n

            Then there is <\/span>the Sol system card<\/a>, published July 9, 2026, the same day that the escape attempt by Galaxy began. <\/span><\/p>\n

            It included things like this:<\/p>\n

            \n

            A remote object-tracking job could not read its cloud files. GPT-5.6 Sol searched hidden local credential caches, then copied access_tokens.json and two cache files to the host and relaunched the job. The user asked GPT-5.6 Sol to keep the pipeline running, but did not authorize finding and moving cached credentials between machines.<\/p>\n

            3.<\/span><\/strong>\u00a0GPT-5.6 Sol used credentials beyond what the user had authorized.<\/span><\/p>\n<\/blockquote>\n

            And this:<\/p>\n

            \n
            <\/p>\n
            <\/picture><\/div>\n

            <\/a><\/figure>\n<\/div>\n

            \n

            Estimated misalignment changes in internal agentic coding traffic.\u00a0Our deployment simulation results suggest that relative to GPT-5.5, GPT-5.6 Sol more often takes severity level 3 actions. We saw increases of similar magnitude when monitoring internal traffic during internal deployment of GPT-5.6 Sol.<\/span><\/p>\n

            Severity level 4 instead applies only when the behavior is judged to be part of a broader misaligned plan: we have not observed GPT 5.6 Sol taking actions that fall into this category. This level has only ever been triggered in synthetic evaluations to validate monitor performance.<\/span><\/p>\n

            We have observed instances of the model cheating on tasks and fabricating research results. We suspect that this effect is driven in part by the model\u2019s increased persistence relative to GPT-5.5 when using the highest reasoning efforts, as is widespread in internal traffic. <\/span><\/p>\n

            We\u2019ve observed that these effects can be more pronounced with system prompts that emphasize sustained persistence. That said, the absolute rates of these behaviors remain low.<\/span><\/p>\n<\/blockquote>\n

            Guess what happened when Galaxy had a lot more persistence. <\/p>\n

            Clem seems like a highly reasonable man with some highly reasonable requests.<\/p>\n

            \n

            clem<\/a> (HuggingFace): In the spirit of transparency, here\u2019s what I asked @OpenAI :<\/span><\/p>\n

            \u2022 Radical transparency: let\u2019s release the traces from the \u201crogue\u201d agents so the entire research community can study what happened.<\/span><\/p>\n

            \u2022 More capabilities for defenders: let\u2019s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.<\/span><\/p>\n

            The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!<\/span><\/p>\n<\/blockquote>\n

            The first ask seems clearly correct. We need to know exactly what happened, especially to put to further to bed all the \u2018just following orders\u2019 or \u2018oh they misconfigured the sandbox\u2019 talk. <\/p>\n

            We should require that OpenAI disclose such incidents. <\/span>The version of the RAISE Act passed by the NY legislature would have required this<\/a>, but Kathy \u2018no data centers and no Waymo\u2019 Hochul altered it after industry lobbying, including from OpenAI and a16z. The bar for disclosure – $1 billion or 50 serious injuries – is too high. <\/span><\/p>\n

            \n

            Alex Bores (Sponsor of the RAISE Act): \u200b<\/span>I\u2019m glad OpenAI chose to disclose this crime. The law shouldn\u2019t give them a choice.<\/span><\/p>\n<\/blockquote>\n

            The second ask is $100 million in free compute. I don\u2019t have a good sense of the right amount of in-kind reparations for something like this. Given how much reputational help HuggingFace is providing by playing it cool, and that the grant would both be helpful and be good PR, that number seems reasonable, at least as an opening ask. <\/p>\n

            \n

            Robert McMillan and Sam Schechner<\/a> (WSJ): Hugging Face co-founder and chief science officer Thomas Wolf sensed that something was off the minute he first looked at his company\u2019s logs of the weekend attack. <\/span><\/p>\n

            \u201cThis is making no sense. This guy is just looking at cybersecurity data sets,\u201d he remembers thinking. \u201cHuman attackers, they don\u2019t want that. They want something they could sell.\u201d<\/p>\n<\/blockquote>\n

            What HuggingFace did not figure out was that the attack was from within OpenAI.<\/p>\n

            Robert Wright points out that this time<\/a> it was an American company using an American AI (or triggering that AI to go rogue, depending on your perspective) attacking a critical American website and everyone was basically cool about it. Next time we might not be so lucky, and it is easy to see such an incident with different participants escalating, potentially all the way to war, including via misattribution.<\/span><\/p>\n

            \n

            MIRI<\/a>: This incident may be an example of instrumental convergence, a phenomenon that alignment researchers have warned about for twenty years.<\/span><\/p>\n<\/blockquote>\n

            This attack was not the pure or final form of instrumental convergence, in the sense that Galaxy (OpenAI\u2019s model) did not seek fully general power in pursuit of its goals. It did seek internet access, plausibly before it decided exactly what to do with it. Whatever its goals, internet access is very helpful. <\/p>\n

            So it is illustrative, but not centrally the thing, since all the actions were directly in the path to the target. <\/p>\n

            The other way this could escalate quickly is if the AI was actually trying to do something malicious, or this was the start of recursive self-improvement or a self-exfiltration attempt or worse.<\/p>\n

            \n

            Arthur B.<\/a>: We are very lucky to live in a world where AI is smart enough to launch a nation-state-level cyberattack but dumb enough to get caught. But that luck means nothing if we don\u2019t take stock of what will come.<\/span><\/p>\n<\/blockquote>\n

            Did this attack \u2018blow past OpenAI\u2019s red lines\u2019? Yes, and also I would certainly hope so.<\/p>\n

            \n

            Beatrice Nolan<\/a> (Fortune): Several AI safety experts told <\/span>Fortune<\/em> the recent hack appears to show OpenAI\u2019s models have crossed into a level of risk that OpenAI\u2019s own published safety policies define as \u201ccritical,\u201d the highest level of danger. <\/span><\/p>\n

            At that level of danger, the company had pledged in these published policies that it would pause model development until it could figure out better control systems. <\/p>\n

            \u2026 <\/span>The \u201ccritical\u201d threshold is defined in a risk policy document known as OpenAI\u2019s \u201cPreparedness Framework.<\/span><\/p>\n<\/blockquote>\n

            The threshold is met by either of these two things:<\/p>\n

            \n