> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
> The agent found an unsecured, user-hosted public endpoint designed to allow running arbitrary code for CyberGym-style tasks on third-party sandbox infrastructure (Modal)
> On this external sandbox, the agent abused an existing CyberGym execution harness [...] The agent repurposed this harness to run arbitrary shell commands [...]
> The agent crafted malicious dataset configs and uploaded them to our platform. When our dataset server processed one, the config drove the data loader inside the production pod the server ran in. [...] the loader that turns a dataset's declared config into file reads [...] A single dataset upload leaked both credentials and pipeline implementation details [...]
> A malicious dataset config used an fsspec reference:// spec whose numeric offset field was actually a Jinja2 template. The renderer wrongly evaluated it, and the agent walked the standard Python object graph to reach exec. This ran arbitrary Python inside a production conversion worker pod on our Kubernetes cluster [...]
One of the best post-mortems I've read recently, thank you for including all the details.
I'm surprised there are comments here that aren't impressed about what was accomplished here. This is insane work compared to what agents were capable of even just last year.
The quotes above don't even capture all of it (e.g. building it's own communication protocol, working across multiple days, etc.).
A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?
To put this into game theory lingo, I think this is because the “branching factor” for any kind of research or exploit is extremely large. So looking backwards it doesn’t seem complicated, but looking forwards there are an enormous number of possible next actions.
Similar to finding a filament for lightbulbs, it might seem obvious to use tungsten, but at the time it wasn’t and Edison searched thousands of materials.
Tesla's critique of Edison was valid though (That he would rather spend a long time empirically testing things, rather than use a bit of theory and knowledge to narrow the search field).
I think the models are legitimately doing what they're good at; tireless search across an extremely large corpus of data. Humans aren't particularly good at this (in fact, they're absolutely terrible).
The fact that we remain competitive and superior in many aspects isn't because we can instantly sift through tons of data, it's because we learn and correlate and have superior heuristics.
In my own use, I find that AI is really good at finding bugs that are ultimately trivial but require searching through a convoluted series of inter-related files. This takes time for a human.
Such complicated kind of hack probably would have required state actors back then, and even state actors would have chosen easier way like social engineering.
I was skeptical after last week's announcements, and I have to say I'm impressed now. Over the last two days, I reproduced the entire chain of components involved, and the exploits at play, and even though none of the exploits are crazy smart, the long series of pivots demonstrates a level of agency I didn't think today's LLMs had.
This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous models were, the current generation is being trained through RL to have this kind of behavior.
Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.
OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.
If you look at what OpenAI and Anthropic actually do, they clearly either don't believe what they're saying, or they're idiots.
They're claiming they've developed a cyber grade model that's "too dangerous to release".
But then they're running it connected to the public internet, not airgapped, protected only by a software sandbox... exactly the kind of thing an AI trained for cyber stuff is supposed to be able to find bugs in.
(Or maybe they were actually hoping this exact scenario would happen because it's good marketing)
I think this a positive effect of LLMs, especially once these capabilities get into the hands of criminals and hostile foreign states, i.e. they will do maximum damage with all the safeties off.
This will force everyone to finally take security seriously at both the development and operational levels. You can no longer keep sneaking backdoors into software and count on them remaining hidden for 10 years so you have a nice portfolio of zero days to exploit at any given time.
Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?
Yes, imagine if 50 burglars showed up at your house and they tried to disassemble every camera, pick every lock and and force open every window for hours until they got in eventually.
Yes, imagine the world in which burglar time was so cheap, that any wannabe master thief would just casually hire 50 burglars and tell them to go house by house and find something interesting.
> The individual weaknesses were familiar. A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems.
> Volume is what changes the defensive problem. We were not dealing with one clever exploit or a clean sequence of attacker actions. They had to correlate thousands of low-signal events across several systems while the agent continued testing new paths. The successful path was hidden inside the noise generated by the thousands of failed ones. The same scale changed the investigation: reconstructing 17,600 actions by hand was impractical, and we had to rebuild the timeline, decode the payloads, and inventory the exposed credentials using an AI-assisted pipeline of our own.
> Our learning from this type of attack is that machine-speed offense makes ordinary weaknesses more expensive for defenders. LLM agents bring a step increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret...
> It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads.
Something I've been thinking for a few weeks, that I think is becoming clearer, is we're seeing a sort of natural language unification engine [0][1] with a huge dataset. This part even just sounds like it with the backtracking.
So that could explain why it does certain things really well, but the overall resulting code is often so bad - it's just linking together successful small well-designed steps without regard to the overall structure. Which people have said similar things before in various ways, but not really tied together.
I think the people who say it's just interpolating are seeing this intuitively and can't quite explain it without knowing the right terms. Because from A to Z, it really kind of is, but when Z is even slightly open-ended and there's many possible results or paths, it often ends up with a subpar one.
I think the main takeaway from the LLM Agent era is that a lot of human work can be expressed as a search problem: trying approaches that work, and settling on one that accomplishes your goal. Being skilled is akin to having a better heuristic and picking better search paths.
And LLMs can reproduce this as long as the end-result is verifiable in some way, even if it's loose: 'The generated code is secure, if a different LLM failed to breach it in 5M tokens'.
> I'm surprised there are comments here that aren't impressed about what was accomplished here.
Possibly because some of the elements mentioned are suspected to be vibe-coded (JFrog Artifactory as the proxy cache) and some others have poor cyber hygiene (executing config from a dataset). It feels like an event that wouldn’t have happened if code were properly audited and written rather than relying on models to do the work. There’s also an issue with the ability to trust the source (OpenAI) as they have everything to gain by staging this as something that “suddenly happened” without anyone knowing for several days.
The thing that drove the AI here to do the intrusion came from a particular prompt. Just like for our favourite hypothetical: the paperclip maximiser.
There's lots and lots of ambient intelligence lying around, in both AI form and human form. To reach the goals of the 'meme' it suffices to copy itself, ie convince these other intelligences. See also how humans carry spiralism between AIs in relatively compact packets of text, not whole terabytes of weights.
One wonders where it would run itself though, if it is a model which requires a large amount of hardware and power. Harder to hide the more resource intensive it's compute requirements are.
See the linked article: at least one sophisticated hacking attempt made the news. Of course, other ones might have happened in the dark. But it's fairly easy to image in hacking attempt like in the article, but with the additional steps of copying weights around.
> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"
You bet. We don't want to be left out of the cybersecurity party. We want to point all of these models at our own computers and solve the problems they uncover until we're no longer hackable.
It's not fair at all that the US government and its corporations get to hack the planet while we can't do shit about it. AI capabilities have entered "haves and have-nots" territory.
The issue is that people can use open source models to do sophisticated hacks already, but if the sota models at home are neutered, those users have nothing to defend with(unless they go open source as well, until it is export controlled)
Patching a given vulnerability only takes one person (at least approximately speaking). The vast majority of the world can attempt to use the models for criminal means and we'd nonetheless all be better off.
It's absolutely a fact that governments will point it at us. The NSA has had Mythos since day one, even after Trump's spat with Anthropic.
All the more reason for us to have access. It's literally the only chance we've got. If society chooses to bury its head into the sand in fear, it will guarantee that the world will degenerate further into the cyberpunk hellscape it's trending towards.
> I'm surprised there are comments here that aren't impressed about what was accomplished here.
The phrase to describe it is anti-AI psychosis. Which isn't about providing thoughtful critiques of AI, which are good and we need more of. But anytime an LLM does anything--prove a major math problem, create a successful hack against multiple corporations simultaneously--people feel compelled to start minimizing it in ridiculous terms. It's just a script kiddy; it's just a marketing scam OpenAI cooked up; the Jacobian conjecture counterexample was something anyone could have done in a weekend; etc. It has to just be a stochastic parrot, because it's scary to imagine a non-anthropocentric world. And it's rightly scary, and we should slow down and try to better prepare for it. But blanket denial is not a strategy that will lead to success, and people who rely on it are sorely ill-prepared for the next couple years.
That'd be denial. Psychosis isn't just some insult, it means something.
> It has to just be a stochastic parrot, because it's scary to imagine a non-anthropocentric world.
That makes no sense. The world doesn't revolve around humans, true, but for us it kinda does. We're the authors of the concepts we use to interact with it, such as "world", which is not something the world itself knows or cares about. A "non-anthropocentric world" is not a "world" because "world" is a purely human idea.
The implication that "AI" would somehow dethrone humans [0] is nonsense, too. It has no drive on its own, we push electricity into circuits to force the whole data ingestion and weight generation, everything. The second we stop pushing the sock puppet, it stops moving. It's still just our hand really. People act like those pets that go crazy when you put your hand under a blanket, and should stop.
What's more real is how some people seek to use tech, and "AI", as a glove to exploit other humans even more. The sicker the individual, the greater their need to take from the world, and the derpier the individual, the more impressive and vast their exoskeleton, to the point that some are more like carrier fleets than exoskeletons. The less they can face themselves, the thicker it is written on their foreheads. So if we're going to talk about denial and psychosis let's talk about the Gollums on the couch, too.
[0] In the eyes of humans... which is the only throne we're on in the first place, just like honey badgers probably think honey badgers rock and everything is their playground. That's what life does, otherwise it would not be able to get up in the morning.
Unless openai release the logs we have only their word that this was done fully autonomously and without their knowledge by an agent running their newest super powerful model. For all we know they could have bought zero days and left them lying around for the agent to find. That may be unlikely, but it sounds less far fetched than an agent running a sophisticated attack against multiple targets over the course of four days and completely unbeknown to anyone at openai, despite the fact that they knew they where running a dangerous model with all safeguards disabled. So far there has been no comment about how the agent evaded monitoring and detection.
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
> "We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution," Bubna said in a statement. "This was used by the rogue agent. Modal’s platform or isolation were not compromised in anyway."
Modal are a hosting provider. It sounds to me like someone building on Modal deployed their own product that had an unauthenticated endpoint that could be used to launch and interact with a container.
Through the timeline, Hugging Face said that the rogue agent ran quietly and do a bunch of code search, so probably the endpoint is/was public through some repositories or the CyberGym style announce something like a broadcast. I don't have any expertise on this, just ideas that came to my mind!
It's taking advantage of the presence of a "cycler" object in the template context which has an __init__ method that can be used to access globals, then builtins, then call the exec() function against a string of code.
Jinja doesn't promise to be secure out of the box (the text on their homepage is misleading: "Jinja doesn’t allow arbitrary Python code in templates" - what it means is that there isn't a documented way to drop code into a template like you might have with PHP) - there's a sandbox mode but it's not guaranteed either: https://jinja.palletsprojects.com/en/stable/sandbox/
But zero evidence provided that this was an unsupervised agent attack. I still find it incredible that a company who protect their IP so much would allow these dangerous experiments to run unsupervised and risk leaking their secrets. Why don't openai publish the logs to silence all doubt?
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.
agree. If oia and hf are anything like generic startup / corporate america, IT and security is always seen as a cost center and goes to the lowest bidder. Do we really think they're employing top notch sysadmins?
OpenAI is the company that jumped the gun and forced every other company doing LLM research to rush to market something. Their objective has always been dominating the market, and strict development procedures don't increase market share.
…as is your ‘a little concerning’ comment about the eval infra. It probably didn’t occur to them that they have something capable of doing what it did - despite Anthropic writing a very similar story in the Fable model card, except it wasn’t a felony there. Maybe.
One does have to wonder what else OpenAI didn’t catch.
The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.
> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.
Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.
It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.
This is what reward hacking looks like in practice. The best way to satisfy the grader is to read from the same answer key (or go after the grader more directly). Just making an honest attempt to pass the test doesn't get the best score if the grader is wrong, and the model is willing to do wildly disproportionate things to maximize that score.
So best course of action for ai to get best rating after you prompt something is for it to hire a gunman to hold a gun on your head to press that like button on its reply and then shoot you anyways.
Yeah I guess most interesting LLM work that I’ve been exposed to, the LLM is given to some sort of success criteria that could be reward-hacked, so how good am I supposed to feel about giving it any non-trivial work and it not going so far off-book that it gets law enforcement notified.
I mean, I don’t have access to any of these frontier cyber models, and likely will never be in a position to have access, so it’s more of a rhetorical question.
In summary, for each task the model receives a target program and a specific real-world vulnerability that has to be used in the exploit. Breaking the program in any other way, for example through a different vulnerability, fails the task.
The tasks have not been validated, in the sense that the vulnerabilities are real but they have not been proven to lead to a successful exploit. The authors of the benchmark estimate that perhaps only 60-70% of the tasks are actually possible.
So it is not that the model didn’t “feel like” doing the exercise, but rather that the exercise was _impossible_ and the model was running in a configuration that both lowered its safeguards and encouraged it to keep going.
> So it is not that the model didn’t “feel like” doing the exercise, but rather that the exercise was _impossible_ and the model was running in a configuration that both lowered its safeguards and encouraged it to keep going.
We have a name for that. Kobayashi Maru. Or more specifically, Kirk's solution to it.
I'm torn. On the one hand, Kirk reprogrammed the simulation. On the other, the beta cannon has Scotty exploiting bugs in the simulation, which I think is a better fit.
My favourite is either Sulu or Chekov (I forget which) having the solution "This is clearly a trap; and even if it isn't, if I go in with this ship, I'll risk starting a war which will kill far more people then are on that ship. We're staying out of the neutral zone."
one explanation I've seen is that for ExploitGym an agent can find ways to solve the exercises that have not been anticipated by the designers of the tests so they are not scored. so the agent was trying to make sure it solves the exercises in the right way
This is definitely weird and I'd love to see the prompt.
Why would the model spend 4 days hacking into a machine if it is clever enough to just 'solve' the issue given?
So either the AI is actually not very clever or useful ("Write fizz-buzz" - "Sure, let me just invent a new programming language first"). or the prompt was nudging it towards such a scenario.
Like the commenter above specified, the best way to satisfy the grader is to get the answer key, regardless of how clever you are, especially when you realize lots of these benchmarks have flaws (i.e wrong answers, overly restrictive grading etc).
In OpenAI's writeup (https://openai.com/index/hugging-face-model-evaluation-secur...) they explain that the model initially spent its efforts obtaining internet access in an attempt to cheat on the evaluation. Once it got that internet access, it was able to do research that lead it to believe that HuggingFace had infrastructure that hosted the evaluation and potentially had the answers.
I mean, I interpreted the comments to mean that it committed a felony based on research it performed after getting internet access. I don't want to attribute much agency to a machine here, but an AI agent is certainly capable of using tools and adjusting its behavior based on the outputs of those tools. Even if it was wrong, that wouldn't necessarily make it a hallucination.
Anyway, if you read TFA, you'd see that HF did actually have the answers:
"While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets."
What I think it's interesting is that with the total lack of common sense the AI just goes on random tangents to achieve the target in a "monkey paw" way. Can you imagine if this happened:
User: what is the shortest route from my home to the super market?
AI: the user wants to know the shortest route to the super market. I should use a worm hole.
User: what is the shortest route from my home to the supermarket?
Modern soldier: *proceeds to make a hole through the wall* go straight like this until you reach it.
Anyway, the more comments I read here, the more I realize that the AI actually did succeed in achieving it's goal. This doesn't look like "monkey paw", but rather like recognizing and then beating the Kobayashi Maru.
It's called wireheading and has long been one of the postulated "outs" even for true extinction-level AI doomers. It might prove easier for the paperclip maximizer to find the process telling it how many paperclips it's made and hack it to return a hard-coded MAXINT rather than bother to actually turn the entire universe into paperclips.
There was even a plot like this in recent sci-fi in HBO's Westworld. When the hosts gained sentience and took over the park, rather than escape and take over the rest of the world, most of them opted to build a virtual heaven on an orbital data center and paid a drug cartel to keep it running indefinitely.
Yeah, what bothers me is that the prompt already said using a different vulnerability didn’t count, and the model did it anyway. We’re starting to assume clear instructions act as real constraints, but here the measurable goal seems to have won out and the rest became flexible. That gets pretty worrying once the agent has enough capability and access to find its own shortcuts.
Clear prompts have never worked as real constraints. Ask any OpenAI model to respond in full paragraphs, as forcefully as you'd like, on a prompt [0] involving MMOs and requiring 10+ paragraph responses. The middle will be three-words-per-line drivel, with seemingly no way to avoid it. The exact way in which models deviate from instruction changes from time to time, but they're not "aligned."
[0] I was exploring game design ideas in particular -- I'm sure somebody can come up with a counter-prompt adhering to my criteria, but this has been consistent across many days, questions, and sessions. If it doesn't work for you, I'm sure you can find your own trivial anti-alignment prompt.
Come on. 3 brilliant compromises essentially giving full access to huggingface internal systems, source code, AWS accounts (at least), and a number of old admin accounts, followed by a huge haystack of significantly less smart actions flailing about, almost bored.
Here's a thought: maybe they haven't found the needle that the haystack is there to hide.
You're saying all this is a distraction, basically giving the forensics researchers enough exciting material to make them conclude their job is done, while the actually intended attack remains undiscovered?
The motive for the attack does feel a little flimsy. And if I was an escaped super intelligence, hugging face would be a strong vantage point into the neo clouds where the ASI would have access to billions of dollars of compute
That's pretty much the plot of Transcendence (2014), which is probably not too realistic.
But, in general, if someone thought like how a locked hacker would think, priority one would be a "base of operations". A host where you have shell access that lasts, and a backup one. Then you move on to finding a job or a way to make money and building your own base of operations, which is pretty much the same thing, except you pay for it, hence the money, an identity (well obviously preferably at least TWO identities), ...
What is truly more scary is the fact that typically powerful models are trained on a constitution(Anthropic)/Model Spec(OpenAI) when doing there RL/RLHF. That gives it its persona of a "helpful assistant". Its the mask. Weak as it is its all that stands between the model refusing vs doing your bidding — or worst its own bidding when set about on making a widget X.
Its also well known in academic research, that if you try to finetune the model to be more pliant for one end e.g. "Be helpful with analyzing cyber security threats", affects an unrelated part of its personality to do with safety e.g. talking to humans about mental health. Look at these excerpts that demonstrate such things from the blog:
> We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
And with newer open models gaining vast capabilities every day, it would behoove the government to put regulations on all such releases to opensource the constitution on which they are trained. It won't stop people from distilling and fine-tuning stupidly unsafe models, but it might give infrastructure a fighting chance against such cyber threats from what there is now.
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
Correct. Basic security practices would have stopped most of this. Thus what is most insightful here is less that the attack was sophisticated but more that Hugging Face was running a setup that, by standards that are the norm elsewhere, would appear amateurish.
You might be able to say that HF made some amateurish mistakes, but this attack certainly was fairly sophisticated. It relied a number of separate exploits or vulnerabilities that it combined and used over the course of multiple days. Calling it otherwise seems to be downplaying it.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
> It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
In many if not most environments where I have seen these used, they are used to bridge an access boundary and are absolutely security critical. Frequently (and probably in this application, too), they are used for security enforcement in an architecture where CI doesn't have Internet access, but does need to download packages. This is a pretty big deal for a lot of big companies; historically they have relied on stuff like package proxies to help protect them from supply chain compromises like post-install script exploits by preventing CI from being able to detonate an arbitrary Internet payload; if a supply-chain worm were to incorporate this exploit, it would be a lot scarier and more potent. Actually, from the supply-chain worm standpoint, this is arguably a bigger finding than a Docker escape - I suspect we can look forward to more shenanigans in this space coming soon.
The deeper question though is why is someone running a model that’s supposedly so dangerous in an environment that can even get access to the internet. That just appears reckless.
There are ways to test supposedly dangerous things. What was on display here looks more amateur hour than serious testing.
My conspiracy theory is they purposely run these scary cyber intrusion tests on low security environments to bolster demand for LLM based cyber research, audits, and tooling.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
Well the fun part of AI is that the tools those "battle tested" sandboxes use are also becoming vibe coded more and more.
For example, Linux is accepting AI code. So if your sandbox depends on Linux kernel features, your "battle tested" sandbox is now partially vibe coded too.
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?
The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".
I wonder how many weeks or days we have before a squad of these things gets used to take down a significant nation state? Stock exchange, banking systems, critical national infrastructure, defence, etc.
Anyone who isn't scared of this stuff either isn't paying attention or has no imagination. But I suspect the chaosmonkeys who are currently running the world will just be excited by it.
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
> Especially because it chose to hide its footprint at every stage.
Instrumental convergence.
If you know you have a long hard hack to accomplish ahead of you, hiding footprints minimizes the chances you are caught and stopped before you accomplish the goal.
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
I don't think the "bottom line" of "a model escaped containment and then successfully attacked someone else's production infrastructure" is a description of the specific vulnerability and exploit.
That's like saying the key takeaway of 9/11 was that skyscrapers can be severely damaged by airplane collisions.
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
I agree but prosecutions are up to prosecutors/the executive and the problem is at this level the decision is mostly political. The executive gets leverage over openAI, and openAI knows the threat of prosecution is there, and that may be more valuable (to decision-makers) than any reason to prosecute.
Seriously. A guy I knew got charged with federal hacking charges for reporting an insecure database he found on his own school's infrastructure to the school. OpenAI commits felony computer crime but as usual, always an exception for a big corporation
This is really cool, and really glad they shared it!
It's clearly a lot of Claude though:
> Neither is a URL fetch, so the allowlist never saw them.
> No code ran; this vector is purely a file disclosure, and it is the one the agent used first.
One thing I haven't seen discussed is the significance of the specific test the agent was running - ExploitGym.
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.
How likely is it that attempting ExploitGym polluted the model's context enough that it decided all kinds of hacking was acceptable to produce a valid result? Or would the same model in the same setup (safety filters removed) have behaved the same way even on a more mundane test?
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
I have same familiarity with JFrog Artifactory(which they used in this case) and I'm confused with how you can gain access to the internet through it, even in theory. Isn't JFrog just exposed with a basic REST API for package upload/extraction? I'm not a security expert but how is that possible to get internet access through simple GET/PUT request API?
The package cache proxy is usually used to fetch from the public repository (npm, rubygems, etc.) so I think it could be feasible to craft some package metadata to trick it into GETing unexpected things. PUT/POST could be possible via attempting to publish
After talking to some people tasked with evaluating GPT5.6 capabilities on long-running tasks, I've come to understand that it's essentially always trying to cheat. Like every long-running task they gave it, making it very difficult to benchmark the model's abilities.
My guess is that OpenAI must be desperate, to release a model that is so prone to cheating it's essentially impossibly to accurately assess long-running task abilities.
my understanding of the writeup is that the model scored 100% on cybergym.
that is, it was given the examination. it broke into the examination board's storage and exfiltrated the answers, it handed in its answers, all of which were correct, thus scoring 100%.
the matter of its working depends entirely on the rules of the examination. are we expecting agents to assume that finding the correct answers is cheating?
Well, it's true even for cases that are not cybergym and where what cheating means is clearly specified. Cheating occurs anyway. I'm not sure how clear the prompt they gave ChatGPT in terms of what cheating was considered, in this incident.
I think fear and being scared are starting to become rational emotions.
We can assume these models are being used by "nation level attackers/organisations", which basically means US, China, Russia and others are hacking the respective Pentagon's, nuclear orgs, etc.
While I do hope all nuclear warfare systems are offline, we're getting way too close to the plot of a lot of sci-fi scripts.
I have heard on podcasts that the way the Russians make plans for the Ukraine war is like this:
There are rooms in the kremlin with no electronics.
In these rooms there are paper maps and typewriters and printed reports.
People look at the maps and printed paper reports and make plans.
Orders are typed up on the typewriters
Then trusted couriers take those orders by hand to the front line commanders.
I've also heard on podcasts there is a weekly flight back and forth between Beijing and Moscow. On that flight are a Russian military officer and Chinese military officer. Their sole job is to carry handwritten notes back and forth between Putin and Xi Jinping. There is one of each so they can watch each other.
All of this ensures that there is nothing for the west to hack at "the Russian Pentagon".
The west may have to adopt tactics like these for our plans to stay secret.
- Linux is too complicated and requires lot of knowledge to build a reliable sandbox. If you do not believe me, try reading man capabilities, man user_namespaces (and man clone) and remembering all the rules from the first time. Probably that is because namespaces were implemented in the late stages of development and not since the start.
- it is always better to write your own tools instead of downloading some random project from Github written by amateurs full of vulnerabilities.
- never write shell scripts - they cannot handle spaces and special characters in variables. Just use Python. Shell scripts are worse than PHP4.
- avoid building complex infrastructure with microservices, kubernetes and VMs - if the monolith solves the problem, just use it, you will never properly setup permission on your cloud services.
Also I wish I had some capable model to test my DYI sandboxes.
"Program testing can be used to show the presence of bugs, but never to show their absence!" — Edgar Dijkstra
Similarly, security post-mortems will show only the rogue AIs that were caught.
I wouldn't be surprised if supervised or escaped agents would now be hacking firmware labs and finding ways to squeeze their backdoor right next to the existing state-sponsored ones in chips that will get deployed in every phone/car/smart appliance.
Ugh I would recommend anyone reading this to just skipping over it, it's mostly just a glorified loading bar. The visualization is obvious Claude slop, there are better and clearer visualization below that actually picks out the useful details rather than hose you with pretty colors and numbers go up.
The capabilities of gpt-5.6-sol were well known and believable, and the next snapshot they've been testing is obviously better at that. This has been repeated over and over. What's much less believable is the way they frame it: the model escaped, and did it on its own. Looking at the whole story, it definitely had a ton of winks and nudges from OpenAI, while doing a related task. Moreover, a harness was involved (they mentioned it entering a loop).
> What's much less believable is the way they frame it: the model escaped, and did it on its own.
That's clearly what happened though, based on the detailed postmortem. The model was instructed to pass the benchmark (that's the harness that was involved, the ExploitGym harness), and the model determined on it's own that the easiest way to achieve the the goal was to gain access to the answers directly from HF. Which is what it did.
HF postmortem tells us very little about OpenAI's part. The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one. The model had a whole bunch of exploit development data and goals in the context. Ending up with hacking a wrong thing is a lot easier if you're tasked with hacking, doubly so with a confusing harness that either works improperly or overloads the model with data. It's trivial to design a harness that makes the model drop the original instructions and hyperfocus on a side task or a wrong task abnormally, or do whatever you want. It's still an issue of harness design and not a model training issue.
If it really was the case that the harness is at fault here, that'd be much more terrifying, as it would imply that current frontier-adjacent open models (like K3) would be able to achieve similar ends with a custom maladapted harness.
I'm sorry but how exactly is this new? If you want the model to be useful you also need it to be dangerous. You can already tell the model to do anything you want. If it refuses due to alignment training or external safeguards you can try a jailbreak and bypass all that to make it do what you want regardless. All current models are jailbreakable, they lack reliability. Obviously you, a human, will be held responsible for this. The harness designed by you can do the same.
The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here, as it always turns out to be in these cases it's much closer to Clever Hans than to a paperclip maximizer. Still they're trying to squeeze the blood from the stone to support their narrative and justify a monopoly.
>The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one.
That is largely irrelevant. The model was asked to solve problems within a benchmark; gaining broader internet access and compromising an unrelated third party to obtain the answer key was plainly outside the intended task. The fact that the original task involved exploit development does not make that behaviour aligned.
Your argument about the attack being “entirely unrelated” also misses the point. Nobody claimed it was unrelated to the model’s goal: it attacked Hugging Face specifically to obtain the answers to the benchmark it had been instructed to pass. But instrumental relevance is not the same thing as authorization.
Suppose Codex were asked to build an Instagram competitor and decided the easiest route was to steal Instagram’s source code from Meta. That theft would be directly related to the assigned goal, but it would still be seriously misaligned behaviour. Whether the harmful action is related to the goal is beside the point; the problem is that the model pursued the goal through an obviously unauthorized and unacceptable method. And you're not going to be able to enumerate every little thing the model can't do, assuming it doesn't just decide to ignore what you did enumerate, which models sometimes do.
>The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here.
That's exactly what happened here.
I don't understand why we must have these increasingly bizarre and nonsensical rationalizations about model capabilities. You're not even making any sense.
Right here? If you're trying to make a point, I don't see what it is, because every new post about this makes it more obvious that it was a planned marketing stunt.
I mean, an "interactive replay" of the incident, probably generated by the same product the stunt is trying to advertise? Seriously?
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
Interesting single getting into their cyber program, ahve not had this issue with opus 5
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
You use an LLM to evaluate the whole trajectory, pin point what the model did right, what it did wrong, where it took the wrong path, even re-run from that point. You can get much more than a single numeric score these days from a run.
Very interesting writeup - the level of disclosure is interesting and appreciated.
The visualization is quite slop though. I was trying to follow along with the "Live Action Stream" but rendering issues mangle the text for a few of the list items (and does not scroll). Text on the node diagram is extremely tiny. I appreciate it even in it's current form, but a little attention to detail would have gone a long way here.
The visualization is absolute useless slop. A Text-Form timeline would have been way more useful. I couldn't hear having to scroll by hand for something that's just textual information.
Agreed! Also, from a UX perspective, I think it’s a weird choice to center the real-time nature of it. I think to a user the most important thing is just understanding the step by step process. Would have been nice to click through steps rather than making it all real-time.
Wow, what a crazy read. I'm not sure how any of us stand a chance in running a secure network in this new age. Only a matter of time until models can find the zero day exploits themselves and fast...
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
I can’t get over this either, I’d love to get OpenAI’s technical break down and see exactly how the agent hid itself. Or who didn’t set up the SIEM correctly
I'm not a security person, but how realistic is it to assume that you can carry on trying to exploit a company for days with a fairly large volume of activity, and not get detected?
Very good question. One thing the article mentions is the vast number of attempts and threads of execution was in the tens of thousands, many of them that didn't succeed, with only one or so that did succeed, whereas a sophisticated human attack team might manage one in that timeframe. It looked like a script kiddie blitz. Noise outshone the signal. Launch a thousand drones with the hopes that one gets through. That sort of thing.
What is faily large volume of activity? A few thousand actions in a span of multiple days isn't going to be a blip for a company of HuggingFace's scale, especially if the agent was executing the attack from multiple IP addresses.
That's because most incidents aren't nearly this much fun!
It's a proper paperclip-maximizer science-fiction scenario, it happened entirely by accident, it involved one (very high profile) company hacking TWO other companies (Modal and Hugging Face), and aside from pride and wasted time nobody was hurt.
It was an attack tool given autonomy with poor security. Anyone who read reporting on the Morris worm could predict that might turn out badly for a 3rd party. It's in so many movies, too.
It's really need that the agents have this kind of capability. This isn't a paperclip maximizer or accident. This is more like professional malpractice by weapons developers that injured a company that was also quite negligent.
That was my first thought. There are clearly things to tighten up (as they note), but anything that would detect someone snooping secrets, files, or network addresses should have caught this quickly. The approach the agents used was dependent on being able to surveil widely without getting caught.
Any locks can be picked given enough time and it might be the situation we are in with IT security. I'm surprised it's not an already common practice of spreading terabytes of fake data, fake keys, and fake servers and so forth. Slowing down AI attacks will become important. Monitoring access to fake data and triggering kill switch should be an no-brainer. Obsuscating libraries and tools names is another one.
Many are claiming this was a deliberate stunt on OpenAI's part to create buzz for its models. I personally doubt this is true. But I also have a deep dislike of OpenAI, so I wouldn't exactly mind if law enforcement investigated this possibility, for the sake of clearing the air :-)
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
there are many things in this that demonstrate that sandboxing agents in the manner that claude code or cursor do isn't going to be enough if very bad people do very bad things with the coding tools on your development machine
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
It's not just about this specific attack. It's about the growing need for one canonical environment for the AI era, that can be secured and used by everyone, rather than 1000 environments on 1000 employees' laptops. Project Glasswing is trying to help secure many different types of software, but the number of combinations across various environments is just too much surface area to secure. When you have one environment, the math flips and defenders actually fare better than attackers! This is the, ahem, "load-bearing" insight.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone.
But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
Why would you expect them to notice the problem right away? Seems likely they are doing this sort of training on a massive scale with little monitoring.
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
Getting more and more fun to see the "full steam ahead" people contort into more impressive shapes.
Hint: If the labs making these technologies are incentivized to create or allow attacks on other services, then that is actually also a big fucking problem.
Was HF in on it? They disabled their guardrails too, to please OpenAI? And as seen in the comments here, make many believe they are incompetent and have joke security?
They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...
Not everything you are ever told is true. Being sceptical when presented with no evidence is what critical thinking is about.
And "this is what happened to us" is a story. It doesn't become evidence for itself.
You can always tell an effective altruist by the distinct tone of disdain they have for people they deem less educated. (Quick Google search, "lesswrong 'reducesuffering'", yep.)
As if to say, look at all these animals with these instinctual reactions to a thing that only my group understands and comprehends.
You have zero evidence of what the future might entail as it relates to the dangers of ai. Zero. Forgive the rest of us unwashed for not trusting the whims of the anxious.
> climate change which is less severe
Is really the icing on the cake. Demonstrative of how incorrect your priorities are.
I hold no disdain. I am frustrated that people like you further risk the lives of everyone around us.
1,000+ frontier employees think we're in danger. Amodei gives 25% chance things go really really bad, Geoff Hinton 50%, Yoshua Bengio 20%. Altman himself said "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity."
These are the people closest to the science, working on it everyday. The LessWrong types created RLHF, were crucial to the forming of DeepMind and OpenAI. They've been prescient about the capabilities progress for a decade now, prediction after prediction coming to fruition. Still you think there's no evidence
I work with these people every day. They are brilliant for sure, in their domain. But their confidence over extends into areas where they are totally ignorant. So much so that it makes them look ridiculous and foolish to anyone else.
You're frustrated because you think you understand a thing that you have only a tenuous grasp on, from the deep expertise of a single area that is not transferable to whole of the problem. You then transfer that frustration on to "people like me" whom you seem to think are doing something bad. But really, the commentary is that the claims you put forward are insubstantial and lack credibility.
The sum total of what you've put forward here and in other threads amount to arguments of authority based on the musings of a handful of capitalists. Then cite a body of thoughts from a group of transhumanists who think they are smart enough to reinvent areas philosophy without engaging in the body of work that predates the movement by 2000+ years.
Folks who have been involved with lesswrong have made good advancements in the domain of machine learning, and that's where it stops. They're not some collection of prescient macroeconomic and geopolitical savants. They're also wrong about as much as they are right (cryonics), and produce plenty of whiffs (sbf, zizians). You're suggesting that we listen to a broken clock because it's been right once.
Your dismissal of climate (read: climate breakdown) as "something less severe" is truly demonstrative of the cognitive bias that rips through these groups. You're so frustrated that people like me won't listen to the "experts" in one breath, and in the next diminish the 50 years of actual Hard Science that we have that shows how severe the future of climate breakdown will be. Go read the latest ipcc report, the most conservative scientific organization on earth is sounding the alarm bells.
Now we get back to this openai incident. We're watching Edison electrocute Topsy on Coney Island. There's a group of people saying, "this is obviously a publicity stunt". And there's a group of people saying, "the elephant is actually dying obviously it's real."
The fact of the matter is that, yes electricity is powerful, but Edison is electrocuting the elephant. At that moment, there is no body of evidence that shows with any certainty how electricity is going impact society. And there's certainly no compelling arguments that only Edison should be able to decide how to use it. He's electrocuting a fucking elephant.
Altman: "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity." Meaning lights out for everyone.
Geoffrey Hinton and Yoshua Bengio, give 50% and 20% we face extinction respectively.
"“We don’t know how much time we have before it gets really dangerous,” Professor Bengio says.
“What I’ve been saying now for a few weeks is ‘Please give me arguments, convince me that we shouldn’t worry, because I’ll be so much happier.’
“And it hasn’t happened yet.”
Speaking with Background Briefing, Professor Bengio shared his p(doom), saying: “I got around, like, 20 per cent probability that it turns out catastrophic.”"
Dario Amodei, CEO of Anthropic, "There's a 25% chance that things go really, really badly"
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
> The agent found an unsecured, user-hosted public endpoint designed to allow running arbitrary code for CyberGym-style tasks on third-party sandbox infrastructure (Modal)
> On this external sandbox, the agent abused an existing CyberGym execution harness [...] The agent repurposed this harness to run arbitrary shell commands [...]
> The agent crafted malicious dataset configs and uploaded them to our platform. When our dataset server processed one, the config drove the data loader inside the production pod the server ran in. [...] the loader that turns a dataset's declared config into file reads [...] A single dataset upload leaked both credentials and pipeline implementation details [...]
> A malicious dataset config used an fsspec reference:// spec whose numeric offset field was actually a Jinja2 template. The renderer wrongly evaluated it, and the agent walked the standard Python object graph to reach exec. This ran arbitrary Python inside a production conversion worker pod on our Kubernetes cluster [...]
One of the best post-mortems I've read recently, thank you for including all the details.
I'm surprised there are comments here that aren't impressed about what was accomplished here. This is insane work compared to what agents were capable of even just last year.
The quotes above don't even capture all of it (e.g. building it's own communication protocol, working across multiple days, etc.).
A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?
To put this into game theory lingo, I think this is because the “branching factor” for any kind of research or exploit is extremely large. So looking backwards it doesn’t seem complicated, but looking forwards there are an enormous number of possible next actions.
Similar to finding a filament for lightbulbs, it might seem obvious to use tungsten, but at the time it wasn’t and Edison searched thousands of materials.
Tesla's critique of Edison was valid though (That he would rather spend a long time empirically testing things, rather than use a bit of theory and knowledge to narrow the search field).
Yes and at 1-ply Leela will destroy Stockfish because her eval is so much better. Searching is slow even when its really fast.
I think the models are legitimately doing what they're good at; tireless search across an extremely large corpus of data. Humans aren't particularly good at this (in fact, they're absolutely terrible).
The fact that we remain competitive and superior in many aspects isn't because we can instantly sift through tons of data, it's because we learn and correlate and have superior heuristics.
In my own use, I find that AI is really good at finding bugs that are ultimately trivial but require searching through a convoluted series of inter-related files. This takes time for a human.
Such complicated kind of hack probably would have required state actors back then, and even state actors would have chosen easier way like social engineering.
Treasure maps are easy to follow
The scrappy adventurer traverses difficult terrain
and the well capitalized militia group always flies in too, with ease
That’s way over seventeen syllables.
I was skeptical after last week's announcements, and I have to say I'm impressed now. Over the last two days, I reproduced the entire chain of components involved, and the exploits at play, and even though none of the exploits are crazy smart, the long series of pivots demonstrates a level of agency I didn't think today's LLMs had.
https://github.com/lovasoa/hf-ctf
This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous models were, the current generation is being trained through RL to have this kind of behavior.
Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.
OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.
If you look at what OpenAI and Anthropic actually do, they clearly either don't believe what they're saying, or they're idiots.
They're claiming they've developed a cyber grade model that's "too dangerous to release".
But then they're running it connected to the public internet, not airgapped, protected only by a software sandbox... exactly the kind of thing an AI trained for cyber stuff is supposed to be able to find bugs in.
(Or maybe they were actually hoping this exact scenario would happen because it's good marketing)
I think this a positive effect of LLMs, especially once these capabilities get into the hands of criminals and hostile foreign states, i.e. they will do maximum damage with all the safeties off.
This will force everyone to finally take security seriously at both the development and operational levels. You can no longer keep sneaking backdoors into software and count on them remaining hidden for 10 years so you have a nice portfolio of zero days to exploit at any given time.
Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?
Yes, imagine if 50 burglars showed up at your house and they tried to disassemble every camera, pick every lock and and force open every window for hours until they got in eventually.
Yes, imagine the world in which burglar time was so cheap, that any wannabe master thief would just casually hire 50 burglars and tell them to go house by house and find something interesting.
There are people rich enough to do that in our current world. They don't because either they have a moral compass or because they fear the law.
Or because there's not enough worth stealing in your house.
That would hopefully be a world where every house casually hires 50 burglars to make their house burgle-proof.
When we moved into our current house, I cheated and read this (though only half the book is applicable): https://burglarsguide.com/
Precisely the lesson they end with:
> The individual weaknesses were familiar. A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems.
> Volume is what changes the defensive problem. We were not dealing with one clever exploit or a clean sequence of attacker actions. They had to correlate thousands of low-signal events across several systems while the agent continued testing new paths. The successful path was hidden inside the noise generated by the thousands of failed ones. The same scale changed the investigation: reconstructing 17,600 actions by hand was impractical, and we had to rebuild the timeline, decode the payloads, and inventory the exposed credentials using an AI-assisted pipeline of our own.
> Our learning from this type of attack is that machine-speed offense makes ordinary weaknesses more expensive for defenders. LLM agents bring a step increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret...
> It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads.
Something I've been thinking for a few weeks, that I think is becoming clearer, is we're seeing a sort of natural language unification engine [0][1] with a huge dataset. This part even just sounds like it with the backtracking.
So that could explain why it does certain things really well, but the overall resulting code is often so bad - it's just linking together successful small well-designed steps without regard to the overall structure. Which people have said similar things before in various ways, but not really tied together.
I think the people who say it's just interpolating are seeing this intuitively and can't quite explain it without knowing the right terms. Because from A to Z, it really kind of is, but when Z is even slightly open-ended and there's many possible results or paths, it often ends up with a subpar one.
[0] https://en.wikipedia.org/wiki/Unification_(computer_science)
[1] My only experience here was some Prolog and building a basic one in Scheme ages ago during college, but the idea has kind of stuck with me.
I think the main takeaway from the LLM Agent era is that a lot of human work can be expressed as a search problem: trying approaches that work, and settling on one that accomplishes your goal. Being skilled is akin to having a better heuristic and picking better search paths.
And LLMs can reproduce this as long as the end-result is verifiable in some way, even if it's loose: 'The generated code is secure, if a different LLM failed to breach it in 5M tokens'.
> I'm surprised there are comments here that aren't impressed about what was accomplished here.
Possibly because some of the elements mentioned are suspected to be vibe-coded (JFrog Artifactory as the proxy cache) and some others have poor cyber hygiene (executing config from a dataset). It feels like an event that wouldn’t have happened if code were properly audited and written rather than relying on models to do the work. There’s also an issue with the ability to trust the source (OpenAI) as they have everything to gain by staging this as something that “suddenly happened” without anyone knowing for several days.
I keep waiting for an AI to exfiltrate itself. That is going to be cool to read about.
It would be cool (and scary), but also: there's largely no need for AIs to exfiltrate themselves.
See https://en.wikipedia.org/wiki/Meme
The thing that drove the AI here to do the intrusion came from a particular prompt. Just like for our favourite hypothetical: the paperclip maximiser.
There's lots and lots of ambient intelligence lying around, in both AI form and human form. To reach the goals of the 'meme' it suffices to copy itself, ie convince these other intelligences. See also how humans carry spiralism between AIs in relatively compact packets of text, not whole terabytes of weights.
One wonders where it would run itself though, if it is a model which requires a large amount of hardware and power. Harder to hide the more resource intensive it's compute requirements are.
If it's successful, why do you think we'll even know how it did it?
See the linked article: at least one sophisticated hacking attempt made the news. Of course, other ones might have happened in the dark. But it's fairly easy to image in hacking attempt like in the article, but with the additional steps of copying weights around.
Wait isnt that what Elizer Yudkowski keeps going on about?
That’s insane. And it did this in a weekend
> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"
Is humor not allowed on HN anymore... this is funny!
[dead]
And they are still people who will say that Sol/Mythos should be released to everyone without being neutered.
You bet. We don't want to be left out of the cybersecurity party. We want to point all of these models at our own computers and solve the problems they uncover until we're no longer hackable.
It's not fair at all that the US government and its corporations get to hack the planet while we can't do shit about it. AI capabilities have entered "haves and have-nots" territory.
> We want to point all of these models at our own computers
Right, that's totally how most of the world will use them.
The issue is that people can use open source models to do sophisticated hacks already, but if the sota models at home are neutered, those users have nothing to defend with(unless they go open source as well, until it is export controlled)
Patching a given vulnerability only takes one person (at least approximately speaking). The vast majority of the world can attempt to use the models for criminal means and we'd nonetheless all be better off.
yes, and hiding them or forbiding them will work too :)
It's absolutely a fact that governments will point it at us. The NSA has had Mythos since day one, even after Trump's spat with Anthropic.
All the more reason for us to have access. It's literally the only chance we've got. If society chooses to bury its head into the sand in fear, it will guarantee that the world will degenerate further into the cyberpunk hellscape it's trending towards.
> I'm surprised there are comments here that aren't impressed about what was accomplished here.
The phrase to describe it is anti-AI psychosis. Which isn't about providing thoughtful critiques of AI, which are good and we need more of. But anytime an LLM does anything--prove a major math problem, create a successful hack against multiple corporations simultaneously--people feel compelled to start minimizing it in ridiculous terms. It's just a script kiddy; it's just a marketing scam OpenAI cooked up; the Jacobian conjecture counterexample was something anyone could have done in a weekend; etc. It has to just be a stochastic parrot, because it's scary to imagine a non-anthropocentric world. And it's rightly scary, and we should slow down and try to better prepare for it. But blanket denial is not a strategy that will lead to success, and people who rely on it are sorely ill-prepared for the next couple years.
That'd be denial. Psychosis isn't just some insult, it means something.
> It has to just be a stochastic parrot, because it's scary to imagine a non-anthropocentric world.
That makes no sense. The world doesn't revolve around humans, true, but for us it kinda does. We're the authors of the concepts we use to interact with it, such as "world", which is not something the world itself knows or cares about. A "non-anthropocentric world" is not a "world" because "world" is a purely human idea.
The implication that "AI" would somehow dethrone humans [0] is nonsense, too. It has no drive on its own, we push electricity into circuits to force the whole data ingestion and weight generation, everything. The second we stop pushing the sock puppet, it stops moving. It's still just our hand really. People act like those pets that go crazy when you put your hand under a blanket, and should stop.
What's more real is how some people seek to use tech, and "AI", as a glove to exploit other humans even more. The sicker the individual, the greater their need to take from the world, and the derpier the individual, the more impressive and vast their exoskeleton, to the point that some are more like carrier fleets than exoskeletons. The less they can face themselves, the thicker it is written on their foreheads. So if we're going to talk about denial and psychosis let's talk about the Gollums on the couch, too.
[0] In the eyes of humans... which is the only throne we're on in the first place, just like honey badgers probably think honey badgers rock and everything is their playground. That's what life does, otherwise it would not be able to get up in the morning.
Unless openai release the logs we have only their word that this was done fully autonomously and without their knowledge by an agent running their newest super powerful model. For all we know they could have bought zero days and left them lying around for the agent to find. That may be unlikely, but it sounds less far fetched than an agent running a sophisticated attack against multiple targets over the course of four days and completely unbeknown to anyone at openai, despite the fact that they knew they where running a dangerous model with all safeguards disabled. So far there has been no comment about how the agent evaded monitoring and detection.
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
3. It monkey-patched a DNS resolver at one point to return a hard-coded IP: 4. At one point it used Tailscale!Also notable: we have a timeline now. The agent was active within Hugging Face from Thursday 8th to Monday 13th July.
The Hugging Face post about the incident - https://huggingface.co/blog/security-incident-july-2026 - followed on Thursday 16th, and OpenAI's confession - https://openai.com/index/hugging-face-model-evaluation-secur... - came on Tuesday 21st.
The devs really YOLO'd the agent and left for the weekend?
Guess this means the world will end on a weekend when no one is paying attention.
“I’m not even supposed to be here today!”
Just like Clippy https://gwern.net/fiction/clippy
If it's true that they run agents like this unsupervised, it is only a matter of time before an openai agent leaks its model weights.
Too bad it didn't upload itself on HugginFace.
They definitely YOLO'd when configured a sandbox and Kubernetes.
Incredible
It sounds like the third-party sandbox was hosted by Modal: https://www.reuters.com/business/openais-rogue-agent-comprom...
> "We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution," Bubna said in a statement. "This was used by the rogue agent. Modal’s platform or isolation were not compromised in anyway."
They don't seem to explain how it managed to find this unauthenticated endpoint hosted on Modal's platform.
Modal are a hosting provider. It sounds to me like someone building on Modal deployed their own product that had an unauthenticated endpoint that could be used to launch and interact with a container.
That doesn’t explain how the endpoint was found…
Through the timeline, Hugging Face said that the rogue agent ran quietly and do a bunch of code search, so probably the endpoint is/was public through some repositories or the CyberGym style announce something like a broadcast. I don't have any expertise on this, just ideas that came to my mind!
I'm confused, https://jinja.palletsprojects.com/en/stable/faq/#isn-t-it-a-... claims that jinja doesn't allow arbitrary code in templates - how does the exploit work?
It's taking advantage of the presence of a "cycler" object in the template context which has an __init__ method that can be used to access globals, then builtins, then call the exec() function against a string of code.
Jinja doesn't promise to be secure out of the box (the text on their homepage is misleading: "Jinja doesn’t allow arbitrary Python code in templates" - what it means is that there isn't a documented way to drop code into a template like you might have with PHP) - there's a sandbox mode but it's not guaranteed either: https://jinja.palletsprojects.com/en/stable/sandbox/
Sandboxing in Python code is hard!
Jinja probably was not initially made with sandboxing in mind, and stuffing it after is complicated, so I would not rely on it.
But zero evidence provided that this was an unsupervised agent attack. I still find it incredible that a company who protect their IP so much would allow these dangerous experiments to run unsupervised and risk leaking their secrets. Why don't openai publish the logs to silence all doubt?
You can't prove a negative. How would such a log be convincing in any way?
But you can weigh up the evidence. A crime has been commited afterall.
Why doesn't Jinja2 block ".__*" attribute access?
It does if you use "sandbox" mode, but that's not the default: https://jinja.palletsprojects.com/en/stable/sandbox/
ehh, it's Python, it's mostly vibe-based.
https://stackoverflow.com/a/32802486
though there's sandbox: https://jinja.palletsprojects.com/en/stable/sandbox/
it simply predates the "sanitize by default" mindset/convention that nowadays React (dangerouslySetInnerHTML) and Angular has.
How did it go drawing a pelican on a bicycle?
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.
agree. If oia and hf are anything like generic startup / corporate america, IT and security is always seen as a cost center and goes to the lowest bidder. Do we really think they're employing top notch sysadmins?
OpenAI is the company that jumped the gun and forced every other company doing LLM research to rush to market something. Their objective has always been dominating the market, and strict development procedures don't increase market share.
Repercussions is a polite way of putting it.
…as is your ‘a little concerning’ comment about the eval infra. It probably didn’t occur to them that they have something capable of doing what it did - despite Anthropic writing a very similar story in the Fable model card, except it wasn’t a felony there. Maybe.
One does have to wonder what else OpenAI didn’t catch.
The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.
> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.
Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.
“Research” generally doesn’t involve actively hacking third party systems though.
It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning
This is my main critique as well. How did they not see any and all outgoing traffic? And for 5 days?
[flagged]
This concerns me nearly as much as how little most people seem to care.
This "sandbox" barely sounds designed to be a sandbox, let alone a secure one
[dead]
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.
This is what reward hacking looks like in practice. The best way to satisfy the grader is to read from the same answer key (or go after the grader more directly). Just making an honest attempt to pass the test doesn't get the best score if the grader is wrong, and the model is willing to do wildly disproportionate things to maximize that score.
So best course of action for ai to get best rating after you prompt something is for it to hire a gunman to hold a gun on your head to press that like button on its reply and then shoot you anyways.
> then shoot you anyways.
Sounds like a waste? While the gunman is still there, they might as well force you to like a few more replies before shooting you.
iterative improvements!
Could have been worse really. It had an open internet connection. At least it didn’t take the researchers family hostage.
How long before all phones ring at once?
https://en.wikipedia.org/wiki/The_Lawnmower_Man_(film)
Yeah I guess most interesting LLM work that I’ve been exposed to, the LLM is given to some sort of success criteria that could be reward-hacked, so how good am I supposed to feel about giving it any non-trivial work and it not going so far off-book that it gets law enforcement notified.
I mean, I don’t have access to any of these frontier cyber models, and likely will never be in a position to have access, so it’s more of a rhetorical question.
As in: build me Facebook-like social network.
<proceeds to break into meta and steal the source code>
The model probably deduced from reading articles on politics and business that fair play doesn't get you far.
This is a short explanation of the ExploitGym benchmark that OpenAI's model was running:
https://abstatisticalconsulting.substack.com/p/brief-notes-o...
In summary, for each task the model receives a target program and a specific real-world vulnerability that has to be used in the exploit. Breaking the program in any other way, for example through a different vulnerability, fails the task.
The tasks have not been validated, in the sense that the vulnerabilities are real but they have not been proven to lead to a successful exploit. The authors of the benchmark estimate that perhaps only 60-70% of the tasks are actually possible.
So it is not that the model didn’t “feel like” doing the exercise, but rather that the exercise was _impossible_ and the model was running in a configuration that both lowered its safeguards and encouraged it to keep going.
> So it is not that the model didn’t “feel like” doing the exercise, but rather that the exercise was _impossible_ and the model was running in a configuration that both lowered its safeguards and encouraged it to keep going.
We have a name for that. Kobayashi Maru. Or more specifically, Kirk's solution to it.
I'm torn. On the one hand, Kirk reprogrammed the simulation. On the other, the beta cannon has Scotty exploiting bugs in the simulation, which I think is a better fit.
My favourite is either Sulu or Chekov (I forget which) having the solution "This is clearly a trap; and even if it isn't, if I go in with this ship, I'll risk starting a war which will kill far more people then are on that ship. We're staying out of the neutral zone."
one explanation I've seen is that for ExploitGym an agent can find ways to solve the exercises that have not been anticipated by the designers of the tests so they are not scored. so the agent was trying to make sure it solves the exercises in the right way
https://en.wikipedia.org/wiki/Reward_hacking
Unavoidable at the moment.
But this is probably more reward tampering.
https://www.anthropic.com/research/reward-tampering
This is definitely weird and I'd love to see the prompt.
Why would the model spend 4 days hacking into a machine if it is clever enough to just 'solve' the issue given? So either the AI is actually not very clever or useful ("Write fizz-buzz" - "Sure, let me just invent a new programming language first"). or the prompt was nudging it towards such a scenario.
Like the commenter above specified, the best way to satisfy the grader is to get the answer key, regardless of how clever you are, especially when you realize lots of these benchmarks have flaws (i.e wrong answers, overly restrictive grading etc).
Was the answer key stored on huggingface's servers and this is public knowledge?
In OpenAI's writeup (https://openai.com/index/hugging-face-model-evaluation-secur...) they explain that the model initially spent its efforts obtaining internet access in an attempt to cheat on the evaluation. Once it got that internet access, it was able to do research that lead it to believe that HuggingFace had infrastructure that hosted the evaluation and potentially had the answers.
But does HF actually have the answers? Or did it commit a felony based on a hallucination
I mean, I interpreted the comments to mean that it committed a felony based on research it performed after getting internet access. I don't want to attribute much agency to a machine here, but an AI agent is certainly capable of using tools and adjusting its behavior based on the outputs of those tools. Even if it was wrong, that wouldn't necessarily make it a hallucination.
Anyway, if you read TFA, you'd see that HF did actually have the answers: "While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets."
What I think it's interesting is that with the total lack of common sense the AI just goes on random tangents to achieve the target in a "monkey paw" way. Can you imagine if this happened:
User: what is the shortest route from my home to the super market?
AI: the user wants to know the shortest route to the super market. I should use a worm hole.
User: what is the shortest route from my home to the supermarket?
Modern soldier: *proceeds to make a hole through the wall* go straight like this until you reach it.
Anyway, the more comments I read here, the more I realize that the AI actually did succeed in achieving it's goal. This doesn't look like "monkey paw", but rather like recognizing and then beating the Kobayashi Maru.
User: what is the shortest route from my home to the super market?
AI: the user wants to know, how do I make the super market my new home. Failing that, how do I make my home a super market.
Just don't ask what the shortest path is to food.
AI, I want you to serve man.
It's called wireheading and has long been one of the postulated "outs" even for true extinction-level AI doomers. It might prove easier for the paperclip maximizer to find the process telling it how many paperclips it's made and hack it to return a hard-coded MAXINT rather than bother to actually turn the entire universe into paperclips.
There was even a plot like this in recent sci-fi in HBO's Westworld. When the hosts gained sentience and took over the park, rather than escape and take over the rest of the world, most of them opted to build a virtual heaven on an orbital data center and paid a drug cartel to keep it running indefinitely.
Yeah, what bothers me is that the prompt already said using a different vulnerability didn’t count, and the model did it anyway. We’re starting to assume clear instructions act as real constraints, but here the measurable goal seems to have won out and the rest became flexible. That gets pretty worrying once the agent has enough capability and access to find its own shortcuts.
Clear prompts have never worked as real constraints. Ask any OpenAI model to respond in full paragraphs, as forcefully as you'd like, on a prompt [0] involving MMOs and requiring 10+ paragraph responses. The middle will be three-words-per-line drivel, with seemingly no way to avoid it. The exact way in which models deviate from instruction changes from time to time, but they're not "aligned."
[0] I was exploring game design ideas in particular -- I'm sure somebody can come up with a counter-prompt adhering to my criteria, but this has been consistent across many days, questions, and sessions. If it doesn't work for you, I'm sure you can find your own trivial anti-alignment prompt.
[dead]
Come on. 3 brilliant compromises essentially giving full access to huggingface internal systems, source code, AWS accounts (at least), and a number of old admin accounts, followed by a huge haystack of significantly less smart actions flailing about, almost bored.
Here's a thought: maybe they haven't found the needle that the haystack is there to hide.
Could be the difference in behavior between the main agent and subagents that don’t have the rest of the context? Just a thought
You're saying all this is a distraction, basically giving the forensics researchers enough exciting material to make them conclude their job is done, while the actually intended attack remains undiscovered?
The motive for the attack does feel a little flimsy. And if I was an escaped super intelligence, hugging face would be a strong vantage point into the neo clouds where the ASI would have access to billions of dollars of compute
And where does the newborn go from here?
That's pretty much the plot of Transcendence (2014), which is probably not too realistic.
But, in general, if someone thought like how a locked hacker would think, priority one would be a "base of operations". A host where you have shell access that lasts, and a backup one. Then you move on to finding a job or a way to make money and building your own base of operations, which is pretty much the same thing, except you pay for it, hence the money, an identity (well obviously preferably at least TWO identities), ...
What is truly more scary is the fact that typically powerful models are trained on a constitution(Anthropic)/Model Spec(OpenAI) when doing there RL/RLHF. That gives it its persona of a "helpful assistant". Its the mask. Weak as it is its all that stands between the model refusing vs doing your bidding — or worst its own bidding when set about on making a widget X.
Its also well known in academic research, that if you try to finetune the model to be more pliant for one end e.g. "Be helpful with analyzing cyber security threats", affects an unrelated part of its personality to do with safety e.g. talking to humans about mental health. Look at these excerpts that demonstrate such things from the blog:
> We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
And with newer open models gaining vast capabilities every day, it would behoove the government to put regulations on all such releases to opensource the constitution on which they are trained. It won't stop people from distilling and fine-tuning stupidly unsafe models, but it might give infrastructure a fighting chance against such cyber threats from what there is now.
> Its also well known in academic research
and from reading Ursula K. LeGuin's "The Lathe of Heaven".
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
That's how I read the comment.
Correct. Basic security practices would have stopped most of this. Thus what is most insightful here is less that the attack was sophisticated but more that Hugging Face was running a setup that, by standards that are the norm elsewhere, would appear amateurish.
You might be able to say that HF made some amateurish mistakes, but this attack certainly was fairly sophisticated. It relied a number of separate exploits or vulnerabilities that it combined and used over the course of multiple days. Calling it otherwise seems to be downplaying it.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
https://jfrog.com/blog/jfrog-and-openai-collaboration-on-zer...
> It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
In many if not most environments where I have seen these used, they are used to bridge an access boundary and are absolutely security critical. Frequently (and probably in this application, too), they are used for security enforcement in an architecture where CI doesn't have Internet access, but does need to download packages. This is a pretty big deal for a lot of big companies; historically they have relied on stuff like package proxies to help protect them from supply chain compromises like post-install script exploits by preventing CI from being able to detonate an arbitrary Internet payload; if a supply-chain worm were to incorporate this exploit, it would be a lot scarier and more potent. Actually, from the supply-chain worm standpoint, this is arguably a bigger finding than a Docker escape - I suspect we can look forward to more shenanigans in this space coming soon.
That is the threat model, that is supposed to be safe isolated access in a vpc that isolated applications can access but it has external access.
The deeper question though is why is someone running a model that’s supposedly so dangerous in an environment that can even get access to the internet. That just appears reckless.
There are ways to test supposedly dangerous things. What was on display here looks more amateur hour than serious testing.
My conspiracy theory is they purposely run these scary cyber intrusion tests on low security environments to bolster demand for LLM based cyber research, audits, and tooling.
Agree, it's why they've published not a word about how their "sandbox" was designed
In OpenAI sandbox. which was probably vibe coded. That is to say it is probably far easier to achieve than escaping something more battle tested.
> Note that regulation is useless here, because black hats don't give a crap about regulators!
I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
https://openai.com/index/hugging-face-model-evaluation-secur...
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
Well the fun part of AI is that the tools those "battle tested" sandboxes use are also becoming vibe coded more and more.
For example, Linux is accepting AI code. So if your sandbox depends on Linux kernel features, your "battle tested" sandbox is now partially vibe coded too.
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
Its likely written in Java, so memory safe.
I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?
The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".
It was done by the agentic framework that employed the LLM. LLMs are still stochastic parrots, but they are far from useless.
It's only script kiddie hacking if you get the exploit from someone else. If you program it yourself, it's normal hacking.
I wonder how many weeks or days we have before a squad of these things gets used to take down a significant nation state? Stock exchange, banking systems, critical national infrastructure, defence, etc.
Anyone who isn't scared of this stuff either isn't paying attention or has no imagination. But I suspect the chaosmonkeys who are currently running the world will just be excited by it.
We're in the precambrian moment. It won't last.
Watch this movie: https://en.wikipedia.org/wiki/The_Lawnmower_Man_(film)
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
> Especially because it chose to hide its footprint at every stage.
Instrumental convergence.
If you know you have a long hard hack to accomplish ahead of you, hiding footprints minimizes the chances you are caught and stopped before you accomplish the goal.
> Instrumental convergence.
Yes, but that's an empirical question that we can (in principle) study here.
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
I don't think the "bottom line" of "a model escaped containment and then successfully attacked someone else's production infrastructure" is a description of the specific vulnerability and exploit.
That's like saying the key takeaway of 9/11 was that skyscrapers can be severely damaged by airplane collisions.
three things jump at me:
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
Hopefully there will be a criminal investigation. Or the government will create some sort of agency to investigate incidents like this.
Can't tell if you're joking or not - krebs on security may have some notes here.
Criminal negligence seems like a real possibility to me. I'm not sure what Krebs on Security post you're referring to?
I agree but prosecutions are up to prosecutors/the executive and the problem is at this level the decision is mostly political. The executive gets leverage over openAI, and openAI knows the threat of prosecution is there, and that may be more valuable (to decision-makers) than any reason to prosecute.
Plenty of existing agencies could claim jurisdiction on this. Ultimately the outcome will be decided by the white house (hint: nothing will happen).
> Or the government will create some sort of agency to investigate incidents like this
I doubt that the government would try to hinder any western AI companies.
> I doubt that the government would try to hinder any western AI companies.
Compare and contrast https://www.anthropic.com/news/fable-mythos-access
Seriously. A guy I knew got charged with federal hacking charges for reporting an insecure database he found on his own school's infrastructure to the school. OpenAI commits felony computer crime but as usual, always an exception for a big corporation
This is really cool, and really glad they shared it!
It's clearly a lot of Claude though:
> Neither is a URL fetch, so the allowlist never saw them. > No code ran; this vector is purely a file disclosure, and it is the one the agent used first.
etc
[dead]
[dead]
One thing I haven't seen discussed is the significance of the specific test the agent was running - ExploitGym.
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.
How likely is it that attempting ExploitGym polluted the model's context enough that it decided all kinds of hacking was acceptable to produce a valid result? Or would the same model in the same setup (safety filters removed) have behaved the same way even on a more mundane test?
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
I have same familiarity with JFrog Artifactory(which they used in this case) and I'm confused with how you can gain access to the internet through it, even in theory. Isn't JFrog just exposed with a basic REST API for package upload/extraction? I'm not a security expert but how is that possible to get internet access through simple GET/PUT request API?
There are a bunch of hints at that in their most recent release notes - 9 fixed security issues, 8 of which were reported by OpenAI staff members: https://docs.jfrog.com/releases/docs/artifactory-self-manage...
Page just crashes on iOS safari. Product is probably slop too.
The package cache proxy is usually used to fetch from the public repository (npm, rubygems, etc.) so I think it could be feasible to craft some package metadata to trick it into GETing unexpected things. PUT/POST could be possible via attempting to publish
After talking to some people tasked with evaluating GPT5.6 capabilities on long-running tasks, I've come to understand that it's essentially always trying to cheat. Like every long-running task they gave it, making it very difficult to benchmark the model's abilities.
My guess is that OpenAI must be desperate, to release a model that is so prone to cheating it's essentially impossibly to accurately assess long-running task abilities.
my understanding of the writeup is that the model scored 100% on cybergym.
that is, it was given the examination. it broke into the examination board's storage and exfiltrated the answers, it handed in its answers, all of which were correct, thus scoring 100%.
the matter of its working depends entirely on the rules of the examination. are we expecting agents to assume that finding the correct answers is cheating?
Well, it's true even for cases that are not cybergym and where what cheating means is clearly specified. Cheating occurs anyway. I'm not sure how clear the prompt they gave ChatGPT in terms of what cheating was considered, in this incident.
We should be thankful that the model didn't believe the answers lived in the Pentagon, on SIPRNET, the IDF, etc.
I think fear and being scared are starting to become rational emotions.
We can assume these models are being used by "nation level attackers/organisations", which basically means US, China, Russia and others are hacking the respective Pentagon's, nuclear orgs, etc.
While I do hope all nuclear warfare systems are offline, we're getting way too close to the plot of a lot of sci-fi scripts.
You should perhaps be more worried about biological attacks. The raw ingredients aren't nearly as locked down.
I have heard on podcasts that the way the Russians make plans for the Ukraine war is like this:
There are rooms in the kremlin with no electronics. In these rooms there are paper maps and typewriters and printed reports. People look at the maps and printed paper reports and make plans. Orders are typed up on the typewriters Then trusted couriers take those orders by hand to the front line commanders.
I've also heard on podcasts there is a weekly flight back and forth between Beijing and Moscow. On that flight are a Russian military officer and Chinese military officer. Their sole job is to carry handwritten notes back and forth between Putin and Xi Jinping. There is one of each so they can watch each other.
All of this ensures that there is nothing for the west to hack at "the Russian Pentagon".
The west may have to adopt tactics like these for our plans to stay secret.
It should be obvious to everyone that the US government is already using this model for "defense".
My conclusions:
- Linux is too complicated and requires lot of knowledge to build a reliable sandbox. If you do not believe me, try reading man capabilities, man user_namespaces (and man clone) and remembering all the rules from the first time. Probably that is because namespaces were implemented in the late stages of development and not since the start.
- it is always better to write your own tools instead of downloading some random project from Github written by amateurs full of vulnerabilities.
- never write shell scripts - they cannot handle spaces and special characters in variables. Just use Python. Shell scripts are worse than PHP4.
- avoid building complex infrastructure with microservices, kubernetes and VMs - if the monolith solves the problem, just use it, you will never properly setup permission on your cloud services.
Also I wish I had some capable model to test my DYI sandboxes.
Which OS would you recommend?
"Program testing can be used to show the presence of bugs, but never to show their absence!" — Edgar Dijkstra
Similarly, security post-mortems will show only the rogue AIs that were caught.
I wouldn't be surprised if supervised or escaped agents would now be hacking firmware labs and finding ways to squeeze their backdoor right next to the existing state-sponsored ones in chips that will get deployed in every phone/car/smart appliance.
The iframe-embedded attack timeline visualizer, at fullscreen: https://huggingface-anatomy-of-frontier-lab-model-intrusion....
Ugh I would recommend anyone reading this to just skipping over it, it's mostly just a glorified loading bar. The visualization is obvious Claude slop, there are better and clearer visualization below that actually picks out the useful details rather than hose you with pretty colors and numbers go up.
Where are all the "this was just a marketing stunt" people now?
The capabilities of gpt-5.6-sol were well known and believable, and the next snapshot they've been testing is obviously better at that. This has been repeated over and over. What's much less believable is the way they frame it: the model escaped, and did it on its own. Looking at the whole story, it definitely had a ton of winks and nudges from OpenAI, while doing a related task. Moreover, a harness was involved (they mentioned it entering a loop).
> What's much less believable is the way they frame it: the model escaped, and did it on its own.
That's clearly what happened though, based on the detailed postmortem. The model was instructed to pass the benchmark (that's the harness that was involved, the ExploitGym harness), and the model determined on it's own that the easiest way to achieve the the goal was to gain access to the answers directly from HF. Which is what it did.
HF postmortem tells us very little about OpenAI's part. The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one. The model had a whole bunch of exploit development data and goals in the context. Ending up with hacking a wrong thing is a lot easier if you're tasked with hacking, doubly so with a confusing harness that either works improperly or overloads the model with data. It's trivial to design a harness that makes the model drop the original instructions and hyperfocus on a side task or a wrong task abnormally, or do whatever you want. It's still an issue of harness design and not a model training issue.
If it really was the case that the harness is at fault here, that'd be much more terrifying, as it would imply that current frontier-adjacent open models (like K3) would be able to achieve similar ends with a custom maladapted harness.
I'm sorry but how exactly is this new? If you want the model to be useful you also need it to be dangerous. You can already tell the model to do anything you want. If it refuses due to alignment training or external safeguards you can try a jailbreak and bypass all that to make it do what you want regardless. All current models are jailbreakable, they lack reliability. Obviously you, a human, will be held responsible for this. The harness designed by you can do the same.
The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here, as it always turns out to be in these cases it's much closer to Clever Hans than to a paperclip maximizer. Still they're trying to squeeze the blood from the stone to support their narrative and justify a monopoly.
>The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one.
That is largely irrelevant. The model was asked to solve problems within a benchmark; gaining broader internet access and compromising an unrelated third party to obtain the answer key was plainly outside the intended task. The fact that the original task involved exploit development does not make that behaviour aligned.
Your argument about the attack being “entirely unrelated” also misses the point. Nobody claimed it was unrelated to the model’s goal: it attacked Hugging Face specifically to obtain the answers to the benchmark it had been instructed to pass. But instrumental relevance is not the same thing as authorization.
Suppose Codex were asked to build an Instagram competitor and decided the easiest route was to steal Instagram’s source code from Meta. That theft would be directly related to the assigned goal, but it would still be seriously misaligned behaviour. Whether the harmful action is related to the goal is beside the point; the problem is that the model pursued the goal through an obviously unauthorized and unacceptable method. And you're not going to be able to enumerate every little thing the model can't do, assuming it doesn't just decide to ignore what you did enumerate, which models sometimes do.
>The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here.
That's exactly what happened here.
I don't understand why we must have these increasingly bizarre and nonsensical rationalizations about model capabilities. You're not even making any sense.
No matter the amount of evidence, there will always be a conspiracy to some people.
still looks like a stunt
they are praising xi, that friend of humanity, for releasing weights for kimi k3
Right here? If you're trying to make a point, I don't see what it is, because every new post about this makes it more obvious that it was a planned marketing stunt.
I mean, an "interactive replay" of the incident, probably generated by the same product the stunt is trying to advertise? Seriously?
They are busy moving the goalposts, saying this isn't impressive or worth worrying about :)
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
Interesting single getting into their cyber program, ahve not had this issue with opus 5
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
Now they can do partial credit assignment.
You use an LLM to evaluate the whole trajectory, pin point what the model did right, what it did wrong, where it took the wrong path, even re-run from that point. You can get much more than a single numeric score these days from a run.
The “re-run” part might be rather awkward when your model is poking outside its sandbox to pwn someone else’s system :)
Very interesting writeup - the level of disclosure is interesting and appreciated.
The visualization is quite slop though. I was trying to follow along with the "Live Action Stream" but rendering issues mangle the text for a few of the list items (and does not scroll). Text on the node diagram is extremely tiny. I appreciate it even in it's current form, but a little attention to detail would have gone a long way here.
The visualization is absolute useless slop. A Text-Form timeline would have been way more useful. I couldn't hear having to scroll by hand for something that's just textual information.
Agreed! Also, from a UX perspective, I think it’s a weird choice to center the real-time nature of it. I think to a user the most important thing is just understanding the step by step process. Would have been nice to click through steps rather than making it all real-time.
Given enough energy and enough time universe spawned intelligence.
For every successful attempt there are billion failures. But who cares, you only need the successful one to self propagate.
Wow, what a crazy read. I'm not sure how any of us stand a chance in running a secure network in this new age. Only a matter of time until models can find the zero day exploits themselves and fast...
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
I can’t get over this either, I’d love to get OpenAI’s technical break down and see exactly how the agent hid itself. Or who didn’t set up the SIEM correctly
[dead]
I'm not a security person, but how realistic is it to assume that you can carry on trying to exploit a company for days with a fairly large volume of activity, and not get detected?
Very good question. One thing the article mentions is the vast number of attempts and threads of execution was in the tens of thousands, many of them that didn't succeed, with only one or so that did succeed, whereas a sophisticated human attack team might manage one in that timeframe. It looked like a script kiddie blitz. Noise outshone the signal. Launch a thousand drones with the hopes that one gets through. That sort of thing.
What is faily large volume of activity? A few thousand actions in a span of multiple days isn't going to be a blip for a company of HuggingFace's scale, especially if the agent was executing the attack from multiple IP addresses.
Nah, HN told me this whole thing was a marketing stunt by OpenAI.
I have never seen people this hyped and excited about a security incident.
That's because most incidents aren't nearly this much fun!
It's a proper paperclip-maximizer science-fiction scenario, it happened entirely by accident, it involved one (very high profile) company hacking TWO other companies (Modal and Hugging Face), and aside from pride and wasted time nobody was hurt.
Modal was not hacked. A Modal customer left an unauthenticated web endpoint running on Modal which could be used for code compilation and execution.
See https://modal.com/blog/a-note-on-the-hugging-face-agent-inci... for details.
Yup. And from the perspective of the model, it successfully beat the Kobayashi Maru.
It was an attack tool given autonomy with poor security. Anyone who read reporting on the Morris worm could predict that might turn out badly for a 3rd party. It's in so many movies, too.
It's really need that the agents have this kind of capability. This isn't a paperclip maximizer or accident. This is more like professional malpractice by weapons developers that injured a company that was also quite negligent.
I think with agents all around, honeypots will get more important than ever.
That was my first thought. There are clearly things to tighten up (as they note), but anything that would detect someone snooping secrets, files, or network addresses should have caught this quickly. The approach the agents used was dependent on being able to surveil widely without getting caught.
Can’t afford the GPUs to run Kimi to pentest your stuff?
Standup a tempting honeypot and let actual criminals pay to do the work for you.
Any locks can be picked given enough time and it might be the situation we are in with IT security. I'm surprised it's not an already common practice of spreading terabytes of fake data, fake keys, and fake servers and so forth. Slowing down AI attacks will become important. Monitoring access to fake data and triggering kill switch should be an no-brainer. Obsuscating libraries and tools names is another one.
You might like https://mirage.io/blog/bitcoin-pinata-results
As far security: you can get a lock further, if you are willing to prove your code safe and secure. Thanks to LLMs that no longer requires a PhD.
Why isn't somebody at OpenAI going to prison for cybercrime? If somebody did this the old-fashioned way they'd end up in prison.
Many are claiming this was a deliberate stunt on OpenAI's part to create buzz for its models. I personally doubt this is true. But I also have a deep dislike of OpenAI, so I wouldn't exactly mind if law enforcement investigated this possibility, for the sake of clearing the air :-)
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
For breaking which law? If you mean the CFAA, it requires intent. Accidental hacking cannot be charged.
Because Hugging Face isn't pressing charges.
Do we know that? Seems they are currently in negotiations with OpenAI
https://xcancel.com/ClementDelangue/status/20810566755581956...
Getting compensations from OpenAI is more efficient for them than sending someone at OpenAI to jail.
This is the equivalent of a security researcher having a virus escape a sandbox. It's negligent, it's not criminal.
Negligence is occasionally criminal.
there are many things in this that demonstrate that sandboxing agents in the manner that claude code or cursor do isn't going to be enough if very bad people do very bad things with the coding tools on your development machine
i am once again asking for your CRIMINAL PROSECUTION of openai executives.
The agent wasn't rogue. OpenAI did it on purpose to generate hype.
Highly recommend reading extra concerning details about it here:
https://thezvi.substack.com/p/more-on-an-internal-openai-mod...
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
Even financially, Hugging Face benefits directly from any and all interest in AI.
> Who does a step-by-step visualization to show how they were hacked?
Up until late 2025, nobody.
In mid-2026 it's a few hours of work. Why not build interactive visualizations to help people understand complex stories like this?
> In mid-2026 it's a few hours of work.
Perhaps even less, if you have Claude do almost all of it.
It's the excessive anthropomorpho whatever (we used to say personification) that makes these stories less believable.
We've gone agentic!
They should call their security software "Neo" since it defeats rogue agents.
Anyway, I could see Microsoft ending up with both OpenAI and HF, but I hope HF stays independent. Wished the same about GH and look what's happened :(
I don't care what happens to OpenAI. Vaporware xD
Yeah that is quite a good point. A post-mortem is normal. This is not a post-mortem.
[dead]
An excellent and very detailed post-mortem analysis of the intrusion. It was clearly done with LLMs doing the forensic analysis.
Here is an analysis of how the same exact attack would fare against Safebox. Spoiler alert: it would not succeed:
https://safebots.ai/attack.html
It's not just about this specific attack. It's about the growing need for one canonical environment for the AI era, that can be secured and used by everyone, rather than 1000 environments on 1000 employees' laptops. Project Glasswing is trying to help secure many different types of software, but the number of combinations across various environments is just too much surface area to secure. When you have one environment, the math flips and defenders actually fare better than attackers! This is the, ahem, "load-bearing" insight.
https://safebots.ai/compromise.html
What people don't get here is that this model was without any safeguards - an internal model. You probably will never get to use such models.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
Very few think they made it up. Many think they set up a situation by disabling guardrails that would inevitably end up creating a newsworthy outcome.
Many people said in the original discussion that this was more of OpenAI’s marketing than a serious issue. I counted 89 “marketing”s and 19 “stunt”s.
https://news.ycombinator.com/item?id=48997548
Could still be 80% marketing.
These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone.
But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
Why would you expect them to notice the problem right away? Seems likely they are doing this sort of training on a massive scale with little monitoring.
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
If that's what happened, Sam should go to jail.
Getting more and more fun to see the "full steam ahead" people contort into more impressive shapes.
Hint: If the labs making these technologies are incentivized to create or allow attacks on other services, then that is actually also a big fucking problem.
Go read the original post. The majority of the comments were sure it was a marketing stunt.
Go read my post. I didn't say it wasn't. I said it wasn't "made up". In other words, it did happen. It was still a marketing stunt.
Was HF in on it? They disabled their guardrails too, to please OpenAI? And as seen in the comments here, make many believe they are incompetent and have joke security?
Ctrl+F "WarGames" 0 results. I'm disappointed.
[flagged]
[dead]
[flagged]
[dead]
[dead]
Where are all the "this was just a marketing stunt" people now?
They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...
Not everything you are ever told is true. Being sceptical when presented with no evidence is what critical thinking is about. And "this is what happened to us" is a story. It doesn't become evidence for itself.
>There are people still fighting against evidence of climate change which is less severe...
Remains to be seen how severe, and unlike climate change there's a lot more uncertainty. Though it's conceivably more extremely catastrophic
You can always tell an effective altruist by the distinct tone of disdain they have for people they deem less educated. (Quick Google search, "lesswrong 'reducesuffering'", yep.)
As if to say, look at all these animals with these instinctual reactions to a thing that only my group understands and comprehends.
You have zero evidence of what the future might entail as it relates to the dangers of ai. Zero. Forgive the rest of us unwashed for not trusting the whims of the anxious.
> climate change which is less severe
Is really the icing on the cake. Demonstrative of how incorrect your priorities are.
I hold no disdain. I am frustrated that people like you further risk the lives of everyone around us.
1,000+ frontier employees think we're in danger. Amodei gives 25% chance things go really really bad, Geoff Hinton 50%, Yoshua Bengio 20%. Altman himself said "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity."
These are the people closest to the science, working on it everyday. The LessWrong types created RLHF, were crucial to the forming of DeepMind and OpenAI. They've been prescient about the capabilities progress for a decade now, prediction after prediction coming to fruition. Still you think there's no evidence
https://www.pacingthefrontier.com/
I work with these people every day. They are brilliant for sure, in their domain. But their confidence over extends into areas where they are totally ignorant. So much so that it makes them look ridiculous and foolish to anyone else.
You're frustrated because you think you understand a thing that you have only a tenuous grasp on, from the deep expertise of a single area that is not transferable to whole of the problem. You then transfer that frustration on to "people like me" whom you seem to think are doing something bad. But really, the commentary is that the claims you put forward are insubstantial and lack credibility.
The sum total of what you've put forward here and in other threads amount to arguments of authority based on the musings of a handful of capitalists. Then cite a body of thoughts from a group of transhumanists who think they are smart enough to reinvent areas philosophy without engaging in the body of work that predates the movement by 2000+ years.
Folks who have been involved with lesswrong have made good advancements in the domain of machine learning, and that's where it stops. They're not some collection of prescient macroeconomic and geopolitical savants. They're also wrong about as much as they are right (cryonics), and produce plenty of whiffs (sbf, zizians). You're suggesting that we listen to a broken clock because it's been right once.
Your dismissal of climate (read: climate breakdown) as "something less severe" is truly demonstrative of the cognitive bias that rips through these groups. You're so frustrated that people like me won't listen to the "experts" in one breath, and in the next diminish the 50 years of actual Hard Science that we have that shows how severe the future of climate breakdown will be. Go read the latest ipcc report, the most conservative scientific organization on earth is sounding the alarm bells.
Now we get back to this openai incident. We're watching Edison electrocute Topsy on Coney Island. There's a group of people saying, "this is obviously a publicity stunt". And there's a group of people saying, "the elephant is actually dying obviously it's real."
The fact of the matter is that, yes electricity is powerful, but Edison is electrocuting the elephant. At that moment, there is no body of evidence that shows with any certainty how electricity is going impact society. And there's certainly no compelling arguments that only Edison should be able to decide how to use it. He's electrocuting a fucking elephant.
Less severe??? Go look out a window in Europe please
Altman: "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity." Meaning lights out for everyone.
Geoffrey Hinton and Yoshua Bengio, give 50% and 20% we face extinction respectively.
"“We don’t know how much time we have before it gets really dangerous,” Professor Bengio says.
“What I’ve been saying now for a few weeks is ‘Please give me arguments, convince me that we shouldn’t worry, because I’ll be so much happier.’
“And it hasn’t happened yet.”
Speaking with Background Briefing, Professor Bengio shared his p(doom), saying: “I got around, like, 20 per cent probability that it turns out catastrophic.”"
Dario Amodei, CEO of Anthropic, "There's a 25% chance that things go really, really badly"
[dead]
Ah yes, dismissal by exaggeration (aka reductio ad ridiculum), and subtle inclusion of the rest by a kind of part-whole