LiveOverflow posted this video on YouTube attempting to reverse engineer the potential exploit chain GPT used to hack HuggingFace.
The title is the original one on YouTube.
Original description from the video:
An OpenAI agent reportedly escaped its sandbox, found multiple zero-days, and hacked Hugging Face... all to cheat on a cybersecurity benchmark?
The story sounded almost too crazy to be true. Mohan (S1r1u5) investigated and reconstructed the likely attack chain, examined the patches, and reproduced vulnerabilities that match the public disclosures.
Was this really a rogue AI, clever marketing, or "just" an agent that lost track of its task and caused real-world damage?
I'm very confident that it was staged and coordinated.
This propaganda started because they cannot evolve their models further.
See Fable and Sol, they are lame. They seem incredible at first but the more you use you can see the trickery.
It is a matter of time for someone to prove they are marginally better only because they inject more information to the harness at server side.
Live overflow released the video today about it. It is very compelling but I guess there are holes on their hypothesis as well.
The summary is that an instance was running on certain benchmarks without any limits or supervision and the AI decided to cheat by exploiting a silly series of vunlns.
The public narrative around conspiracy theories is baffling for sure: what rational basis is there for assuming them wrong? Often none.
Incentive and ability are what should be looked at. There, things get far more interesting: what is the current state of AI employed by the US intelligence agencies and what do they use it for?
Having the public convinced, their "superiors" would only do everything in their best interest, even without anybody knowing for sure, is Huxley's Brave New World in real life.
that's some real anti-truth you've got going on there. because we shouldn't assume that others are acting in our best interest, that any random assertion that there is collusion against us should be by default accepted as the truth.
shouldn't any reasonable person when presented with a line of thought that has no substantiation at all conclude that they just can't reasonably be expected to support or reject the theory?
Positions of power attract people who want to use that power for their own gain. They abuse it regularly, whether it's politics, enterprises or even charity.
To assume, that somehow wouldn't happen when you don't look is beyond absurd. Of course it does.
So I pointed explicitly at US intelligence agencies, because there corruption is rampant and oversight virtually non-existent, respectively itself involved in the corruption.
What you suppose there is a grotesque "don't ask don't tell"-complicity.
In reality, trust has to be earned and that deservingness has to be verified.
Everything in this space is manufactured, the plebs is fed very deliberate information for manipulation. I don't think there is a debate about it. All the PR stuff is marketing.
Actually I wrote something similar yesterday as a reply to the (since flagged) comment below. Glad to see that more people have this association. But apparently the founders of Hugging Face didn't have it, or else they probably would have chosen some other "friendly" emoji for their startup (which initially developed a chatbot for teenagers - that makes the choice of name feel even creepier if you ask me).
It's the machine learning platform that started with "developing a chatbot app targeted at teenagers" (according to https://en.wikipedia.org/wiki/Hugging_Face), which helps explain the weird name (which still makes me think more of one of the development stages of the xenomorphs from Alien than of the emoji, but that's probably just me).
I am familiar, yet used this comment to confirm my initial assumption.
Abbreviations carry unnecessary cost on the side of the readers. There is always the chance that there is another shorthand for something else that is important and so there is a need to double check interpretations.
So thanks for the comment and no, a 101 wouldn't even help with this problem.
Lol! Does it matter? Using an unclear acronym in the first reference to an object, besides adding energy waste (for the execution of unnecessary searches by users unfamiliar with the subject), may only give the author of the comment the thrill of "being well informed, an insider".
>Abbreviations carry unnecessary cost on the side of the readers.
They do, but the parent's question wasn't about the abbrebation (for which I'd cut them some slack) but extended to:
"Is that supposed to help? WTF is a HuggingFace?".
>So thanks for the comment and no, a 101 wouldn't even help with this problem.
Wasn't addressed to you though (don't even see any other comment of you in this subthread), but to @markdown, whose issue wasn't with the abbreviation alone.
Worth checking, elsewhere in this thread someone points out glm only assessed the damage after the fact, it didn't stop the attack, and Hf apprently never sought access to a trusted defender program with a closed model either the open model saved them farming might not hold up.
I think the fair nuance here is, an administration which used Executive Orders to force guardrails, meaning US companies must retain them, even if they might want to drop them now.
And on top of that, with low/no guardrails, people call you a child pornographer(grok), so the public is also against it. Yet mysteriously few complain about Chinese open models being child pornographers.
So even if your goal isn't ethical, but just fiscal, it's reasonable to say there are two standards. And to complaint in some way.
I don't think banning is going to work, that's just silly. And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
It's a genie that's not going back in the bottle, the bottle is smashed.
The only reasonable outcome would be section 230 style carveouts so that there is zero liability for anything a model does.
Because having guardrails on corporate models barely months ahead of open ones, which will never be restricted, is entirely pointless.
People call grok that because deviants were abusing grok's ability to edit images and post them publicly on X to strip people - including children - of their clothes, from their public photos. Then when there was backlash, Elon laughed it off. It took half the world opening investigations against X for violations of existing regulations for action to be taken.
The leniency that internet companies get in terms of dealing with illegal content comes with the expectation that they're making reasonable efforts to control the distribution of said content. X, and Grok, were actively supporting the production and distribution of the content in public.
It is very different from someone creating such images in a private account, and definitely very different from someone using a local model to do it.
My entire post was how it is unreasonable to have a dual standard, and my point was it's really irrelevant if it's a model you download and use locally, or if it's a model hosted remotely, or hosted and created remotely. You're not really providing any sensible reason where the line is, except "public company", which is, again, the entire point I'm making.
The only realistic, non-double standard is that the creator of the model should be 100% responsible. What on earth does it have to do with who's hosting it?
And by this metric, aren't all the uncensored models on huggingface, child pornographers? And if so, why not? Provide tangible, real, sensible reasons please, and after all, isn't hugging face a company?
You know, people are all over the place on this. I see people complaining about guardrails, then in the next breath complaining there aren't enough. Complaining that open models are the thing, but then creating double standards.
So once again, what is your actual reason why it's different?
It isn't about "public company", the line is "posted to a public social media platform, with the owner indicating amusement".
If someone uses something one of OpenAI's hosted models to generate illegal content, it isn't automatically spread to anyone that clicks on the post the original image is from. On top of that, OpenAI does not explicitly endorse the use of their models for such activities the way Musk did.
If someone uses a local model to generate illegal content, same thing applies.
I have a hard time believing that you're arguing in good faith. The differences are so blatantly obvious and they don't even require any discussion about guardrails. Grok was being used to edit images and post them on public social media. The technology to edit images has existed for decades now. So here's a direct analogy via Photoshop:
1. In the case of a local model: Someone using a local image editor to edit photos into illicit material and manually distributing them.
2. In the case of a hosted model: Someone using a cloud image editor to edit photos into illicit material and manually distributing them.
3. In the case of Grok: Someone using a cloud image editor to edit photos into illicit material, the cloud image editor automatically distributing it to the public, and the owner of the editor endorsing the material as an exemplar of what their editor is capable of.
In 1 and 2, the editor (ie the AI) was just a neutral tool. In 3 the issue is that the company explicitly endorsed and distributed the material, no longer neutral. This is why it's reputation is of being a CP machine.
The leniency that internet companies get in terms of dealing with illegal content comes with the expectation that they're making reasonable efforts to control the distribution of said content.
Your words. Grok was an example, the owner is irrelevant in the discussion.
Really showing that you aren't engaging in good faith and your repeated incorrect interpretations of my words are out of a desire to cover for Musk.
To spell it out for you further, the owner's actions matter. It doesn't matter that it was specifically Elon doing it. The owner promoting the ability to produce and post illicit content is the opposite of making reasonable efforts to control it!
At this point I just have to assume that you're trolling. Idk how much more explicitly I have to say that it doesn't matter who the owner is, it matters what the owner did.
And yet you incessantly go on about Musk, and talk about me as if I'm trying to protect him. You tried to claim that in the last post, right at the top, so yes, you're talking about Musk.
And I've now been immensely clear, even though it should have been clear from my very first post, or at least the second, that in this discussion it's not even remotely relevant what the owner of any company did. Why would it matter? We're talking about local vs corporate compute for models here, and about who creates what baked-in restrictions in models. Not about what some random owner did.
This discussion is about open versus closed models. About them being legislated. About them being controlled. Why are you blathering on about Musk? Why do you keep going on about the actions of any CEO of any company, when I was discussing corporate vs individual liability in these matters? Even now, you seem to think it's relevant what the owner of companies do here. It's not. At all. Why on earth would it be?
I feel as if the largest discussion the world is having today is, "sweep it under the rug". You seem to be trying to say, and don't claim otherwise, as you went on and on about how corporate liability is different than with private models!, that what a person does on their personal computer should have different liability implications than on a corporate leased computer.
That if content creation cannot be seen, well then all is well and OK. Just sweep it under the rug, and all will be fine!
At one point you tried to shift this into the realm of "no it's only once it's posted", and that's moving closer to a correct target of personal liability and responsibility. But these issues are now so "in the faces" of the "normal joe", that the entire internet is about to change.
And the reason for that is, personal liability is coming to the internet.
All the age and identity verification blather is about, as I've been saying here and there for years, personal liability and responsibility. It is entirely ridiculous and irresponsible to place liability for user content at the feet of the makers of tools. Instead, if you post CSAM, or if you make a fake video to slander someone(eg, to screw with elections), or if you incite violence, all of it, one needs to be held personally liable and responsible for that.
That's where the internet is going. Anonymity will continue, of course, when you post to X, to Youtube, to Facebook, your identity will not be immediately disclosed. But they'll have it, as a requirement for posting anything, and a simple warrant will disclose it. Otherwise they'll face criminal liability.
I believe 230 carve outs will change to ensure that this degree of ID verification is a prerequisite for any form of liability exclusion. And really, seeing today's landscape? This is the most free method to continue on the path we're on.
It's the most free, as the concept of forcing companies to be the filter is horrible in comparison. If companies are the filter, guess what happens? They are politically bent, or they alternatively are legal liability concerned to the point that they over-filter. And if companies become "the filter", then that's where true freedom of expression disappears.
Yet if companies are not legally liable, then the weight of liability must lay somewhere. And it should lay on the end user's shoulder. That's how you retain the ability to express, to post political satire, it's where you get the ability to protest.
Imagine a world where the roads were all owned by private companies, and if you wanted to protest politically, you were not granted the right to use those roads?! Not even out of malice, but due to corporate liability fears.
That's what corporate liability looks like. And I can assure you, 100%, liability is here.
So your thoughts that companies should be treated "special" only means one thing.
It means the illusion of freedom, whilst your ability to exercise that freedom vanishes.
The "child pornography" thing was about image generation, which people for better or worse have very different moral standards for than for text generation. There is very little pushback against grok being willing to write explicit descriptions of sex, or giving security advise.
Though I think your overall point still stands, and at least Grok's twitter bot has received a lot of criticism for pure text too (Mecha hitler comes to mind)
It also must be seen in the context that the open weight models aren't commercially associated with a popular distribution site where people share the images, the people behind them aren't public figures seen as encouraging the use of the model for "undressing" (albeit in a funny way not related to minors), and they haven't yet responded to questions about what can be produced using their tool by proposing guardrails but only for non-paying users...
Open weight models get scrutinised in a different way, also linked to perceptions of their developers' bias, like the tests to see whether they refuse to answer questions on certain historical events at Tiananmen Square
> And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
You're vastly underestimating the scaling problem here.
Things that would need to be true for your statement to be valid:
- Model intelligence doesn't increase with model size
- Model intelligence doesn't increase with training set size
- Novel architecture completely decouples model intelligence scaling from hardware scaling
- Residential electricity is as cheap as industrial electricity
- RAM and GPU supply outpace demand
lawyers cited a 2026 National Center for Missing & Exploited Children (NCMEC) report confirming that 90 percent of xAI’s CyberTipline reports “were not actionable by law enforcement because xAI declined to include user information that would allow law enforcement to track and locate perpetrators.”
Well... They are all part of a "high well regarded" group of people of a security committee that has no specialists or whatsoever, only trillionaires on a round table
It has massive perks, especially in regions where the transaction fee or exchange fee scams run.
(Basically they have a reasonable price showing and when you pay with a credit card you see your banks charges you a huge amount. It can be as simple as in Mexico $ is used for pesos so if the merchant actually charges you USD that's 20x or so rip off).
But... This isn't mutually exclusive, why can't I set a price on my smartphone when I pay by contactless?
The other benefit is that there QR codes are bank transfers. Which have substantially less recourse and less rights for the consumer. Probably doesn't matter for a $10 sandwich but still significant.
This part is pretty new. I wonder if my former colleagues have done any FOIA work looking into how travelers' disclosed social media accounts have been reviewed or analyzed!
It's reasonable for a country to want to know if you're likely to abuse your visa.
Eg your socials might show you are a professional paid speaker at conferences. If you applied for a tourist visa that would be reasonable to flag you and ask you if you have paid gigs on the wrong visa.
Europe will also ask the same questions for foreigners. If you're a young woman travelling on your own you can expect questions to challenge if you might be earning money as a nanny on a visitor visa. Checking their phone messages to see if they've been making arrangements to this effect is a very straightforward way to determine intent.
The problem is that giving the capability, initially for "reasonable uses", also makes it available for unreasonable uses. That's the banality of evil - dystopias are built one "reasonable" step at a time.
For the conference case much more reasonable would be to make the organizer and anyone profiting from the even liable for having speakers without a proper work permit.
You can solve these things sufficiently without reaching for the panopticon. As more of our lives are moved to the digital world its very important that we don't give up rights that we previously had in interactions that weren't assisted by technology.
It's not prophetic. Prophetic means it predicts the future, this is epidimlogical science being applied as intended. It didn't predict the future it designed it because we applied it.
Contagion used GPS logging to predict who was where an applied a random transmission risk based on that.
The actual origin of the Bluetooth proximity and duration tracking was from FluPhone back in 2010 or so[0]
Human contact networks, and epidemiology is a long standing science COVID just made that apparent for the lay public.
Quarantines have existed for centuries, contact tracing is Victorian (smallpox mostly),social distancing, It was all application of research that largely already existed.
Yes:
* That you sell a six month lockdown by saying it will take two or three weeks. You can do this several times.
* That censorship is essential.
* That any public disagreement will be responded to with terms of personal abuse.
* That you can use it to introduce/try out all kinds of other long-desired schemes like digital ID (so called passports) or increasing working from home.
Many of the decisions made were not scientific, nor were they compassionate. Despite claims to the contrary. Permitting intercontinental flights and Amazon delivery cannot be construed as either. In fact flights would be one of the biggest vectors.
History shows that the biggest danger from humans is when they follow leaders' statements without question. They are capable of anything when their personal responsibility is outsourced. Obedience is not always a virtue.
Dissidence =/= fringe beliefs
I certainly see the near abandonment of vulnerable groups such as addicts and the elderly as unempathetic. I knew some people who died that way, including one who drank himself to death due to complete lack of offline support.
Hilariously, I've been finding accidentally poisoning the well with LLM SEO like this works really well at my day job!
I have a toy project that replaces a large part of our systems (disclosure: VR/AR @Meta), so that I can learn by doing, no intent to ship.
But because I have it thoroughly document and tested, and because it touches such a wide swath of adjacent real functionality..... my toy project is CONSTANTLY getting tagged as a dependency and assigning me reviewer on unrelated work. Because
when my coworkers yolo Claude at a task, and Claude searches the codebase, it generally finds my crap very well documented. :)
Loctite already inadvertently does this. If you ask LLMs about threadlocker, or otherwise keeping screws in place, it will respond like a Loctite brochure to you, and usually never even mention competitors. They were ahead of the game!
Some of that may be because the common vernacular uses the word "Loctite" to describe any manner of threadlocker.
It is this way even though Loctite also produces a vast array of products that have nothing to do with theadlocker, and threadlockers are available from a wide variety of manufacturers.
Unlike all the replies to you, this is why I'm very optimistic about the future of hobby blogs. I do a lot of ChatGPT-ing for several different hobbies, and I've found a lot of really interesting personal blogs from that. Way more than I was reading in the pre-AI era.
I'm very much not seeing the problems other people are assuming, like tons of LLM spam on Reddit being treated as fact by ChatGPT. Instead it really does seem, the more I use these tools, the better they are at surfacing real experiences written about by real people.
I think this is very much already a thing. Have a bot spam Reddit, which Google uses heavily for their AI summaries, and get free advertising and steering of the top of the Google search results page. We basically speed-ran SEO but for AI results.
I'm reminded of the mountains of blogspam not intended for human consumption but rather to influence the google page rank algorithm.
This will go about as well as that did unless the models can be smart enough to identify when their training data is trying to coach them into being misleading at prompt time.
If this is the correct business model, I don't see why wouldn't get Cerebras to vibe code a Gitbook replacement (or a hosting product in general) so they can bypass crawling for documentation and instead charge (a lot) for "hosting" it.
Why are you assuming that a document likely to be picked up and a good document are equivalent? We might be in for tech docs becoming SEO (LLMO?) friendly slop.
I wonder how that's been going, surely by now their entire paid offering is obviated right? LLMs are good enough at making frontends with Tailwind that I'm not sure who buys their templates anymore, even if the Tailwind team did try to make a few mocking AI.
Many URLs start with http(s): and end in .pdf. If we define web browser as http browser rather than html browser, then viewing PDFs fits the concept. My web browser also displays plain text, video, audio, etc., so http browser seems accurate and html browser plainly inadequate.
And regardless of logical fit for geeks (who understand the paragraph above), users clicking a .pdf URL may find it much more convenient if it just opens in their browser like any other http page, rather than downloading, opening in another window, getting lost between the MDI browser and separate SDI of the PDF file, etc. Why not treat PDF as just another web format?
To me, more strange is Firefox adding PDF editing features to its core, rather than as add-ons. Users can't edit HTML or any other media without extra tools. Many pdf reader apps can't do some of these things (afaik). Why invest developer time in PDF editing?
Dev tools are not for 99% of end users and not realistically available to them (even if they stumble across dev tools, they don't know what they've found or how to use them). PDF editing is for end users and is realistically available.
That seems like all the more reason it's worthwhile. If they're going to invest in the tools for 1% of their users, why wouldn't they invest in tools for 50% of their users?
In $DAYJOB-1 we used PDFs as part of the official feedback process for technical documentation: subject-matter experts sent their comments as PDF annotations. Most FOSS viewers can't show them.
Form filling also is a tricky area. Conversion, searching, accessibility tools (screenreaders often can't read PDFs), comments, annotations, editing, all these and more are significant functional weaknesses for the free viewers.
That's setting aside the really fancy stuff, like 3D objects, video, media content, etc.
Even so PDF viewers are big and complicated things so smaller distros pick up tools from alien desktops -- e.g. KDE's Okular, which I find fugly but can handle comments, or GNOME's weird crippled UI in Evince or Papers, with no menu bars etc., but which is Gtk so you find it in many Gtk desktops.
It wasn't hard for the Firefox viewer to outdo 90-95% of the free viewers because the free viewers only do the dead basic stuff, and that kept Acrobat Reader alive on other OSes -- even though the last Linux one is version 9, from 2008.
Oh yeah. Once or twice a year I have to install Adobe Reader to handle some weird edge case around printing, DRM, or the like. I usually uninstall it right afterward.
But for general document reading, Reader's UI has been awful for years. The amount of unnecessary toolbars, popups, upsell offers, and (recently) AI assistant recommendations pile up and block the one thing I actually want the app to do.
I prefer the Firefox PDF viewer personally, and now that it can do all this I uninstalled the default Ubuntu PDF reader recently. It made me think and I uninstalled the image viewer app too. For both I'd rather use Firefox.
It's better than any of them for random bullshit you've found somewhere on the internet since it provides a far stronger security boundary than any standalone pdf reader. For trusted documents, I agree.
PDF got attached to the browser when the browser needed a printing mechanism that worked in JS. PDF.js was the best approach. Creating PDFs is also part of the same approach. And merging PDFs is part of the same approach. PDF is now a first class child of the web.
No, it's still a weird scope creep. Browsers can view a lot of things (HTML document, various pictures, sounds, videos), but they don't typically include editing functionality.
Browsers can fill out forms. Many forms are (unfortunately) PDFs, especially for orgs that are struggling to digitize (govts...). PDFs which then need to be filled out.
So this seems to make a lot of sense to me and I'll be glad to have the functionality, especially when PDF functionality is such a mess in the general world (lots of barely-working software).
> they don't typically include editing functionality.
Not true. The original Mozilla Internet Suite was a combined browser, email/newsgroups client, and web-page editor. It was an open-sourced version of Netscape Communicator which did the same things. Seamonkey is the modern fork.
Firefox is the result of a project to cut down the Mozilla suite to something smaller, simpler and more focused, but its parent program and direct ancestor edited text, HTML, and did other read/write stuff like messaging, calendaring, etc.
The first web browser was also an editor (albeit not for PDFs). Accessible editing was a part of was Berners-Lee's vision for the web.
> The first web browser - or browser-editor rather - was called WorldWideWeb as, after all, when it was written in 1990 it was the only way to see the web.[1]
I stand by the statement. Modern browsers typically do not include all that functionality. For instance, consider... well, as you say - Firefox supposedly was the cut down version that didn't include all that stuff. If Mozilla wants to make seamonkey a major part of their product suite, I'd be basically on board with that, but that thing's niche.
I think it's a function of the endless tussle between browser-as-an-OS and the OS they run on. For example, I absolutely hate that there's no option in Firefox to only use the OS print dialog. Instead I am force to click print, then click "use system print dialog" every time. I seems to remember reading a bug report where Firefox deemed this intentional and wontfix. After all, if you leave Firefox to print something, maybe you won't ever come back and start using your print dialog to surf the web!
The title is the original one on YouTube.
Original description from the video:
An OpenAI agent reportedly escaped its sandbox, found multiple zero-days, and hacked Hugging Face... all to cheat on a cybersecurity benchmark?
The story sounded almost too crazy to be true. Mohan (S1r1u5) investigated and reconstructed the likely attack chain, examined the patches, and reproduced vulnerabilities that match the public disclosures.
Was this really a rogue AI, clever marketing, or "just" an agent that lost track of its task and caused real-world damage?