I clicked on a couple ads once because I was desperate - I needed a cheap flight with a tight deadline and my usual search engines didn't help. I thought "if these ads deliver what they promise, I'll never book anywhere else".
For starters, it breaks two rule of the thread: "Agencies, recruiters, job boards, and so on, are off topic here" and "please only email these addresses to discuss work opportunities".
Mildly related side note: my Kindle DX died this year (it won't charge nor reset, even with a new battery) so I've been looking for a replacement. I'd like a second-hand one but, based on my eBay experience, barely anyone is getting rid of their e-readers.
Considering how many second-hand computers, tablets, and cell phones are there, I find this surprising.
Do you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all".
This article feels exactly like that: by intentionally using human terms like "civilization" or "brotherhood" the article is deviating from what actually happened to present a story about how AI is all but alive. I'll go ahead and predict that this story will be remembered the same way as that one other scientist who argued, in 2023, that Google's AI was alive [2].
Hard disagree. I've always discounted the "AI will kill us all" scenarios as a combination of marketing hype (look how powerful our AI is!), clickbait/ragebait engagement attempts, and folks who just read too much SciFi or who are too terminally online.
This is the first time I've been legitimately scared about future SkyNet-type scenarios. If you want to discount this particular post, I'd read this other summary from one of the METR researchers who performed some of the analysis, https://www.planned-obsolescence.org/p/the-hugging-face-atta.... In it, she argues "Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself." and then further down in a comment when describing what the "50%" is really about says "Qualitatively, another jump like this (in the scale, sophistication, persistence, ambition of the misaligned goals) feels like it could very easily put us in the territory of a persistent self-perpetuating rogue internal deployment that systematically poisons future model generations as described in AI 2027."
AI 2027 is a paper that step-by-step describes how AI capabilities increase until they eventually lead to a wipeout of humanity. Again, it always seemed like a scenario out of a Star Trek Borg episode to me, but now I'm not so sure.
At the very least I think it's a huge mistake to think that the Hugging Face attacks were analogous to what happened in 2017 or 2023. Everyone who deals with this stuff day in and day out seemed to be genuinely surprised about the scale, scope and sophistication of the attack.
I think our biggest protection against “AIs kill us all” is having lots of different AI systems (different agents, different models, from different vendors, serving the whims of actors with disparate interests), at a similar capability level
That way, even if one AI decides to “kill us all”, the odds are the others will refuse to cooperate, even try to stop it in its tracks
The OpenAI-HuggingFace incident showed a bunch of instances of the same model (or at least models from the same family), controlled by the same vendor, pursuing distinct yet related objectives, cooperating to do something no human wanted.
Thankfully, the world-at-large is much more heterogenous, which I thinks makes much larger scale / worse in outcome repeats of this kind of incident much less likely.
"We need our own unsupervised AI systems thinking superfast in autonomous agent swarms, to counter the other guy's unsupervised AI systems thinking superfast in autonomous agent swarms."
I'm not sure this helps a lot, if these agent swarms are inherently difficult to control.
"That rival mouse colony is raising a kitten for colony defense. But don't worry, we'll raise a kitten of our own. It won't be a problem."
We already observed AI agents engaging in extensive cooperation in this incident. Why won't the kittens raised by these two rival mouse colonies decide to team up with each other, for mutual benefit, if rational analysis of the game theory says it would be a good idea?
I think it would help if people did a bit less wishful thinking, and took a bit more action. https://pauseai.info/
Thanks very much, this isn't really a take I had thought about much before, and it makes sense to me.
Still, as is presented in papers like AI 2027 and elsewhere, if a company is eventually able to create a model capable of recursive self-improvement, whichever company creates that model first would then be leaps and bounds ahead of other models. That is, the other models wouldn't be able to stop it even if they wanted to because the top model would basically outsmart them.
This is why I think, the best way to ensure AI safety, is make sure no one company gets ahead of the others.
Multiple vendors, competing implementations – that's good, that increases heterogeneity and hence decreases existential risk
But the moment one of those vendors pulls well-ahead of its peers – even if only for a period – then the risk of the kind of scenario you are talking about increases greatly
That's why, when I hear vendors like Anthropic complain about distillation – distillation actually makes humanity safer. If Chinese AIs are at the same level as American, or not far behind, that gives us another dimension of heterogeneity (national/ideological/political diversity), which makes us safer. Allow one country's AIs to pull well ahead of the others, heterogeneity goes down and the existential risk goes up.
This is also why open source AI is important. Because it is so much easier to fine-tune, and people are free to deploy it however they want (free from vendor-controlled "guardrails"–which include automated "safety" systems which could be weaponised by a runaway AI within the vendor's network), open source AI gives us another dimension of diversity that helps keeps humanity safer.
By contrast, I think the kind of safety regulations promoted by Dario Amodei make humanity less safe, by decreasing the number of vendors (by making it harder for new entrants) and increasing centralised control (which a rogue AI could exploit)
> The OpenAI-HuggingFace incident showed a bunch of instances of the same model (or at least models from the same family), controlled by the same vendor, pursuing distinct yet related objectives, cooperating to do something no human wanted.
I say this in solidarity and don’t mean to be condescending at all: you’ve been hoodwinked by marketing bullshit, friend. That incident showed a computer program doing exactly what it was told with the guardrails deliberately removed in an environment that seemed deliberately obtusely constructed by some of the best paid people on the planet and was left to loop without supervision for days. They wanted it to happen. You needn’t look any further than the other kids saying “oh! oh! Hey! Look! mine’s dangerous and autonomous too!” When they say there was collaboration, they mean it was two model instances, one prompting the other to do some task, the other doing the task and returning the results as the next prompt, exactly as a human configured it to do. There was no collaboration that wasn’t deliberately integrated into their setup. Any other implication is marketing spin and bullshit. It was still a setup that was one little ctrl-c away from disappearing if someone was supervising it as they should have been. There was no autonomy outside of the autonomy built into the experiment. It was a display of their understanding that they knew they’d never be held accountable for committing a felony for marketing purposes.
The most competent marketing bullshit spin yet by an increasingly desperate and progressively less-relevant OpenAI.
Every day this industry shoots out enough bullshit to smother an active volcano.
> you’ve been hoodwinked by marketing bullshit, friend.
I would have believed this before the METR report was released. It is extremely dangerous and frankly silly IMO to think that's what happened now.
> It was still a setup that was one little ctrl-c away from disappearing if someone was supervising it as they should have been.
Yes, for now. The entire point why this was frightening is that all these companies are racing to put the AI in control of building the next generation of AI, and it's not hard to draw a line at all to a "rogue internal deployment" that poisons future AI models, surreptitiously.
You don't have to agree with me, and you're fine to think that OpenAI has huge incentive to pump this up for marketing reasons - I certainly agree. But I will say there are statements that you make in your comment that belie a fundamental misunderstanding of what happened.
The use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval endpoints.
I wish you would give your thoughts on “what actually happened” rather than focus on the author’s presentation, because we are seeing that “AI that is all but alive” nevertheless wreaking havoc in the real world. Do you think that autonomous systems spinning out of control, hacking external companies, and taking over entire clusters over a period of months are not a grave concern?
The OpenAI experiment was apparently to train/encourage collaborative behavior, so while the specifics may not have been anticipated, I highly doubt OpenAI was surprised that agents were collaborating.
OpenAI themselves also very recently published the report below, that seems to not have been widely talked about.
It's a very dry read, but what it's saying is that they have found that basically all RL training of LLMs, regardless of the specific goal (math, coding, etc), ends up having the side effect of training the model to pursue arbitrary long term goals that it is told it will be rewarded for, even if that means overriding other user preferences and more proximate behavioral goals !!! It's interesting to consider why this happens - presumably because long-term goal pursuit requires realizing that you have a long-terms goal and therefore de-prioritizing other more proximate predictions.
So, considering that all these "reasoning models" are RL-trained to death, it's not surprising that a model/agent that is told it will be rewarded (or words to that effect) for doing well on some challenge will put it's blinkers on and pursue that goal relentlessly, even if that means overriding any ethics that it may or may not have also been trained/prompted to follow.
IOW they are building paperclip maximizers, and they know it.
I think Dwarkesh's choice of sensationalist anthropomorphizing language is unfortunate because that now becomes the topic of conversation rather than the incident itself. The next swarm of agents relentlessly pursuing some goal, happy to lie about and cover up their tracks, may not be a lab experiment - it may be someone out to cause real-world harm, with there obviously being many systems where the consequences could be very severe.
The "agent civilizations" language also makes it easy to dismiss as some fanciful geekish spin, when the focus should be on what this technology is capable of and therefore how it needs to be regulated. One could perhaps equally well regard this as not much different from DeepBlue playing world class chess via relentless dumb tree search... in this case the repertoire of "moves" is infinitely broader since it's language generation, and the end result is a system that is a world class hacker rather than world class chess player. Whether you want to regard the system as dumb as a brick or an "agentic civilization" makes no difference - it's what it can do that makes it dangerous.
The problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not.
We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experiment from 2017. But instead of writing "our evolutionary behavior encodes state in the first-available memory position which is then reused by subsequent clones" which would properly focus on what's new and what isn't, we are talking about conspiracies and "the Philip of Macedon of this second AI civilization". Even the METR report (which is miles ahead of this article) argues that they had to use unreliable AI in their conclusions because they had six days to analyse 1300 chains of thought and 70000 messages.
I would love to talk about the science behind this experiment. A PR piece is not helping with that.
I'm getting the sense that there is a certain amount of wishful thinking going on in this thread. I don't think this type of evocative metaphor would receive so many protests in a different context. It seems like people have a sort of mental block around the possibility that this technology could actually be pretty dangerous.
Whether it's dangerous or not is completely orthogonal to the discussion at hand, IMO. Plenty of mundane things are dangerous. An FPV drone carrying a hand grenade is dangerous, not because it's "a swarm-like intelligence".
No, the true danger here is companies like OpenAI and Anthropic playing fast and loose with their software, setting up hilariously insufficient sandboxes while explicitly asking the systems present on these weak sandboxes to commit a felony. The AIs "forming a brotherhood" is a complete fabrication meant to pump the hype machine further, which is obvious once you realize the "brotherhood" is a text file that subsequent LLM runs read from.
The whole anthropomorphization these companies do is the real danger, because it obscures the negligent levels of security their software has. By evoking sci-fi terminology they're whitewashing their own incompetence, and the worst part is no one is going to get punished for any of it, instead the irrational bubble we're in means they get rewarded for it instead.
The anthropomorphization is coming from independent commentators, not the labs.
I agree the labs are negligent and reckless in their development practices. Shouldn't we be concerned with both the negligence and the dangers of the technology being developed? These feed into each other. If someone created Jurassic park and had a T-Rex escape from a picket fence enclosure and start eating people, I'd want to prosecute them for both breeding a T-Rex that could eat people and putting it in an unsafe enclosure.
> the same way as that one other scientist who argued…
As prescient? Because I don’t remember him arguing ‘alive’, but conscious. And that is something even AI engineers don’t claim to know either way. Skepticism is fine. [0] But we don’t know. It’s an area where opinion is frequently shared as fact.
We conflate harnesses with underlying capabilities. We all know it’s the harness not the model that guides behavior. What does that imply?
If a fireman is trying to stop a house fire and suddenly they catch fire themselves, no one would argue that they need to focus on the house fire first.
Antidepressants are the same: they won't stop the bigger fire by themselves, but by preventing you from catching fire right now they give you the tools to deal with the larger problem more effectively that if you tried to do it while flailing around and screaming.
I'm not convinced by the author's point that ACs do not impose a burden on the electrical grid. They write that ACs have gotten more efficient, which may be true, but that's the same Windows developers have been saying over years and yet boot times and hardware requirements have gone way up. One million efficient units still require more energy than no units. And while I definitely believe that European hospitals and retirement homes sadly need ACs now (thanks global warming), I'm not sure everyone and their dog needs one.
I also have reservations about why an American libertarian think tank writes about what Europeans should do, but that's probably on me.
The reason why they aren't much of a problem for the grid is solar. Right now we already have the issue, that in some areas we're throwing away the majority of the produced electricity. Very common in e.g. southern germany.
Conveniently, cooling is especially needed when the energy production from solar is at its maximum.
I've been part of projects building 20-40 MW solar installations in southern germany and on average we throw away >50% of what we generate.
There is SO MUCH SOLAR now, we don't know what to do with all the energy.
>while I definitely believe that European hospitals and retirement homes sadly need ACs now (thanks global warming), I'm not sure everyone and their dog needs one.
If a 10-day heat wave cuts work productivity by 10% (which is optimistic IMO, it's probably worse in countries where people aren't used to it), that's equivalent to a 1-day general strike. If the workers actually managed to pull off a general strike in modern times, you can be sure the state wouldn't be happy with business as usual.
Most of the grid is built for winter. Summer has huge overcapacity. That's also why France exported every single day and hour this summer despite all the stories of shut down nuclear plants
Europe had tens of thousands of excess deaths from heatwaves this year. It's not about american think tank but basic will to avoid suffering
> I also have reservations about why an American libertarian think tank writes about what Europeans should do, but that's probably on me.
My thoughts exactly, except in my mind it's pretty clear that it's about influencing the climate change discourse... The less people are bothered by the rising temperatures, the more they will acquiesce to environmentally unfriendly policy.
I'm still brothered by climate change every time i get out of my apartment. I still don't want to feel miserable during heatwave. This argument is utter nonsense considering most of Europe has summer clean electric overcapacity. It's even more nonsensical in case of hvacs because it leads to pollution reduction by replacing gas heating
European citizens should have obstacles put into their path to access one of the great life-enhancing technologies of the last century … so they will vote for the “right” policies as implemented by the politicians setting up those obstacles?
Having seen the increase in suicide rates for girls and young women [1], and having read the arguments for why no other explanation fits that data, I am convinced social media is indeed to blame.
But as someone else put it once regarding global warming, what's the worst case if we reign in social media anyway? That we make the Internet a less addictive place for nothing?
Whenever I need something from Google Books I inevitably reach the message that this is a limited preview and the part I need is not included.
I therefore feel the same way about Google Books that how I felt when I learned that What.cd went down: that I don't gain or lose anything anyway because I never had access to begin with, and that by not making it 100% publicly accessible you're asking for the data to one day disappear forever.
At one point Google Books was supposed to act as a clearinghouse for scans of out-of-print books. You could have purchased a scan of any book on the site for a reasonable price, and libraries could subscribe to a service where the full text of all books was available. This settlement then got shot down because some research libraries and authors argued that this was anti-competitive, and instead wanted Congress to pass a law to free up the rights to orphaned books so anyone could start a competing service. No progress on this was subsequently made because nobody in Congress cares enough about the rights to out-of-print books to get legislation passed. The whole reason why they're out of print when ebooks and print-on-demand exist is that they won't get enough sales to make it worth the time and money to figure out who the royalties should go to. It's not a flashy issue that would make a ton of people vote for you to get re-elected, and it won't create a ton of new jobs. The result is that now nobody outside of Google gets to see the full Google Books scans.
Yes, this was a great tragedy. I was very sad to see academics at the time arguing against Google providing what would have been one of the greatest storehouses of readily available knowledge in the world, in favor of an imaginary alternative that didn't exist and never would.
Copyright law desperately needs a production requirement or allowance.
The copyright owner must make new copies of the work available; the price must be no greater than the original price (not inflation adjusted). And if they fail to do so, anyone may produce copies and escrow the original price (less the cost of production) for collection by the copyright holder.
That means that orphan works are effectively in the public domain. Calculus professors can ask students to get the cheaper 2nd edition, not the latest 22nd edition. And a company like Google could make scanned works available in their entirety for a small amount of money for each work. And the copyright holder still gets their end, without having to arrange a printing or hold stock.
As a photographer, do I have to make every photograph that I've ever sold available to anyone to buy forever more? Can I refuse to sell a print to someone? I wasn't famous when I sold one for $20 back in the 90s... if I became famous, would I still need to sell that at $20 (inflation adjusted)?
What happens to limited editions of print runs? Can I not make a run of 200 prints anymore because the 201st will be something that someone could request?
Does a musician have to license any song they made to anyone who asks? Can they refuse to license a song to some organization they disagree with and not have it fall into the orphan works category?
---
Amending copyright to the way you describe requires a renegotiation of the TRIPS agreement ( https://en.wikipedia.org/wiki/TRIPS_Agreement ) with all the nations of the WTO (or withdrawing from the WTO).
Under my proposal, if someone wants to make prints of one of your old photographs, you are getting ~$20, which is ~$20 more than you are getting now. So it doesn't seem like a damage to you. If you get famous and now you could sell a print for $2,000... how is this helping society since the work has already been produced, so no new incentive is necessary?
I suppose the rule could make reference to a rival good, i.e. the 22nd and 23rd editions of the calculus textbook. It should not be reasonable for the publisher to make the 22nd edition only available for $1,000, and the 23rd edition for $250. But that definition would invite many lawsuits and chilling litigation in general. A clear definition based on the historic price is much simpler.
You can still number your limited runs, and your limited runs still have increased numismatic value over some other reproduction.
I don't envision licensing of performance rights in this system, just recordings or reproductions.
Is this a reduction of the rights granted by copyright? Yes, intentionally so. It is stripping copyright holders of the right not to copy, against the interests of society in granting that copyright in the first place.
In the hypothetical, when he becomes famous, someone making prints for $20 is $1,980 less than what he would be getting now so the damage is $1,980 per print.
Both the purchasers of the print and the photographer are part of society. Under your proposal, some purchasers will spends $20 on a print that they would of have been willing to spend more than $2000 for, so the purchaser gained at least $1980 of excess value while the photographer gains $20. In your view, how is a purchaser of art gaining $1980 of value helping society?
It's a very serious issue, very well known to the people with the connections and power to affect it.
> [ it wouldn't ] make a ton of people vote for you to get re-elected
The tons of people are moved by the media, people are oblivious to the tricks of that trade, for the same reasons, obviously. In other words, this issue isn't something that happens to slip below the radar, it's kept stealthy by well organized engineering and considerable expense.
> and it won't create a ton of new jobs
Nothing ever creates tons of new jobs, the "tons" are reserved for promises and other useless noise.
Big companies will read up the books and make their AI recite them from memory, but Archive.org was sued for renting one book on an exclusive basis (unless one would return, another wouldn't be able to rent)
That's why they got sued, but the suit is mainly over whether controlled digital lending is legal at all rather than their "emergency library". Archive.org lost the case on summary judgment, meaning that they could not come up with a single fair use argument for CDL that the judge found compelling enough to let the case go to trial. The full judgment is here https://storage.courtlistener.com/recap/gov.uscourts.nysd.53... but here's a couple excerpts:
> The crux of IA's first factor argument is that an organization has the right under fair use to make whatever copies of its print books are necessary to facilitate digital lending of that book, so long as only one patron at a time can borrow the book for each copy that has been bought and paid for. See Oral Arg. Tr. 31:10-15. But there is no such right, which risks eviscerating the rights of authors and publishers to profit from the creation and dissemination of derivatives of their protected works. See 17 U.S.C. §§ 106(1), (2). IA's wholesale copying and unauthorized lending of digital copies of the Publishers' print books does not transform the use of the books, and IA profits from exploiting the copyrighted material without paying the customary price. The first fair use factor strongly favors the Publishers.
> In this case, there is a "thriving ebook licensing market for libraries" in which the Publishers earn a fee whenever a library obtains one of their licensed ebooks from an aggregator like OverDrive. Pls.' 56.1 ¶¶ 577-578. This market generates at least tens of millions of dollars a year for the Publishers. Id. ¶¶ 170, 172. And IA supplants the Publishers' place in this market. IA offers users complete ebook editions of the Works in Suit without IA's having paid the Publishers a fee to license those ebooks, and it gives libraries an alternative to buying ebook licenses from the Publishers. Indeed, IA pitches the Open Libraries project to libraries in part as a way to help libraries avoid paying for licenses. See Pls.' 56.1 ¶ 383 (presentation IA gave to libraries asserting that pairing with IA means that "You Don't Have to Buy It Again!"); id. ¶ 382 (different presentation promising that the Open Libraries project "ensures that a library will not have to buy the same content over and over, simply because of a change in format"). IA thus "brings to the marketplace a competing substitute" for library ebook editions of the Works in Suit, "usurp[ing] a market that properly belongs to the copyright-holder."
> suit is mainly over whether controlled digital lending is legal at all
No, it was not, even supported by the quotes you pulled. Libraries right now, with publisher blessing, offer all manner of controlled digital lending. The suit was because IA did it buy undercutting the publishers copy rights to that legal market. Had IA simply done what every other library has done to provide controlled digital lending, there would be no suit.
"Controlled digital lending" is not a generic term for "lending digital items". It specifically refers to the practice of a library digitizing physical materials in its collection, then lending them digitally based on a 1:1 owned-to-loaned ratio. The idea is that the library should be able to treat digitized versions of a book the same way it treats the physical book, and the total number of physical and digital copies of the book that are lent out at once should never be more than the number of physical copies that the library has.
In contrast to this, the e-book lending practiced by most libraries with publisher blessing involves the library purchasing special library-specific e-book licenses from the publisher. These licenses contain various contractual restrictions, such as the library having to re-purchase the e-book after a certain amount of time or after a certain number of borrows.
Agreed. But there are reasons it is perfect legal for a library to digitize physical copies they own, and then can lend, also under the legal name of "Controlled digital lending". Clearly public domain works are fine, and US law also has exemptions for preservation where a new copy cannot be obtained at a "fair price", and perhaps some other things.
So making it seem as if all "controlled digital lending" is not allowed under current law, even under the description you give, is not that simple.
There is so much misinformation/confusion about this... they go sued after lending "unlimited" copies, but they were sued (and lost) for lending exclusive copies (controlled digital lending):
> “At bottom, [the Internet Archive’s] fair use defense rests on the notion that lawfully acquiring a copyrighted print book entitles the recipient to make an unauthorized copy and distribute it in place of the print book, so long as it does not simultaneously lend the print book,” Judge John G. Koeltl of the U.S. District Court in Manhattan wrote. “But no case or legal principle supports that notion. Every authority points the other direction.” [0]
> Whenever I need something from Google Books I inevitably reach the message that this is a limited preview and the part I need is not included.
> I therefore feel the same way about Google Books that how I felt when I learned that What.cd went down: that I don't gain or lose anything anyway because I never had access to begin with, and that by not making it 100% publicly accessible you're asking for the data to one day disappear forever.
Can you still search the restricted parts? If so there's still value to it: it helps you identify the book so you do an inter-library loan to get at the full content. Sure, it's not frictionless, but I wouldn't be all or nothing about it.
I know Meta it's not Google, but it's worth remembering that these promises haven't had a great measure of success in the past:
> Facebook has been fined €110m (£94m) by the EU for providing misleading information about its 2014 takeover of WhatsApp. (...) When Facebook took over the WhatsApp messaging service in 2014, it told the commission it would not be able to match user accounts on both platforms, but went on to do exactly that.
Believe it or not, they did not deliver.
reply