I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing. Not general "intelligence" or "reasoning" but fast and effective offense.
Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can check the capability, you can teach offense to learn defense, but it seems like they are literally benchmaxxing it. Why?
OpenAI are like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen. "We need to slow down. Somebody please stop us". The thing that is unaligned here is not the models. Everyone in the story (especially the humans) just keeps doing what they think will get the most reward.
Million? BILLIONS. So I ask myself: are they calling for a pause for the sake of mankind? Or are they proposing to put a throttle on their own gargantuan investment plan ?
I believe these "A.I. Armageddon" warnings are completely self-serving to pump up their share price now that interest rates are rising. Soon, it may become financially unsustainable to finance the A.I. boom and the entire sector will deflate.
Do you really believe that Anthropic or the other A.I. hoods will stop selling A.I. and only make it available to a few government spooks and evaluators? That won't bring in the money to fuel the boom and could lead to share price collapse!!
I'm convinced that a large reason for their apocalyptic warnings is because they're working to build moats for their businesses.
Get enough bills restricting AI passed through legislation, and they can cut off or slow down any newcomers that simply don't have the resources to compete with those that are already entrenched in the industry.
Who would have thought that putting a heap of extra hosts on the internet (in the form of massive new data centres), running a heap of extra clients (in the form of agents), running autonomous workloads could cause problems?
I know this is intended as a joke but there is something to it. The public market has a way of elevating profit over everything else, even if you start with the best intentions. Remember “don’t be evil”?
But not before the second coming claims pentacostal fundamentalist christains!!
Congrats! Certain companies are running with the same fear driven crazy nonsense that blows through the other side time to time. Hell, get together! Maybe ill get a wedding invite. I'll hang on it the wall. It'll give me something to think about.
To be sure the craziness is for $$$$$$$. They want to look like the inevitable next big thing ... which has similarity to the other s... well they want their 10% cut regardless
"potentially causing hundreds of billions of dollars in damage"
The global economy is over $126T this year (ecommerce alone is like $7T) how can something as scary sounding as AI swarm taking over the internet cause so little damage?
IMO it is their FULL responsibility to create an internet just for humans before they go full scale autonomy here with agents that are indistinguishable from humans to most actual humans. It is very dangerous there will be no safe place to obtain information you can just trust. For example, the recent earthquakes in Indonesia. I was scared to share that information with others because it looked like it could be AI and was only being shared by small accounts.
In the US at least, removing safe and trustworthy sources of information is a goal, I can tell this by listening to politicians lambasting media companies and the hostility shown toward reporters.
What it really shows is that Dario is in total panic mode over his business.
Ever it is because it is financially becoming unsustainable for them, they have hard time to raise more money, and the craziness of finance of all of that will be exposed soon.
Or that he is so scared of new open source models that might make his company obsolete in the long run in the sense that he will never be able to return the income expected by investors by fighting cheaper competitors.
You can run opus 4.8 level models on $10k in hardware or less. Just seems like there is little argument to explain why there are 10s of trillions in future profits for these labs.
I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing. Not general "intelligence" or "reasoning" but fast and effective offense.
Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can check the capability, you can teach offense to learn defense, but it seems like they are literally benchmaxxing it. Why?
OpenAI are like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen. "We need to slow down. Somebody please stop us". The thing that is unaligned here is not the models. Everyone in the story (especially the humans) just keeps doing what they think will get the most reward.
They are writing these letters to investors. You can read them all as: Our models are so good it’s scary.
Million? BILLIONS. So I ask myself: are they calling for a pause for the sake of mankind? Or are they proposing to put a throttle on their own gargantuan investment plan ?
I believe these "A.I. Armageddon" warnings are completely self-serving to pump up their share price now that interest rates are rising. Soon, it may become financially unsustainable to finance the A.I. boom and the entire sector will deflate.
Do you really believe that Anthropic or the other A.I. hoods will stop selling A.I. and only make it available to a few government spooks and evaluators? That won't bring in the money to fuel the boom and could lead to share price collapse!!
I'm convinced that a large reason for their apocalyptic warnings is because they're working to build moats for their businesses.
Get enough bills restricting AI passed through legislation, and they can cut off or slow down any newcomers that simply don't have the resources to compete with those that are already entrenched in the industry.
Who would have thought that putting a heap of extra hosts on the internet (in the form of massive new data centres), running a heap of extra clients (in the form of agents), running autonomous workloads could cause problems?
We really have to postpone the IPOs because of security reasons, these LLMs cannot be trusted
I know this is intended as a joke but there is something to it. The public market has a way of elevating profit over everything else, even if you start with the best intentions. Remember “don’t be evil”?
Yes, you have a point here
But not before the second coming claims pentacostal fundamentalist christains!!
Congrats! Certain companies are running with the same fear driven crazy nonsense that blows through the other side time to time. Hell, get together! Maybe ill get a wedding invite. I'll hang on it the wall. It'll give me something to think about.
To be sure the craziness is for $$$$$$$. They want to look like the inevitable next big thing ... which has similarity to the other s... well they want their 10% cut regardless
Don't worry, ~~NetWatch~~ Cloudflare is actively building the Black Wall to save us from all these rogue AIs !
So jail the CEO, anyone else invovlved & shut down the company, right ?
As if AI scrapers haven't already!
"potentially causing hundreds of billions of dollars in damage"
The global economy is over $126T this year (ecommerce alone is like $7T) how can something as scary sounding as AI swarm taking over the internet cause so little damage?
AI companies are making no money so a crisis might divert attention.
So stop building it
They could just unplug the network cable? No internet connection, problem over. IDK, this whole hype seems painfully obtuse.
most claims coming from the frontier models are already partly true. agents have taken over a decent chunk of the internet, it's saddening.
IMO it is their FULL responsibility to create an internet just for humans before they go full scale autonomy here with agents that are indistinguishable from humans to most actual humans. It is very dangerous there will be no safe place to obtain information you can just trust. For example, the recent earthquakes in Indonesia. I was scared to share that information with others because it looked like it could be AI and was only being shared by small accounts.
In the US at least, removing safe and trustworthy sources of information is a goal, I can tell this by listening to politicians lambasting media companies and the hostility shown toward reporters.
It already has. Google search results out front should have told you.
[dupe] Discussion on source: https://news.ycombinator.com/item?id=49672510
What it really shows is that Dario is in total panic mode over his business.
Ever it is because it is financially becoming unsustainable for them, they have hard time to raise more money, and the craziness of finance of all of that will be exposed soon.
Or that he is so scared of new open source models that might make his company obsolete in the long run in the sense that he will never be able to return the income expected by investors by fighting cheaper competitors.
You can run opus 4.8 level models on $10k in hardware or less. Just seems like there is little argument to explain why there are 10s of trillions in future profits for these labs.
He thinks this is a warning. My dude, when you're the one making it, it's a threat.
I hereby Triple Dog Dare him to let that swarm loose. It would be a huge improvement over the wasteland of social media.
Social media would have so much to report by doing that. certainly, would be interesting living in that world
"Mom, uncle Dario is ranting about AI swarms again!"
His handler yanking that chain like it owes money.
Rents due Dario.
“They are just saying that as a tactic. We need to keep accelerating because we will lose if we don’t.”
"If only there was someone could stop me, the CEO of Torment Nexus Inc, from creating the Torment Nexus."
[dead]
- "LLMs might kills us all"
- "Terrorists are hacking claude API to configure drones"
- "LLM swarm might take over the internet"
- "LLMs will take everyone's jobs"