The post focuses on AI social engineering a lab to make something for it, but I think that's the wrong concern. I think the bigger concern with AI is that an individual, or small group of individuals, uses AI as a really useful assistant to learn how to develop deadly viruses on a DIY level.
I'm pretty bearish on the idea of AI becoming Skynet from Terminator 2, but I do think there's some credibility to the bioterrorism argument. When I used to talk with people that work in orgs tasked with preventing terrorism, this was the thing that was keeping them up at night, and this was before AI.
I remember being told that coming up with the alloy for the oxygen side pre-burner in Spacex's Raptor engine was a tedious trial and error grind over many days and nights and not a euphoric 'a ha!' moment. I wonder if AI could help with these kinds of problems, if it can be simulated in software then let AI do the grinding.
/the oxygen side pre-burner in a full flow staged combustion rocket engine is a nasty place and figuring out a material that will survive in that environment was the key (or one of the keys) to making Raptor a reality.
Is the term "One-shotting" here even appropriate? In the context of AI one-shotting means giving your llm a single prompt and having it return a surprisingly good result (usually code related) without requiring further input, but the llm (probably) still iterates heavily before showing the results. When Codex "one-shots" something it depends a ton on the fact that code is easily testable.
Lately "one-shotting" means "without human steering / feedback"
I think its a consequence of the improvements in harnesses and LLMs of the last year. Before they had a hard time self-reviewing the results and proceeding to a second iteration, now they are quite good at it although not perfect.
You can use AI to brute-force different crystal configurations, run some MD/DFT and identify candidates. I'd be (not completely, but) shocked if there wasn't a research group doing this already. To rapidly iterate with physical samples is still beyond our capabilities AFAIK.
The issue is, you kind of have to not get wrecked by the negative impacts to collect those positive impacts.
Unfortunately, "AI that's competent at bioweapon R&D" might be an easier mark to hit than "AI that can put a dent in the aging process". And biorisks can have a very favorable "resource input -> suffering output" rate of returns.
Cutting sugars, not smoking, regular checkups, watching your weight, managing stress, and getting enough exercise and sleep still trumps any other therapeutic-based life extension approach. But I know, I know, it's actual effort, and technology is about making things easier so it's a non-starter.
But I do agree that if we're on the verge* of von Neumann replicators and getting replaced by Pokhumans** to cheer on the ASI, then curing cancer, immortality and cryo-suspension are likely trivial achievements in comparison.
*Nope, not even close, but don't tell that to the ASI cultists, that's like questioning the veracity of the gospels now.
**Yes, that's the strong claim in AI 2027. And the democratic party apparently believes it.
Has anyone in the rationalist, doomer community actually seriously engaged the real biology/virology of a “virus that wipes out humanity”? Playing Plague Inc doesn’t count.
I mean, we’re talking about real physical things, not first principles thought experiments here. Thus, they have limits imposed by physics and biology. Limits on transmission rate, lethality, particle size, evolutionary change rates, and so on. All of these interact and define a design space that nature has been searching for at unimaginable scale for as long as humans have existed. And yet here we are! Seems like “wiping humanity” might be really hard, perhaps impossible for a virus.
Perhaps I’m being uncharitable or insufficiently creative, but these people strike me as deeply unserious in their engagement with the actually real physical world of biology.
EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?
But a virus's "goal" would not be to wipe out a species, that would be a dead end. Nature has explored flight on an unimaginable scale, yet our machines fly better (at least in most aspects) than birds, some even go to space.
Yes, but also viruses do not actually have goals as such - maybe 'replicate' could be their only goal. So a virus _could_ come up with a mutation that kills all its hosts, it just dies out as well in the process. Not very reassuring for the host!
As a slight aside - since I was talking about the weakness of biological analogy on HN recently - the way humans fly is very very different to how (most) birds do. If nanomachines actually could be built by humans (non-biological nanomachines) they might be as different to virus/cells/organisms as plane wings are to bird wings.
Of course, also please do not prompt a frontier model to make nanomachines, either, thankyou.
I doubt our flying machines are anywhere near as energy efficient as birds. Our flying machines combust jet fuel, birds can fly by ATP. Not to mention earth-destructing.
And going into space is pointless why would any animal ever develop such a facility :)
> I doubt our flying machines are anywhere near as energy efficient as birds
This kind of nerd sniped me, so I looked into it and turns out no, planes are more efficient than birds!
You have to look at the energy spent for moving a fixed mass over a fixed distance and for the most efficient birds I looked into, that’s about 300g of fat burned for 300g of “dry weight” for a trip of ~11,000km. (bar-tailed godwits)
For a plane that’s roughly 80 tonnes of kerosene for 170 tonnes dry weight on a similar distance (787). Energy density of fat and kerosene are actually pretty close so you can just look at the kg of fuel burned per kg of payload for the trip. For a bird that’s 1kg of fuel per kg transported and for the plane that’s ~0.5kg per kg. So roughly twice as efficient.
So from a pure energy point of view it’d be more efficient to put all the migratory birds in a 787 than letting them fly on their own!
Fascinating! Though how much of that would be due to a birds baseline metabolism? Wouldn’t be a fair comparison if that includes the birds basic energy costs of living.
To be completely fair to the birds, I think a lot of this comes from the fact that what matters for flying efficiency gets better as you size up thanks to the square-cube law: the drag is reduced and the engine efficiency increases.
You could not scale down a plane to the size of a bird and match their efficiency. On the other hand you also couldn’t scale a bird to the size of a plane so I guess we can call it even!
Biology is very messy and very data constrained. We think that human anatomy works one way- but surgery and radiology are hard because every human is very different.
Missing organs, odd tissue performing unclear functions, different, new or missing muscles.
It’s unclear whether a world ending virus is even possible. There might be better luck with prions, or fungi.
yes - I don't want to publicize what I know. I worked briefly at a leading university bio lab. It's haunted me for awhile that it would only take 1 disgruntled postdoc to wipe out humanity. There are {AI things} now that will tell you how.
The obvious argument is that natural organisms are restricted in ways that artificial ones are not (e.g. natural viruses are not selected to kill their host too quickly, natural selection can only reach solutions which can be reached incrementally).
>EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?
This does not follow automatically. E.g. it is enormously easier to create an ICBM than to create something capable of intercepting an ICBM. Some technologies naturally benefit an attacker and some naturally benefit a defender.
An AI doesn't need to kill even nearly 100% of humanity in order to render us a non-threat.
Also there are plenty of doomer scenarios along the lines of e.g. some ethnosupremacist cult making a virus that kills everyone outside of their group.
One of the scariest science fiction stories I've ever read is "The White Plague" by Frank Herbert... yeah, the Dune guy. In this story a molecular biologist's family is murdered in a terrorist event, goes insane, and decides to create a plague that is carried by men but kills women, so that everyone else has to experience the loss he did. It's not scary because the story is any more particularly filled with death or destruction or gore or other viscerally scary things like that. It was scary because as I was reading it, there was no hole in the logic I could take refuge in. There's simply no reason that book can't happen.
And that was my reaction 15 or 20 years ago. It is only harder to poke holes in the logic now than it was then.
Uninformed speculation from the rationalist cult forum that wants to will science fiction into reality.
If you actually look at the trajectory of post-LLM AI developments, the successes have all been in areas where the major labs have some way to get their hands on s**tons of correlated data. That's why they're pushing into maths now. I see no reason to assume that will be replicated with biological science data, which is extremely proprietary and not required to be exposed as part of the end product. It doesn't even have an equivalent to a black market in piracy like books or movies did. This data was not valuable to the public before now.
We're saying the same thing. The ability of the system to iterate is dependent on the digitizability of its artifacts, and its familiarity with such artifacts through its training data.
Not only do we have to live with endless LLM slop now, it seems we also need to accept an endless sequence of 2-cent philosophical blog posts where people describe the "world with AI" based on two minutes of meandering thought.
Hah, well, you must have never encountered LessWrong before. It has been like that for well over a decade, in fact it's probably approaching its second decade...
One thing you've got to give them credit for is that they've been talking about AI -- about the present moment, really -- from the beginning. Nick Bostrom, who wrote "Superintelligence" about ten years ago, has always been in their orbit.
It started in 2006, although it didn’t have its own site until 2010-ish.
I’m not sure it will ever be given its dues for its prescience. The whole rationality project was started with the explicit aim of upgrading participants’ reasoning abilities so that when AI came along for real, there would be people capable of thinking it through without making all the usual mistakes humans make when trying to reason about things.
In the discussions I read today, everyone involved from commenters up to CEOs are making all and exactly those mistakes, meaning it is in my opinion perhaps the greatest called shot of all time.
He’s right though, this post says nothing non-obvious. It boils down to “AIs that can do some difficult things might be able to do other difficult things. Also: Danger!”.
On the contrary this couple paragraphs is aimed strictly at 'normies' as in people who are riding the current exponential with little history of thinking about this moment. Most people's beliefs and knowledge needs updating and patching without hoping that 'it obviously follows from assumptions' because most people don't bother to think about the assumptions in the first place.
IOW:
> There isn’t enough here for it to be interesting.
True - for you and the dozens of folks like you, to paraphrase a meme.
is the horsepower behind ai speculation less than, greater than or equal to human speculation? seems they’re recursively related, though human speculation has more torque since it occurs naturally or at least as spontaneously as it takes for an epiphany instance to occur.
and then transposed into an ai chat prompt to get the analytical momentum for some grand scheme that gets published to whatever extent ie a comment, blog post, or whatever textual content that then gets ingested as training data…llm rinse repeat. not sure where this is going. is anyone?
Yes, it’s also the way human society processes new stuff, be it novel technologies and the philosophical ramifications or other topics. Call it distributed Bayesian processing.
Sure each 2-cent philosophical posts isn’t worth much on its own but add up many of them over time and we get something more than its parts.
I guess you haven’t yet read Amodei’s essays? There is a significant overlap between the early SF AI scene and LessWrong contributors.
That comes directly from the top, look at Anthropic leadership
Grifters don’t usually spend their entire lives devoted to preparing for something that doesn’t yet exist, giving up whatever else they like doing to work on that problem, and spending years trying to plan out how to make things go well for everyone when that thing materializes.
Grifters are opportunistic short-termists. Whatever you can say about these rationalists, they aren’t that.
Eliezer spent his life devoted to science fiction and building a cult around himself, and now him and the rest of them are trying to vindicate themselves before they fade into obscurity.
> if you can do the worst “bad” thing possible, you can also do the best “good” thing possible
You say this like it's a truism, but I don't think it is. The bad things may be categorically easier than the good things. It's easier to destabilize a delicate system than to stabilize it. It's easier to break something than to fix it.
The post focuses on AI social engineering a lab to make something for it, but I think that's the wrong concern. I think the bigger concern with AI is that an individual, or small group of individuals, uses AI as a really useful assistant to learn how to develop deadly viruses on a DIY level.
I'm pretty bearish on the idea of AI becoming Skynet from Terminator 2, but I do think there's some credibility to the bioterrorism argument. When I used to talk with people that work in orgs tasked with preventing terrorism, this was the thing that was keeping them up at night, and this was before AI.
You don't need an AI to tell you where to buy castor beans or their seeds.
I remember being told that coming up with the alloy for the oxygen side pre-burner in Spacex's Raptor engine was a tedious trial and error grind over many days and nights and not a euphoric 'a ha!' moment. I wonder if AI could help with these kinds of problems, if it can be simulated in software then let AI do the grinding.
/the oxygen side pre-burner in a full flow staged combustion rocket engine is a nasty place and figuring out a material that will survive in that environment was the key (or one of the keys) to making Raptor a reality.
Is the term "One-shotting" here even appropriate? In the context of AI one-shotting means giving your llm a single prompt and having it return a surprisingly good result (usually code related) without requiring further input, but the llm (probably) still iterates heavily before showing the results. When Codex "one-shots" something it depends a ton on the fact that code is easily testable.
Lately "one-shotting" means "without human steering / feedback"
I think its a consequence of the improvements in harnesses and LLMs of the last year. Before they had a hard time self-reviewing the results and proceeding to a second iteration, now they are quite good at it although not perfect.
You can use AI to brute-force different crystal configurations, run some MD/DFT and identify candidates. I'd be (not completely, but) shocked if there wasn't a research group doing this already. To rapidly iterate with physical samples is still beyond our capabilities AFAIK.
The positive impact on human aging/lifespan/healthspan is not being talked about enough when it comes to AI+biosciences.
Have we already seen that impact? Or is it speculative? Because the speculative part is discussed all the time
The issue is, you kind of have to not get wrecked by the negative impacts to collect those positive impacts.
Unfortunately, "AI that's competent at bioweapon R&D" might be an easier mark to hit than "AI that can put a dent in the aging process". And biorisks can have a very favorable "resource input -> suffering output" rate of returns.
Cutting sugars, not smoking, regular checkups, watching your weight, managing stress, and getting enough exercise and sleep still trumps any other therapeutic-based life extension approach. But I know, I know, it's actual effort, and technology is about making things easier so it's a non-starter.
But I do agree that if we're on the verge* of von Neumann replicators and getting replaced by Pokhumans** to cheer on the ASI, then curing cancer, immortality and cryo-suspension are likely trivial achievements in comparison.
*Nope, not even close, but don't tell that to the ASI cultists, that's like questioning the veracity of the gospels now.
**Yes, that's the strong claim in AI 2027. And the democratic party apparently believes it.
Has anyone in the rationalist, doomer community actually seriously engaged the real biology/virology of a “virus that wipes out humanity”? Playing Plague Inc doesn’t count.
I mean, we’re talking about real physical things, not first principles thought experiments here. Thus, they have limits imposed by physics and biology. Limits on transmission rate, lethality, particle size, evolutionary change rates, and so on. All of these interact and define a design space that nature has been searching for at unimaginable scale for as long as humans have existed. And yet here we are! Seems like “wiping humanity” might be really hard, perhaps impossible for a virus.
Perhaps I’m being uncharitable or insufficiently creative, but these people strike me as deeply unserious in their engagement with the actually real physical world of biology.
EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?
But a virus's "goal" would not be to wipe out a species, that would be a dead end. Nature has explored flight on an unimaginable scale, yet our machines fly better (at least in most aspects) than birds, some even go to space.
Yes, but also viruses do not actually have goals as such - maybe 'replicate' could be their only goal. So a virus _could_ come up with a mutation that kills all its hosts, it just dies out as well in the process. Not very reassuring for the host!
As a slight aside - since I was talking about the weakness of biological analogy on HN recently - the way humans fly is very very different to how (most) birds do. If nanomachines actually could be built by humans (non-biological nanomachines) they might be as different to virus/cells/organisms as plane wings are to bird wings.
Of course, also please do not prompt a frontier model to make nanomachines, either, thankyou.
I doubt our flying machines are anywhere near as energy efficient as birds. Our flying machines combust jet fuel, birds can fly by ATP. Not to mention earth-destructing.
And going into space is pointless why would any animal ever develop such a facility :)
> I doubt our flying machines are anywhere near as energy efficient as birds
This kind of nerd sniped me, so I looked into it and turns out no, planes are more efficient than birds!
You have to look at the energy spent for moving a fixed mass over a fixed distance and for the most efficient birds I looked into, that’s about 300g of fat burned for 300g of “dry weight” for a trip of ~11,000km. (bar-tailed godwits)
For a plane that’s roughly 80 tonnes of kerosene for 170 tonnes dry weight on a similar distance (787). Energy density of fat and kerosene are actually pretty close so you can just look at the kg of fuel burned per kg of payload for the trip. For a bird that’s 1kg of fuel per kg transported and for the plane that’s ~0.5kg per kg. So roughly twice as efficient.
So from a pure energy point of view it’d be more efficient to put all the migratory birds in a 787 than letting them fly on their own!
Fascinating! Though how much of that would be due to a birds baseline metabolism? Wouldn’t be a fair comparison if that includes the birds basic energy costs of living.
Did some quick maths, it’s roughly 10% for the bird (2W, approx. 4g/day of fat, over 10 days) so not negligible but doesn’t change the full story.
And you could argue the plane also has a baseline metabolism ;)
Haha thanks for the check, I am, tbh, pretty surprised. It's within the same order of magnitude, which is interesting too.
To be completely fair to the birds, I think a lot of this comes from the fact that what matters for flying efficiency gets better as you size up thanks to the square-cube law: the drag is reduced and the engine efficiency increases.
You could not scale down a plane to the size of a bird and match their efficiency. On the other hand you also couldn’t scale a bird to the size of a plane so I guess we can call it even!
Biology is very messy and very data constrained. We think that human anatomy works one way- but surgery and radiology are hard because every human is very different.
Missing organs, odd tissue performing unclear functions, different, new or missing muscles.
It’s unclear whether a world ending virus is even possible. There might be better luck with prions, or fungi.
yes - I don't want to publicize what I know. I worked briefly at a leading university bio lab. It's haunted me for awhile that it would only take 1 disgruntled postdoc to wipe out humanity. There are {AI things} now that will tell you how.
I currently work in a chemical biology lab and I don’t think one disgruntled postdoc could wipe out humanity.
Also disgruntled is more or less unnecessary when combined with postdoc. Sort of goes without saying.
.
Are you asking for humanity to be wiped out so you can win an argument on the internet?
I'm going to remove specific reference because I think even being directed to the right thing is dangerous. You are right.
Was that also the case before AI? Or is it novel?
The obvious argument is that natural organisms are restricted in ways that artificial ones are not (e.g. natural viruses are not selected to kill their host too quickly, natural selection can only reach solutions which can be reached incrementally).
>EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?
This does not follow automatically. E.g. it is enormously easier to create an ICBM than to create something capable of intercepting an ICBM. Some technologies naturally benefit an attacker and some naturally benefit a defender.
Same idiots think folks are at risk from firearms printed from a desktop PLA 3d printer
An AI doesn't need to kill even nearly 100% of humanity in order to render us a non-threat.
Also there are plenty of doomer scenarios along the lines of e.g. some ethnosupremacist cult making a virus that kills everyone outside of their group.
One of the scariest science fiction stories I've ever read is "The White Plague" by Frank Herbert... yeah, the Dune guy. In this story a molecular biologist's family is murdered in a terrorist event, goes insane, and decides to create a plague that is carried by men but kills women, so that everyone else has to experience the loss he did. It's not scary because the story is any more particularly filled with death or destruction or gore or other viscerally scary things like that. It was scary because as I was reading it, there was no hole in the logic I could take refuge in. There's simply no reason that book can't happen.
And that was my reaction 15 or 20 years ago. It is only harder to poke holes in the logic now than it was then.
Uninformed speculation from the rationalist cult forum that wants to will science fiction into reality.
If you actually look at the trajectory of post-LLM AI developments, the successes have all been in areas where the major labs have some way to get their hands on s**tons of correlated data. That's why they're pushing into maths now. I see no reason to assume that will be replicated with biological science data, which is extremely proprietary and not required to be exposed as part of the end product. It doesn't even have an equivalent to a black market in piracy like books or movies did. This data was not valuable to the public before now.
The successes are in areas where you can get AI to execute an iterative loop autonomously. Try, fail, refine, retry - succeed eventually.
Coding is, inherently, quite amenable to that. Is bioscience?
They're certainly working on making it so:
https://www.anthropic.com/news/model-hardware-standard-resea...
We're saying the same thing. The ability of the system to iterate is dependent on the digitizability of its artifacts, and its familiarity with such artifacts through its training data.
Not only do we have to live with endless LLM slop now, it seems we also need to accept an endless sequence of 2-cent philosophical blog posts where people describe the "world with AI" based on two minutes of meandering thought.
Hah, well, you must have never encountered LessWrong before. It has been like that for well over a decade, in fact it's probably approaching its second decade...
One thing you've got to give them credit for is that they've been talking about AI -- about the present moment, really -- from the beginning. Nick Bostrom, who wrote "Superintelligence" about ten years ago, has always been in their orbit.
It started in 2006, although it didn’t have its own site until 2010-ish.
I’m not sure it will ever be given its dues for its prescience. The whole rationality project was started with the explicit aim of upgrading participants’ reasoning abilities so that when AI came along for real, there would be people capable of thinking it through without making all the usual mistakes humans make when trying to reason about things.
In the discussions I read today, everyone involved from commenters up to CEOs are making all and exactly those mistakes, meaning it is in my opinion perhaps the greatest called shot of all time.
He’s right though, this post says nothing non-obvious. It boils down to “AIs that can do some difficult things might be able to do other difficult things. Also: Danger!”.
There isn’t enough here for it to be interesting.
On the contrary this couple paragraphs is aimed strictly at 'normies' as in people who are riding the current exponential with little history of thinking about this moment. Most people's beliefs and knowledge needs updating and patching without hoping that 'it obviously follows from assumptions' because most people don't bother to think about the assumptions in the first place.
IOW:
> There isn’t enough here for it to be interesting.
True - for you and the dozens of folks like you, to paraphrase a meme.
I don’t think it’s good at doing that either.
is the horsepower behind ai speculation less than, greater than or equal to human speculation? seems they’re recursively related, though human speculation has more torque since it occurs naturally or at least as spontaneously as it takes for an epiphany instance to occur.
and then transposed into an ai chat prompt to get the analytical momentum for some grand scheme that gets published to whatever extent ie a comment, blog post, or whatever textual content that then gets ingested as training data…llm rinse repeat. not sure where this is going. is anyone?
[dead]
Yes, it’s also the way human society processes new stuff, be it novel technologies and the philosophical ramifications or other topics. Call it distributed Bayesian processing.
Sure each 2-cent philosophical posts isn’t worth much on its own but add up many of them over time and we get something more than its parts.
I guess you haven’t yet read Amodei’s essays? There is a significant overlap between the early SF AI scene and LessWrong contributors. That comes directly from the top, look at Anthropic leadership
Well with each new thing we get the grifters, lesswrong is an AI grifter hangout.
Grifters don’t usually spend their entire lives devoted to preparing for something that doesn’t yet exist, giving up whatever else they like doing to work on that problem, and spending years trying to plan out how to make things go well for everyone when that thing materializes.
Grifters are opportunistic short-termists. Whatever you can say about these rationalists, they aren’t that.
Eliezer spent his life devoted to science fiction and building a cult around himself, and now him and the rest of them are trying to vindicate themselves before they fade into obscurity.
They are true believers. Terrible philosophers but true AI believers
if you can do the worst “bad” thing possible, you can also do the best “good” thing possible
and then what?
was the needle move at all?
> if you can do the worst “bad” thing possible, you can also do the best “good” thing possible
You say this like it's a truism, but I don't think it is. The bad things may be categorically easier than the good things. It's easier to destabilize a delicate system than to stabilize it. It's easier to break something than to fix it.
Suppose it matters what order you do it in.
The best good thing possible for humans who are all suffering from an extremely painful, incurable disease might just be ending that misery.
The worst bad thing possible in a perfect utopia might be a minor inconvenience comparatively.
[dead]