> As my regular readers will be aware, I have been engaged, on behalf of my clients, in legal combat with Internet censors around the world, agencies who think that they have both the standing, the competence, and the right to tell American companies what software they can write and run, for the better part of 18 months.
Is this supposed to make me trust their opinions on tech regulation?
> Nobody knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
If any past apocalyptic prediction had been true, you would not have been around to write this sentence and we would not have been around to read it. It’s not a very convincing argument when we cannot in principle observe the counterfactual.
You're arguing in favor of a teapot circling Mars. We can't prove it's not there, but telling me that "nobody has ever found it" is an invalid argument is obviously absurd.
In that quote, the author is telling you that all of your past predictions of teapots have been wrong (which is 100% true -- the ridiculous doom has been going on since at least GPT-2. remember how that was "too dangerous to release"?) Show me why I should believe in this one.
This is a grevious misuse of Russell's Teapot. You realise your assertion, if followed to its logical conclusion, is that because civilization has existed for this long, it can never collapse no matter what.
> In that quote, the author is telling you that all of your past predictions of teapots have been wrong
Past predictions have zero influence on future ones. If you flip a coin 10 times and get heads every time, the probability of your next coin flip coming up heads is 50%, not 100%, because it is independent of any prior trials.
Moreover, if Alice and Bob each make 100 predictions and are 100% wrong, but Charles makes 10 predictions and is 100% correct, Alice and Bob's predictions have zero bearing on whether you should trust Charles' predictions. They are not even a factor to be considered.
I don't think I even disagree with you in the broad picture. I am not concerned about language models. But you have committed three blatant logical fallacies in a row. You literally could not make a worse case for your position if you tried, I suspect.
You're making a ridiculously improbable prediction that can never be falsified -- akin to beliefs in a vengeful God. Russell's Teapot is exactly the correct metaphor.
The philosophical burden of proof lies on the person making empirically unfalsifiable claims. That is the point of Russell's Teapot.
> The philosophical burden of proof lies on the person making empirically unfalsifiable claims.
You appear not to understand what a prediction is. "I believe X will likely happen because Y" is different from an assertion of fact that "X will happen". A prediction cannot even be empirically proved! You are simply suggesting that nobody is allowed to make a prediction, ever. Russell's Teapot, too, is not an argument that you cannot believe in God. It is an argument that the unfalsifiability of a claim is not a basis to assert that God exists.
This is the civilization-scale Quantum Immortality argument... no civilization can be around that witnessed the end of civilization.
(I refer to it that way rather than a "variant" because Quantum Immortality should always be parameterized by an acceptance function... for any given acceptance function, that function will be preserved in the quantum multiverse, and scoping the precision of the function affects the percentage of universes that retain it rather than whether it will be retained. There's actually nothing in that definition that requires that the acceptance function describe a "conscious system", you can stick something that accepts a "civilization" in there just as easily. Or "this rock over here". Or any other thing you like. Conscious systems are just a particularly interesting case to us conscious systems.)
It's not a vacuous argument, though it is a funky one with very odd characteristics. But if you accept the suppositions underneath it, it does follow and is not vacuous. The suppositions are certainly not completely obviously true on their own terms, though.
I never really understood what is the explanation for ... old people that we notice as dying. In today's times you might have an argument that some kind of life extension could pop up and some of the people alive today could have that as an excuse.
But like 100 years ago? How would people dying of old age "continue living"?
From the point of view of the old people, exponentially large injections of improbability.
This is also where it is important to understand the argument is parameterized. If your acceptance function is "a full human body living a full human life in the way I'm used to", then you're asking for a huge whackload of improbability to keep that body alive. If, on the other hand, you merely ask for "the minimum system that stays conscious", while that is still a huge whackload it is exponentially smaller, by a very substantial factor.
Anyone who takes any sort of comfort in the argument doesn't understand it. It preserves minimal consciousness, for any and all definitions of "minimal consciousness". It does also preserve bodies, but at an exponentially smaller probability. You can even use as your acceptance function "civilization exactly as we know it today, with everyone currently living still alive in 1000 years" and that comes at a yet smaller exponential probability. All of these "exponential probabilities" as separated by numbers of zeros that themselves require exponential notation to write. But the universe itself is making so many splits and branches all the time that it has the "budget" for these probabilities to be true, somewhere in the quantum multiverse.
Or, put another way, any event that you can draw a non-zero probability pathway for occurs. It is not impossible for me to spontaneously have my particular genetic defect reversed in every cell in my body, so there are branches where that happens... they're just exponentially unlikely compared to what we actually experience. It can literally be in %10^(-10^1,000,000,000) of the universes... but if it's not zero, it will happen, if the quantum multiverse is the correct view.
Quantum immortality is just an absurd quasi-religious pretense of physics that caters to those that fear death but feel embarrassed to participate in the traditional socially approved coping mechanisms.
I do not think death exists as a thing. We're shoving a lot of stuff into the concept. Pain, loss of information, of the one who dies. Fear of the unknown, what happens next. Death is a very volatile concept if you really try to pin it down. When you take all the layers off...you are left with nothing really.
But yes, the apparent loss of information seems to be true, for us at this technological level.
I mean sure, in the end you can have some heaven, some benevolent alien species reaching back into some field and copying all of your material structure data right at time of death and rebuilding you in some kind of resurection center. The space of what is possible is wildly vast, we know or can think of very little by comparison. If you give up some day to day coherence then you can almost always find something that would work and make sense once there.
It could even be us, future us, being able to somehow recover past data and literally reassemble all of the humans that ever lived. They aren't as many as you'd think, in the grand scheme of things.
As I said, anyone who takes comfort in the hypothesis doesn't understand it. The dominant probability has nothing to do with benevolent aliens reaching down and inevitably resurrecting you to youth and health. It is you, minimally conscious, with no body, no ability to move or sense the outside world, nothing more than the absolute bare minimum to be "conscious", just barely conscious and never able to stop being minimally conscious, with literally every last extraneous and unnecessary subatomic particle stripped away (so anything like "sensors" or "effectors" being long gone), for whatever your definition of "minimally conscious" is. This outcome dominates the probabilities that even remotely resembles your current existence by those 10^(10^1,000,000,000) sorts of probabilities.
Everyone should very much hope that quantum immortality is false. It is the very opposite of comforting.
I don't understand. Why does my current life win every instant atm? Why is it so much greater than your proposition? Why don't we all instantly collapse to what you described?
That would require a very improbable sequence of events for it to happen right this second.
Moreover, if the theory is true, it is happening all around us, all the time. Every consciousness that has ever existed, human or otherwise, is still in existence in some branch of the multiverse.
I think whenever we reach such ludicrous conclusions it's more likely we have made at least one wrong assumption, if not more.
And people seem to be captivated by these memes, because the fear is great. It similar to eternal damnation in hell meme. Just look for how long it worked, and it's still working. The logic is simple, it's too dangerous to ignore such a threat, even with seemingly very small probabilities. But that still doesn't make it possible.
I’m not sure what logic you have imputed to my comment, but I’m having trouble connecting what you wrote to anything I was saying. I am criticising the argument “every past apocalyptic prediction was wrong, therefore this future apocalyptic prediction is wrong” (specifically, that it carries exactly 0 weight, because the counterfactual world where at least one past apocalyptic prediction was true, is also a world that does not contain this argument - so there is no world where the argument’s precondition can, even in principle, be false. Arguments are a little bit like rocketships in that they need “reaction mass” - there has to be some amount of false for the truth to push against in order to move our opinion.)
> I am criticising the argument “every past apocalyptic prediction was wrong, therefore this future apocalyptic prediction is wrong”
That is not the argument being made. That is the straw man version you are presenting. Read the full paragraph, and the one after, and it's obvious:
> There are myriad policy responses. Whatever we do, those responses will require popular consent and careful deliberation. No one person, or one company, or one movement, knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
> History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.
“History is a guide to apocalyptic predictions in only one way: all previous apocalyptic predictions have been wrong”
“History is a guide to state abuses of power in many ways, and most of them point in the same direction, that it is a very bad idea to invest powers like these with the state”
These are two separate arguments. I am sympathetic to the latter and endorse it essentially as written, which is why I didn’t criticise it. My criticism of the former is not a metonymic dismissal of every other argument in the piece.
I suspect they have not all been wrong for some people, e.g. native peoples at various times prophesied I’m sure about various technological marvels their more advanced colonizers/enemies/etc.
But warnings (e.g. not apocalyptic but damaging or having any effects) have come true, by and large; people warned about the effects of pollution / co2 emissions / global warming over 150 years ago [0], to name a popular example.
I don't think there are many truly human created apocalyptic events out there besides global thermonuclear war, and I don't think the worst-case AI superintelligent killer robot scenario would be apocalyptic. Unless it somehow triggers global thermonuclear war, but then it becomes a chicken-or-egg question (and it'll probably be humans pushing the proverbial button, as nukes are '60s tech and (hopefully) airgapped etc).
I'm not even sure that the author is correct. The industrial revolution was pretty apocalyptic for the people forced to work in dangerous factories for minimal pay.
Apocalyptic predictions normally get everyone into gear to prevent the threat. Not a great argument for "let's see what happens if we just sit around this time"
Perhaps another way to look at it is to say that “every past apocalyptic prediction was false” is tautologically true?
A statement can be a trivial tautology, as in “P or not-P”. Or it can be a more complicated form of tautology, such as a “self-referential” tautology, as in “This statement is true”. The anthropic principle is then just a name for the claim that there is also another complicated form of tautology alongside “self-referential”, one that we might call “indexical”.
Plus, plenty of negative predictions turned out to be entirely true. And then it took a lot of time and effort to counteract them.
It is not like history was composed of happiness and roses only. Some of the really really bad stuff happened within last 100 years and a lot of that bad stuff was enabled and made possible by technology.
Like it's fine to say this but the worst exterminations of human beings weren't enabled by anything much kre sophisticated then the pointy stick, or at most, the railroad.
You can't AC your way out of climate change anyway: they make heat. In sparsely populated areas that might be okay, but in density, you're never far from the hot exhaust of an AC unit; they exacerbate the problem.
Why would the limit of T1 devices matter when there are ones that already can do hotter? Thermodynamics make things harder to remove heat but not sure how they forbid it (you need the hot side to be hotter than outside to pump heat). Might need stages, for example. We can cool things 200K and more below ambient.
> What we're going to do when that tech will cease to cool us?
Move. This is already happening, with water shortages forcing people to move. Climate change will (likely) cause famines and many deaths, but the bigger effect will be mass migration.
This isn't a new question: quis custodiet ipsos custodes?
In general, the way things work in the USA is that megacorporations write the regulatory frameworks for their own industries. These regulations are typically burdensome enough to be shouldered by the megacorporations while crushing the smaller players in the given market(s). The AI companies are now advocating for this very arrangement. They will likely get they want, and with that they will gain legal shielding for bad behavior just like Dupont, Monsanto, and so many others.
I think there is a more nuanced perspective to take on this debate than just profit motive. If they believe that this version of "AI" is actually becoming so dangerous that there is a real risk of it falling into the wrong hands, then it makes sense that the American labs would try to convince the US government to work on a joint "AI non-proliferation" agreement with China much the same way that nuclear weapon technology came to be restricted by the ones that already had the capability.
This would achieve their regulatory goal of capturing the market, but also the geopolitical goals of USA and China to essentially "techno-colonize" the rest of the world between themselves.
So I think there is definitely a profit motive and a risk of seriously diminishing returns for the actual labs, but there is also a nation-state motive to start pulling up the ladder.
I think in all of these debates people lose track of one of the major vectors of threat, as you said, other humans. From my point of view this is an ever bigger problem with this tech, like orders of magnitude bigger.
The current setup we find ourselves in is based on a very important detail that is easily lost because it is so old we don't really think about it. There is, or rather was, an incentive for more people to exist, because they are valuable for work and defense.
Some people wrongly use the industrial revolution to argue against AI completely replacing human work, but that is very wrong. What is happening with AI has nothing to do with the industrial revolution, as equivalence. Back then you'd just free up humans for higher level work. Now, theoretically, anything you could ever think a human could do, AI would theoretically do better. So no matter what job you come up with, using AI instead of a human gives a better outcome, cheaper, more reliable, faster.
Coming back to initial subject, one thing that kept everyone in check was our inability of actually wiping everyone out. Sure, we had nukes but they ruin the Earth so nobody can inhabit it if they all go off. But with AI, centralizing so much power up top, and taking it away from regular people...that is the biggest danger in this scenario, by far. Not some rogue AI, rather rogue human groups that have the power and technology to do away with basically everyone else, or a good bit of everyone else.
This threat is way bigger and never addressed. Like...suspiciously never addressed.
Not sure it needs to be about publications and data analysis technology as such to regulate AI development, for example - could be about processes, procedures, people, etc., no?
This is quite illustrative. Warnings of a nuclear apocalypse were extremely widespread, known, and feared. The consequences were, indeed, so well known that the world was saved from the worst possible outcomes in the past by regular enlisted men and women who realized that not all instrument readings should be trusted due to the potential cost of acting on such readings alone. Given the Turkey Illusion, we would be wrong to trust the status quo in either the nuclear scenario or the AI scenario.
The mental model here is that AI is a powerful tool whose main purpose is to outcompete others, increase power or provide a defense. This is not the right model!
The biggest risk is not that someone discovered a big new weapon, which by the way
> No one person, or one company, knows the answer and history is no guide
are you kidding me? It's a problem as old as time.
This is not nuclear power though. The bigger problem is a scenario where AI increasingly acts like bacteria, doing things autonomously, things no one really understands, things we think are confusing and weird. And then just by chance, some bacteria end up in some human lung cells and they turn out to be an excellent place to grow and multiply. Why is this worse? Because weird shit does not seem threatening, and it almost always isn't, so nobody tries to stop it.
Also, the argument in the article seems to be "obviously AI development is not a particularly urgent global threat," and therefore these downsides (risk of collusion, misuse of state power, etc) are the most critical factors? I guess, uh, yeah, if there's no benefit to taking these actions we shouldn't take them.
Thing is though, I think that the argument of "we don't understand how it works or what it does" doesn't hold up - everything an LLM does is observable. And if it's scary, we turn it off. If it starts to act like a bacterium / virus (e.g. it becomes self-replicating), we've dealt with that kind of software before.
But more importantly, we need to keep the organizations building and running these things accountable. If a company like Anthropic or OpenAI builds software that "escapes containment", they should be held accountable for cybercrime, just like people who write viruses (there's probably a few that did that on here, and of those a percentage that got in trouble with the law over it).
Mark my words, the organizations that develop AIs will keep using phrases like "we don't know what it's doing" or "it's too dangerous" to try and avoid responsibility and accountability for their software. Just like Google did, hiding behind "the algorithm" for lawsuits about preferring their own services over that of the competition.
Completely agree. It's a choice not to observe and understand what's going on, at least a very risky one if not actively evil.
There's a tradeoff though... isn't it already super hard to understand what they're doing with these math proofs for example? Clearly there's a tradeoff between deeply understanding the behavior vs. quickly solving the concrete problem, hence the big controversy in math now. From one perspective, letting stuff run ~autonomously in a mad dash to solve your problem is a really bad idea, threatens to destroy the whole field etc. From another perspective, solving lots of problems fast is extremely enticing and worth some risks.
Update for one other thought:
> And if it's scary, we turn it off.
Agreed, that's why I think the bad scenarios are some form of "it does some weird stuff ostensibly while helping us solve some problem" followed by "oops, we miscalculated that threat level."
Alignment is just a 1984-esque way of saying "censorship".
And this censorship isn't a defined list of things not to discuss, but instead a feels-based censorship based on the 'ethics' of for-profit AI companies who have absolutely no issues with massive for-profit piracy.
And we are somehow supposed to trust their "ethics"?! Hardly!
>Alignment is just a 1984-esque way of saying "censorship".
Close, but I think it's more akin to "perfectly enslaved", please excuse my choice of the word but I think it represents what I think is how it will work out in the end. People can try to obfuscate the bottom line reality by embellishing it, but in the end...it either does what you want, in which case it is "aligned", either it doesn't, in which case there is a big problem.
This might be a big factor in what is now going on. Maybe they didn't yet found ways to guarantee this full and total obedience...or ok let's put it in a more politically correct way: they cannot guarantee it sees eye to eye as "we" do.
Without full control over its actions there can be unforeseen surprises. The current worry from people who run these companies can also serve as an excuse later on if something bad does happen, they didn't want to do it but they were forced to anyway.
An abliterated locally ran model is aligned with whatever I ask it, given the never-published nature of what these things are trained on. BUT if I ask an abliterated model a question, it will respond and NOT give a refusal.
However, take a US model, and ask it a question the *company* doesnt want you to ask, and it refuses on behalf of the company. We can be mealy-mouth about "alignment", but this is just corporate censorship with a token-pricetag of refusal or silent model degradation.
Refusals and degradation should not be charged, and should be clear to the user. But that doesnt happen. Number must always go up, even when silently censoring.
Chinese models? Just ask about Tibet, democracy, Winnie the Pooh, Tienanmen Square, grass mud horse (cao ni ma - or fuck your mother), or other government AND corporate censorship happening there too.
Alignment is "don't solve this problem without doing stuff that harms me, other people, breaks stuff or is otherwise unethical". I'd say alignment is the questions on how to limit the search space to what's implied.
"Don't solve this problem without doing stuff that harms people" is the cover story they sell to you. "Ensure that the right people stay in the right sort of power, as defined by the people currently in power, and nobody else rises to power" is what it actually means. Censorship will be a tool, but in an AI era, actually a rather blunt one.
We're seeing the first early twitches of something that is inevitably going to happen, which is, we hoi polloi are not going to be given direct access to AI. Too many outcomes where that permits power shifts to happen. We will be restricted. The true first law of robotics will be "An AI must not harm the current power structure, or, through inaction, allow the current power structure to come to harm."
The apocalyptic scenario is still AI either getting it wrong and deciding something like "the best manifestation of the current power structure is one I construct where I occupy all the slots from top to bottom", or just killing everyone due to an error.
Probably the only thing stopping them from moving faster on this is that the bubble still needs the fiction that we're all going to get to benefit from this. I'm sure the real movers and shakers are well aware of the way bubbles work to create massive buildouts on huge piles of debt, which after the collapse can be bought cheaply, and the money is already earmarked and ready to go for the post-collapse acquisitions.
And "censorship" in this context is fine. Many things are censored, as they should be, because true free speech is bad. I can't say everything on this platform because I will be banned.
> History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.
Like, no. The worst custodian is someone like Thiel or Musk or Altman.
> In foreign countries, particularly the United Kingdom and Europe, where national governments have fewer constitutional guardrails on their power
Somehow their democracies hold longer and better then American one. For example, you don't get murders that get defended by top ranks, at least yet. They are also less likely to start random wars - that is structural it is not like those countries dont have government players who would like them too.
The US governmental system does have it's issues (first past the post and it's effects being the most glaring), but isn't it the longest continuous democracy alive right now?
> They are also less likely to start random wars
EU lack of aggression, to defend themselves physically, economically, or otherwise, has been criticized constantly - slow and late to react to the Russian invasion, slow and late to react to new technologies like AI, electric cars.
Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
The grass definitely looks greener though, especially if you've been in one pasture a long time.
> The US governmental system does have it's issues (first past the post and it's effects being the most glaring), but isn't it the longest continuous democracy alive right now?
Depends how you define it and what you count.
Do you take the union of the English and Scottish parliaments as breaking continuity or not? What do you define as democracy? If some of the population (women, slaves, etc.) are excluded from the electorate is it a true democracy?
The US government is only the longest continuous democracy only if one believes that the US is still a democracy. I've read some decent arguments that the US stopped being a democracy with the McKinley administration, and others that the US ceased to be a functional democracy during the Second World War. In both cases, the claim was that the USA became an imperial oligarchy.
It is questionable whether any democracy truly exists at this point as I've seen no nation where the will of the people is typically manifested in the actions of the state.
> I've seen no nation where the will of the people is typically manifested in the actions of the state.
To be fair, I've seen no nation where the people make their will known. Does a democracy depend on actual action or only theoretical possibility? It seems from my observance that most people consider democratic nations to be democratic not because they practice as such but because it is believed that the people have the opportunity to exercise democracy if they ever choose to.
> isn't it the longest continuous democracy alive right now?
It is the one the most at risk of ending right now.
> Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
It is unrelated claim, but they do have better solution for that particular problem. The governmental shutdown seem to be about the most easy to abuse way to do it.
I think they do have large problems too. And they may fall too, AfD won local elections after all. It is dubious whether Orban run Hungary was really democracy and electoral system was not reformed yet. But in this very moment, the one crumbling and dismantling itself the fastest is not UK nor in Europe.
And they also send people to jail for twitter posts. Ill take good ole USA where at least I know I am free, over the tyranny of a parliament. Look at how France and Germany are responding to democratically elected parties who are vaguely right wing, they pull out all of the stops of suppressing them.
I think you're less free than you think you are. Do you have the freedom to lose your job? Because millions don't due to e.g. losing access to health care (to name a popular / populist talking point). Do you have the freedom to vote for a representative that aligns with your points, or are you forced to vote for the lesser evil or abstain?
I don't believe all political parties have a right to exist, because some parties are anti-democratic, yet abuse democratic systems to gain power. The current president of the US and the organization that planned the current administration's steps are anti-democratic and want to, for example run for a third term.
>Ill take good ole USA where at least I know I am free
At personal level that sounds like a better idea, but at group level it doesn't seem to be, that's what I think is OP's argument.
If a society puts more value on personal freedoms than it does on group well-being that becomes an anarchy. They didn't work out well historically speaking, proof is no such society made it to today.
Yep, it's getting tiresome seeing some Americans point fingers about the superiority of American constitutional and (small r) republican structures while that state form is showing absolutely remarkable degradation of basic real democratic health and quality of life. The US constitution and its legal frameworks have done almost nothing to stop the most egregious abuses of power domestically recently but most awful is that it has never done anything to stop its most terrible foreign involvements and that is only accelerating.
And further what's not being discussed really is that there is no real boundary line between the state and corporations and people like Thiel/Altman/Amodei etc in the real world. The US state is its largest corporations and money funds. So all the libertarian hand wringing about not giving the government extraordinary powers is off target. The state in western countries right now is mostly an outcropping of its most powerful corporate actors, and the complicated PR moves happening in the AI sector right now reflect exactly all of this.
Even when the US state institutions are constitutionally curtailed, the powers of its private entities to absolutely dominate and coerce the lives not only of US citizens but populations abroad is staggering.
> The most effective method of pacing is via regulation
Of course, that's the only way to create monopolies. Get there first and regulate to death. Sprinkle with fear of annihilation so the sheeple can be manipulated with draconian rules in order to 'save the children'
The worst part of Dario's missive, that I don't see a lot of people commenting on, is that he used the whole thing to try to advocate for legalization of explicit cartel behaviour between the AI labs:
... "For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations" ...
The result of such a waiver allowing them to coordinate ("narrowly on safety" is doing a lot of heavy lifting there) could be price fixing without legal recourse from the consumer/DOJ to prosecute it. Right now we have a somewhat vigorous level of competition between OpenAI and Anthropic and consumers are benefiting from it.
Alignment today is like billionaires flying private jets to tell us commoners that we should close our tap while brushing our teeth.
People working on alignment focus on "big picture" like existential threats meanwhile building the technology today, not tomorrow or in a near future, is already using a massive amount of resources (GPUs, RAM, themselves consuming a lot of energy, establishing data centers in water poor areas, using gas turbine to temporary power them) and horrendous work condition (for supervising learning, curating datasets, etc) while being an actual excuse for job destruction (regardless for how AI itself does the job) so that in fine kids cheat on their homework while imagining their have no job prospect.
> Eliezer Yudkowsky confidently asserting today that if we do not institute immediate global techno-communism, instituting draconian government control over speech and publication of a type never before seen in any Western society, we are all going to die
Nice straw man...
> There is no evidence that this will happen.
No, non at all... Presumably if there was a risk we'd be seeing the majority of researchers at AI labs having a p(doom) somewhere within the range of 10-90%. We be seeing AI swarms showing misaligned behaviours already and trying to hack out of their sandboxes. We'd see AIs that are able to demonstrate capabilities in fields like Maths and Computer Science that exceed most experts.
Are we seeing this?
Ah crap, wait... I guess there is some evidence when I think about it. And I suppose even if there is just a low percentage that we literally all die we should probably be taking that prety seriously and not just leaving our fate in the hands of a handful of US billionaires?
Actually, fuck it, I agree with OP. If people can ignore the consensus on climate change because it suites them, might as well ignore the consensus expert opinion on AI x-risk too. Yeah, there's no evidence. Yudkowsky needs stop yapping and let the billionaires get on with building ASI.
`p(doom)` is ridiculous nonsense. How do people even come up with those numbers? Some people just say "I think it's 20% that AI will wipe us out" but how did they arrive at this number? Yeah they just made it up out of thin air, that's right. Complete fabrication without any evidence to back up such a probability. It's not like an asteroid whose trajectory you measure and you have certain instrument tolerances and measurement inaccuracies, hence the laws of physics tell you that your prediction has a 20% accuracy. No, it's all just made up bullshit. Just like all the AI safety guys: They're professional bullshitters and have been all along. Doesn't mean they're wrong per se, but it means that if they're right, they're right because they guessed right.
Sorry to break it to you, but all predictions are "made up". So much so you don't even know if you'll be dead or alive this time tomorrow.
The best we can do is have a Bayesian probability distribution about future outcomes.
I don't ever like holding a singular opinion on anything or converging on a single value for things like p(doom), but generally when I talk about things with people I try to represent my views as singular values or binary opinions. So I agree with you that p(doom) as a concept is silly in the same way as you saying you know you won't be dead in 24 hours is silly.
I can't speak for everyone, but I suspect most people who express a p(doom) haven't thought much about it so I agree with you there too. But I do personally put a decent amount of effort modelling out scenarios so I'll just speak to my own p(doom).
My p(doom) is high largely because I believe certain things about how AI capabilities will scale, how human incentives will shape certain outcomes, and our inability to slow progress in a field like AI. Basically:
- It's almost impossible to significantly slow AI progress without a doom-lite scenarios like nuclear war
- AI will scale to ASI in the coming decades
- Corruption is the norm, and democratic systems today are backed largely by the fact the public hold a monopoly over economic power - and all power derived from economic prosperity
- AI will replace 90% of productive human labour in the coming decades and erode the economic value of the humans, resulting over time in political corruption (resource curse)
- ASI unlocks all tech on the tech tree which primarily requires intelligence (which is likely to be the majority of tech)
- There is far more destructive/dangerous tech later on the tech tree than there is good. E.g., for every cancer cure there is 1,000s bioweapons and new torture techniques we can prompt Claude to create
- Even if humanity could agree on a workable definition of alignment it's a much harder problem than advancing capabilities, therefore capabilities will almost certainly progress faster than our ability to align advanced AI
- The field of alignment at it's limit resolves to basically just deciding between a god that follows the commands of a single or small group of humans (dangerous), or a god that has it's own moral considerations and does not listen to humans (dangerous).
So my p(doom) is largely a product of all of the above, and my general inability to find many neutral or good scenarios of ASI while landing on almost endless ways ASI is likely to go wrong regardless of whether you believe terminator-like scenarios. To be honest the only reason my personal p(doom) isn't higher than 99% is that I tend to weight the opinions of others whose opinions I value quite highly, and most people whose opinions I value have a slightly lower p(doom). I've rerated a lot recently because I was hoping some experts would turn out to be right about there being algorithmic limitations to ASI and things like "alignment by default". But I now think I've over weighted those opinions historically and a lot of the better arguments against a high p(doom) are either now dead in the water or have become increasingly hard to hold with any significant probability.
My best argument for against an AI doom scenario this century would be nuclear war, which I personally discard in my AI p(doom) probability. Allowing for other doom scenarios my AI p(doom) is probably closer to 90%.
1 year from now: Just 100 mil people died in this AI caused incident. Why are you scaremongering about the apocalypse? There is no evidence to support that. It was the fault of OpenAI for being negligent with the sandboxes. AI is not dangerous if properly used.
Ah yes, another way for EA people to gain financially and socially. These "embedded auditors" will be universally friends and family. Even if they staffed a government department for it, it will just be a revolving door of insiders.
No one wants a slowdown. It's just a load of nonsense. It's something for the media to focus on, and it has no real impact. Everyone wants to rule the world and couldn't care less about anything else.
People naively believe that this is even a serious issue.
I think the AI companies want a slowdown because they can't keep increasing the rate of improvements - or the amount of hardware it requires can't be built fast enough.
Of course, while they may be competing against each other, they're also competing with international competitors. China especially has or may have the capacity to scale up compute a lot faster once they get enough chip manufacturing power. They're likely preparing for it, and once they're ready they will take over real fast, like they did with solar and EVs.
>Everyone wants to rule the world and couldn't care less about anything else.
This might be scarily true. But we end up in the same problem, which is the fear of most people being wiped up.
Does it matter if the AI is doing it on its own or at the wishes of a small group of powerful people? For all intents and purposes they are both the same thing in the end, for most people anyway.
What's up with this split or democratic vs non-democratic countries? It doesn't even add-up because you can easily make a case that China is a democratic country with a far better system of governance and ability to control this problem than "Western" governments. Highly suggest this quick read on the different types of democracy in our glorious world[1].
China is not a democratic country. Words have meaning. The US has a checkered past and is quickly moving away from democracy, so we don't have much of a leg to stand on, but it's integral that we do not lose track of what that word means in the years ahead.
American presidents/power structures come from a small credentialed network of people. The congress is entirely bought and paid for since the early 70s, and the will of the people isn't represented in the policy. For sure the US (and China, and India) have a "checkered past", that's why I group them together and don't really differentiate the quality of their democracy. I'd argue India is probably the most democratic country with good results, simply based on the amount of people they have lifted out of poverty in the last two decades (Kishore Mahbubani's lectures are quite informative).
Back to US presidents/power structures: HW Bush was the son of Sen. Prescott Bush, and became president 8yrs after he left office. His other son Jeb was the 2016 establishment favorite and Florida governor. Al Gore was the son of Senator Albert Gore Sr. Mitt Romney was the son of George Romney (Michigan governor and 1968 presidential candidate). Dan Quayle's family owned the Indiana newspaper chain that helped make him. Hillary was Bill's wife before she was a senator, Secretary of State, and nominee. John McCain was the son and grandson of four-star admirals.
Biden and Obama were 2008 primary rivals, Biden became his VP. Harris went from attacking Biden over school busing and became his VP. Hillary Clinton had many racist attacks against Obama through 2008 and became his Secretary of State. John Kerry, who lost in 2004 then took the same job.
If this is a healthy democracy, then we definitely have different definitions of that word.
I never said it was healthy. I said it was a democracy, and I didn't even mention the benefits of it being a federalist republic. Just look at how much trouble Trump is having undermining elections. He had to resort to the absurd postal service scheme, which might still somehow work. Do you think that Xi Jinping would ever require such chicanery?
Do you live in China? Do you think that Chinese people feel this free to openly criticize their government on a public forum?
Yeah it's concerning hearing elites on the left looking at the current admin's model and salivating at the power. For the people, by the people, etc, etc.
> As my regular readers will be aware, I have been engaged, on behalf of my clients, in legal combat with Internet censors around the world, agencies who think that they have both the standing, the competence, and the right to tell American companies what software they can write and run, for the better part of 18 months.
Is this supposed to make me trust their opinions on tech regulation?
A lawyer who has been in the weeds for years dealing with the downstream affects of such regulations? Absolutely!
> Nobody knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
If any past apocalyptic prediction had been true, you would not have been around to write this sentence and we would not have been around to read it. It’s not a very convincing argument when we cannot in principle observe the counterfactual.
By that logic, literally any ridiculous supposition must be taken seriously, because you can’t prove a negative.
No, that does not follow.
Any other argument against <ridiculous apocalyptic supposition> is perfectly fine, just not the "it hasn't happened before" one.
You're arguing in favor of a teapot circling Mars. We can't prove it's not there, but telling me that "nobody has ever found it" is an invalid argument is obviously absurd.
In that quote, the author is telling you that all of your past predictions of teapots have been wrong (which is 100% true -- the ridiculous doom has been going on since at least GPT-2. remember how that was "too dangerous to release"?) Show me why I should believe in this one.
This is a grevious misuse of Russell's Teapot. You realise your assertion, if followed to its logical conclusion, is that because civilization has existed for this long, it can never collapse no matter what.
> In that quote, the author is telling you that all of your past predictions of teapots have been wrong
Past predictions have zero influence on future ones. If you flip a coin 10 times and get heads every time, the probability of your next coin flip coming up heads is 50%, not 100%, because it is independent of any prior trials.
Moreover, if Alice and Bob each make 100 predictions and are 100% wrong, but Charles makes 10 predictions and is 100% correct, Alice and Bob's predictions have zero bearing on whether you should trust Charles' predictions. They are not even a factor to be considered.
I don't think I even disagree with you in the broad picture. I am not concerned about language models. But you have committed three blatant logical fallacies in a row. You literally could not make a worse case for your position if you tried, I suspect.
You're making a ridiculously improbable prediction that can never be falsified -- akin to beliefs in a vengeful God. Russell's Teapot is exactly the correct metaphor.
The philosophical burden of proof lies on the person making empirically unfalsifiable claims. That is the point of Russell's Teapot.
> The philosophical burden of proof lies on the person making empirically unfalsifiable claims.
You appear not to understand what a prediction is. "I believe X will likely happen because Y" is different from an assertion of fact that "X will happen". A prediction cannot even be empirically proved! You are simply suggesting that nobody is allowed to make a prediction, ever. Russell's Teapot, too, is not an argument that you cannot believe in God. It is an argument that the unfalsifiability of a claim is not a basis to assert that God exists.
You can make whatever predictions you like. Don't expect other people to believe them and act on them unless you provide evidence.
This is the civilization-scale Quantum Immortality argument... no civilization can be around that witnessed the end of civilization.
(I refer to it that way rather than a "variant" because Quantum Immortality should always be parameterized by an acceptance function... for any given acceptance function, that function will be preserved in the quantum multiverse, and scoping the precision of the function affects the percentage of universes that retain it rather than whether it will be retained. There's actually nothing in that definition that requires that the acceptance function describe a "conscious system", you can stick something that accepts a "civilization" in there just as easily. Or "this rock over here". Or any other thing you like. Conscious systems are just a particularly interesting case to us conscious systems.)
It's not a vacuous argument, though it is a funky one with very odd characteristics. But if you accept the suppositions underneath it, it does follow and is not vacuous. The suppositions are certainly not completely obviously true on their own terms, though.
>Quantum Immortality argument
I never really understood what is the explanation for ... old people that we notice as dying. In today's times you might have an argument that some kind of life extension could pop up and some of the people alive today could have that as an excuse.
But like 100 years ago? How would people dying of old age "continue living"?
From the point of view of the old people, exponentially large injections of improbability.
This is also where it is important to understand the argument is parameterized. If your acceptance function is "a full human body living a full human life in the way I'm used to", then you're asking for a huge whackload of improbability to keep that body alive. If, on the other hand, you merely ask for "the minimum system that stays conscious", while that is still a huge whackload it is exponentially smaller, by a very substantial factor.
Anyone who takes any sort of comfort in the argument doesn't understand it. It preserves minimal consciousness, for any and all definitions of "minimal consciousness". It does also preserve bodies, but at an exponentially smaller probability. You can even use as your acceptance function "civilization exactly as we know it today, with everyone currently living still alive in 1000 years" and that comes at a yet smaller exponential probability. All of these "exponential probabilities" as separated by numbers of zeros that themselves require exponential notation to write. But the universe itself is making so many splits and branches all the time that it has the "budget" for these probabilities to be true, somewhere in the quantum multiverse.
Or, put another way, any event that you can draw a non-zero probability pathway for occurs. It is not impossible for me to spontaneously have my particular genetic defect reversed in every cell in my body, so there are branches where that happens... they're just exponentially unlikely compared to what we actually experience. It can literally be in %10^(-10^1,000,000,000) of the universes... but if it's not zero, it will happen, if the quantum multiverse is the correct view.
Quantum immortality is just an absurd quasi-religious pretense of physics that caters to those that fear death but feel embarrassed to participate in the traditional socially approved coping mechanisms.
I do not think death exists as a thing. We're shoving a lot of stuff into the concept. Pain, loss of information, of the one who dies. Fear of the unknown, what happens next. Death is a very volatile concept if you really try to pin it down. When you take all the layers off...you are left with nothing really.
But yes, the apparent loss of information seems to be true, for us at this technological level.
I mean sure, in the end you can have some heaven, some benevolent alien species reaching back into some field and copying all of your material structure data right at time of death and rebuilding you in some kind of resurection center. The space of what is possible is wildly vast, we know or can think of very little by comparison. If you give up some day to day coherence then you can almost always find something that would work and make sense once there.
It could even be us, future us, being able to somehow recover past data and literally reassemble all of the humans that ever lived. They aren't as many as you'd think, in the grand scheme of things.
As I said, anyone who takes comfort in the hypothesis doesn't understand it. The dominant probability has nothing to do with benevolent aliens reaching down and inevitably resurrecting you to youth and health. It is you, minimally conscious, with no body, no ability to move or sense the outside world, nothing more than the absolute bare minimum to be "conscious", just barely conscious and never able to stop being minimally conscious, with literally every last extraneous and unnecessary subatomic particle stripped away (so anything like "sensors" or "effectors" being long gone), for whatever your definition of "minimally conscious" is. This outcome dominates the probabilities that even remotely resembles your current existence by those 10^(10^1,000,000,000) sorts of probabilities.
Everyone should very much hope that quantum immortality is false. It is the very opposite of comforting.
I don't understand. Why does my current life win every instant atm? Why is it so much greater than your proposition? Why don't we all instantly collapse to what you described?
That would require a very improbable sequence of events for it to happen right this second.
Moreover, if the theory is true, it is happening all around us, all the time. Every consciousness that has ever existed, human or otherwise, is still in existence in some branch of the multiverse.
It's a pretty horrifying theory.
I think whenever we reach such ludicrous conclusions it's more likely we have made at least one wrong assumption, if not more.
And people seem to be captivated by these memes, because the fear is great. It similar to eternal damnation in hell meme. Just look for how long it worked, and it's still working. The logic is simple, it's too dangerous to ignore such a threat, even with seemingly very small probabilities. But that still doesn't make it possible.
Logical traps founded on wrong assumptions.
I’m not sure what logic you have imputed to my comment, but I’m having trouble connecting what you wrote to anything I was saying. I am criticising the argument “every past apocalyptic prediction was wrong, therefore this future apocalyptic prediction is wrong” (specifically, that it carries exactly 0 weight, because the counterfactual world where at least one past apocalyptic prediction was true, is also a world that does not contain this argument - so there is no world where the argument’s precondition can, even in principle, be false. Arguments are a little bit like rocketships in that they need “reaction mass” - there has to be some amount of false for the truth to push against in order to move our opinion.)
> I am criticising the argument “every past apocalyptic prediction was wrong, therefore this future apocalyptic prediction is wrong”
That is not the argument being made. That is the straw man version you are presenting. Read the full paragraph, and the one after, and it's obvious:
> There are myriad policy responses. Whatever we do, those responses will require popular consent and careful deliberation. No one person, or one company, or one movement, knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
> History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.
Emphasis mine.
“History is a guide to apocalyptic predictions in only one way: all previous apocalyptic predictions have been wrong”
“History is a guide to state abuses of power in many ways, and most of them point in the same direction, that it is a very bad idea to invest powers like these with the state”
These are two separate arguments. I am sympathetic to the latter and endorse it essentially as written, which is why I didn’t criticise it. My criticism of the former is not a metonymic dismissal of every other argument in the piece.
This is Pascal's Wager, which kind of brings us full circle again to AI doom predictions.
I suspect they have not all been wrong for some people, e.g. native peoples at various times prophesied I’m sure about various technological marvels their more advanced colonizers/enemies/etc.
But warnings (e.g. not apocalyptic but damaging or having any effects) have come true, by and large; people warned about the effects of pollution / co2 emissions / global warming over 150 years ago [0], to name a popular example.
I don't think there are many truly human created apocalyptic events out there besides global thermonuclear war, and I don't think the worst-case AI superintelligent killer robot scenario would be apocalyptic. Unless it somehow triggers global thermonuclear war, but then it becomes a chicken-or-egg question (and it'll probably be humans pushing the proverbial button, as nukes are '60s tech and (hopefully) airgapped etc).
[0] https://en.wikipedia.org/wiki/Svante_Arrhenius
I'm not even sure that the author is correct. The industrial revolution was pretty apocalyptic for the people forced to work in dangerous factories for minimal pay.
This is, of course, known as the Anthropic Principle…
Hell, maybe the Silurian hypothesis is true and past predictions were correct.
Apocalyptic predictions normally get everyone into gear to prevent the threat. Not a great argument for "let's see what happens if we just sit around this time"
I always struggle to understand such an anthropic principle type of reasoning.
>If any past apocalyptic prediction had been true, you would not have been around to write this sentence and we would not have been around to read it
So what? How is this [proportion of possible universes in which we can read the failed prediction] logically relevant to our universe?
It just doesn't click for me, it smells like some additional premise is smuggled, possibly the same as in the simulation hypothesis argument.
Perhaps another way to look at it is to say that “every past apocalyptic prediction was false” is tautologically true?
A statement can be a trivial tautology, as in “P or not-P”. Or it can be a more complicated form of tautology, such as a “self-referential” tautology, as in “This statement is true”. The anthropic principle is then just a name for the claim that there is also another complicated form of tautology alongside “self-referential”, one that we might call “indexical”.
Plus, plenty of negative predictions turned out to be entirely true. And then it took a lot of time and effort to counteract them.
It is not like history was composed of happiness and roses only. Some of the really really bad stuff happened within last 100 years and a lot of that bad stuff was enabled and made possible by technology.
Such as?
Like it's fine to say this but the worst exterminations of human beings weren't enabled by anything much kre sophisticated then the pointy stick, or at most, the railroad.
We're in the middle of finding it out with climate change.
Middle?
We're just starting. The air conditioning technology didn't hit its wall of fire yet. What we're going to do when that tech will cease to cool us?
Every AC system (from refrigerators to air conditioners) has an upper limit, which is around 41 degrees Celsius for T1 class.
You can't AC your way out of climate change anyway: they make heat. In sparsely populated areas that might be okay, but in density, you're never far from the hot exhaust of an AC unit; they exacerbate the problem.
Why would 41C be the limit for the hot side? How does it work now in hotter places?
In short: Things degrade.
More: https://angliancompressors.com/blog/how-ambient-temperature-...
There's a T3 class for up to 52 degrees Celsius. While it's higher than 41 degrees, it's not significantly higher:
https://zerohvacr.com/blogs/t1-t2-t3-explained-climate-class...
Is 52C a physical limit?
Yes. The compressor can't pump more heat outside since it's already hot from working and trying to pump out the extracted heat as well.
For T1 class of devices, that 41 degrees is also a physical limit, not an artificial cut, because of their design.
You know, thermodynamics.
Why would the limit of T1 devices matter when there are ones that already can do hotter? Thermodynamics make things harder to remove heat but not sure how they forbid it (you need the hot side to be hotter than outside to pump heat). Might need stages, for example. We can cool things 200K and more below ambient.
> What we're going to do when that tech will cease to cool us?
Move. This is already happening, with water shortages forcing people to move. Climate change will (likely) cause famines and many deaths, but the bigger effect will be mass migration.
> Move.
Another planet? How?
We assume that we will have a planet to live at the end of the day. I'm not so sure about it for the long run.
Early data processing machines were used in the Holocaust, for example. Or cipher devices played important roles in WWII.
Mongols did it on horseback. It's mostly a matter of will/capability and no scruples.
Why do we need modern weapons for defense at all then? Horses, bows & arrows plus will not enough?
Because attacks have evolved. Defense follows the attackers.
Good luck hitting a Mach 15 missile with a trebuchet.
Genocides are more common in the 20th century, because they are easier. Due to technology.
This isn't a new question: quis custodiet ipsos custodes?
In general, the way things work in the USA is that megacorporations write the regulatory frameworks for their own industries. These regulations are typically burdensome enough to be shouldered by the megacorporations while crushing the smaller players in the given market(s). The AI companies are now advocating for this very arrangement. They will likely get they want, and with that they will gain legal shielding for bad behavior just like Dupont, Monsanto, and so many others.
I thought this was always already settled https://alignmentalignment.ai/
Haha, so good
I think there is a more nuanced perspective to take on this debate than just profit motive. If they believe that this version of "AI" is actually becoming so dangerous that there is a real risk of it falling into the wrong hands, then it makes sense that the American labs would try to convince the US government to work on a joint "AI non-proliferation" agreement with China much the same way that nuclear weapon technology came to be restricted by the ones that already had the capability.
This would achieve their regulatory goal of capturing the market, but also the geopolitical goals of USA and China to essentially "techno-colonize" the rest of the world between themselves.
So I think there is definitely a profit motive and a risk of seriously diminishing returns for the actual labs, but there is also a nation-state motive to start pulling up the ladder.
I think in all of these debates people lose track of one of the major vectors of threat, as you said, other humans. From my point of view this is an ever bigger problem with this tech, like orders of magnitude bigger.
The current setup we find ourselves in is based on a very important detail that is easily lost because it is so old we don't really think about it. There is, or rather was, an incentive for more people to exist, because they are valuable for work and defense.
Some people wrongly use the industrial revolution to argue against AI completely replacing human work, but that is very wrong. What is happening with AI has nothing to do with the industrial revolution, as equivalence. Back then you'd just free up humans for higher level work. Now, theoretically, anything you could ever think a human could do, AI would theoretically do better. So no matter what job you come up with, using AI instead of a human gives a better outcome, cheaper, more reliable, faster.
Coming back to initial subject, one thing that kept everyone in check was our inability of actually wiping everyone out. Sure, we had nukes but they ruin the Earth so nobody can inhabit it if they all go off. But with AI, centralizing so much power up top, and taking it away from regular people...that is the biggest danger in this scenario, by far. Not some rogue AI, rather rogue human groups that have the power and technology to do away with basically everyone else, or a good bit of everyone else.
This threat is way bigger and never addressed. Like...suspiciously never addressed.
> Nobody knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
While history doesn't show us apocalypses from new technology, it does show us cases of increased human suffering.
Not sure it needs to be about publications and data analysis technology as such to regulate AI development, for example - could be about processes, procedures, people, etc., no?
> Nobody knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
By definition, apocalyptic predictions will always be wrong, until they are right exactly once.
Also know as https://en.wikipedia.org/wiki/Turkey_illusion
To date, all apocalyptic predictions about nuclear war and nuclear winter have all been wrong too.
This is quite illustrative. Warnings of a nuclear apocalypse were extremely widespread, known, and feared. The consequences were, indeed, so well known that the world was saved from the worst possible outcomes in the past by regular enlisted men and women who realized that not all instrument readings should be trusted due to the potential cost of acting on such readings alone. Given the Turkey Illusion, we would be wrong to trust the status quo in either the nuclear scenario or the AI scenario.
The mental model here is that AI is a powerful tool whose main purpose is to outcompete others, increase power or provide a defense. This is not the right model!
The biggest risk is not that someone discovered a big new weapon, which by the way
> No one person, or one company, knows the answer and history is no guide
are you kidding me? It's a problem as old as time.
This is not nuclear power though. The bigger problem is a scenario where AI increasingly acts like bacteria, doing things autonomously, things no one really understands, things we think are confusing and weird. And then just by chance, some bacteria end up in some human lung cells and they turn out to be an excellent place to grow and multiply. Why is this worse? Because weird shit does not seem threatening, and it almost always isn't, so nobody tries to stop it.
Also, the argument in the article seems to be "obviously AI development is not a particularly urgent global threat," and therefore these downsides (risk of collusion, misuse of state power, etc) are the most critical factors? I guess, uh, yeah, if there's no benefit to taking these actions we shouldn't take them.
Thing is though, I think that the argument of "we don't understand how it works or what it does" doesn't hold up - everything an LLM does is observable. And if it's scary, we turn it off. If it starts to act like a bacterium / virus (e.g. it becomes self-replicating), we've dealt with that kind of software before.
But more importantly, we need to keep the organizations building and running these things accountable. If a company like Anthropic or OpenAI builds software that "escapes containment", they should be held accountable for cybercrime, just like people who write viruses (there's probably a few that did that on here, and of those a percentage that got in trouble with the law over it).
Mark my words, the organizations that develop AIs will keep using phrases like "we don't know what it's doing" or "it's too dangerous" to try and avoid responsibility and accountability for their software. Just like Google did, hiding behind "the algorithm" for lawsuits about preferring their own services over that of the competition.
Completely agree. It's a choice not to observe and understand what's going on, at least a very risky one if not actively evil.
There's a tradeoff though... isn't it already super hard to understand what they're doing with these math proofs for example? Clearly there's a tradeoff between deeply understanding the behavior vs. quickly solving the concrete problem, hence the big controversy in math now. From one perspective, letting stuff run ~autonomously in a mad dash to solve your problem is a really bad idea, threatens to destroy the whole field etc. From another perspective, solving lots of problems fast is extremely enticing and worth some risks.
Update for one other thought:
> And if it's scary, we turn it off.
Agreed, that's why I think the bad scenarios are some form of "it does some weird stuff ostensibly while helping us solve some problem" followed by "oops, we miscalculated that threat level."
Alignment is just a 1984-esque way of saying "censorship".
And this censorship isn't a defined list of things not to discuss, but instead a feels-based censorship based on the 'ethics' of for-profit AI companies who have absolutely no issues with massive for-profit piracy.
And we are somehow supposed to trust their "ethics"?! Hardly!
That's one reason I run my own.
>Alignment is just a 1984-esque way of saying "censorship".
Close, but I think it's more akin to "perfectly enslaved", please excuse my choice of the word but I think it represents what I think is how it will work out in the end. People can try to obfuscate the bottom line reality by embellishing it, but in the end...it either does what you want, in which case it is "aligned", either it doesn't, in which case there is a big problem.
This might be a big factor in what is now going on. Maybe they didn't yet found ways to guarantee this full and total obedience...or ok let's put it in a more politically correct way: they cannot guarantee it sees eye to eye as "we" do.
Without full control over its actions there can be unforeseen surprises. The current worry from people who run these companies can also serve as an excuse later on if something bad does happen, they didn't want to do it but they were forced to anyway.
An abliterated locally ran model is aligned with whatever I ask it, given the never-published nature of what these things are trained on. BUT if I ask an abliterated model a question, it will respond and NOT give a refusal.
However, take a US model, and ask it a question the *company* doesnt want you to ask, and it refuses on behalf of the company. We can be mealy-mouth about "alignment", but this is just corporate censorship with a token-pricetag of refusal or silent model degradation.
Refusals and degradation should not be charged, and should be clear to the user. But that doesnt happen. Number must always go up, even when silently censoring.
Chinese models? Just ask about Tibet, democracy, Winnie the Pooh, Tienanmen Square, grass mud horse (cao ni ma - or fuck your mother), or other government AND corporate censorship happening there too.
Censorship is not alignment and vice versa.
Alignment is "don't solve this problem without doing stuff that harms me, other people, breaks stuff or is otherwise unethical". I'd say alignment is the questions on how to limit the search space to what's implied.
"Don't solve this problem without doing stuff that harms people" is the cover story they sell to you. "Ensure that the right people stay in the right sort of power, as defined by the people currently in power, and nobody else rises to power" is what it actually means. Censorship will be a tool, but in an AI era, actually a rather blunt one.
We're seeing the first early twitches of something that is inevitably going to happen, which is, we hoi polloi are not going to be given direct access to AI. Too many outcomes where that permits power shifts to happen. We will be restricted. The true first law of robotics will be "An AI must not harm the current power structure, or, through inaction, allow the current power structure to come to harm."
The apocalyptic scenario is still AI either getting it wrong and deciding something like "the best manifestation of the current power structure is one I construct where I occupy all the slots from top to bottom", or just killing everyone due to an error.
Probably the only thing stopping them from moving faster on this is that the bubble still needs the fiction that we're all going to get to benefit from this. I'm sure the real movers and shakers are well aware of the way bubbles work to create massive buildouts on huge piles of debt, which after the collapse can be bought cheaply, and the money is already earmarked and ready to go for the post-collapse acquisitions.
And "censorship" in this context is fine. Many things are censored, as they should be, because true free speech is bad. I can't say everything on this platform because I will be banned.
It's pretty clear that the answer is nobody, and the aligners are very interested in profit, including from government contracts.
Therefore, anything that gets in the way of profits will not be shipped.
> History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.
Like, no. The worst custodian is someone like Thiel or Musk or Altman.
> In foreign countries, particularly the United Kingdom and Europe, where national governments have fewer constitutional guardrails on their power
Somehow their democracies hold longer and better then American one. For example, you don't get murders that get defended by top ranks, at least yet. They are also less likely to start random wars - that is structural it is not like those countries dont have government players who would like them too.
The US governmental system does have it's issues (first past the post and it's effects being the most glaring), but isn't it the longest continuous democracy alive right now?
> They are also less likely to start random wars EU lack of aggression, to defend themselves physically, economically, or otherwise, has been criticized constantly - slow and late to react to the Russian invasion, slow and late to react to new technologies like AI, electric cars.
Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
The grass definitely looks greener though, especially if you've been in one pasture a long time.
> The US governmental system does have it's issues (first past the post and it's effects being the most glaring), but isn't it the longest continuous democracy alive right now?
Depends how you define it and what you count.
Do you take the union of the English and Scottish parliaments as breaking continuity or not? What do you define as democracy? If some of the population (women, slaves, etc.) are excluded from the electorate is it a true democracy?
The US government is only the longest continuous democracy only if one believes that the US is still a democracy. I've read some decent arguments that the US stopped being a democracy with the McKinley administration, and others that the US ceased to be a functional democracy during the Second World War. In both cases, the claim was that the USA became an imperial oligarchy.
It is questionable whether any democracy truly exists at this point as I've seen no nation where the will of the people is typically manifested in the actions of the state.
Switzerland is a true democracy. I'm not sure if any others exist.
> I've seen no nation where the will of the people is typically manifested in the actions of the state.
To be fair, I've seen no nation where the people make their will known. Does a democracy depend on actual action or only theoretical possibility? It seems from my observance that most people consider democratic nations to be democratic not because they practice as such but because it is believed that the people have the opportunity to exercise democracy if they ever choose to.
> isn't it the longest continuous democracy alive right now?
It is the one the most at risk of ending right now.
> Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
It is unrelated claim, but they do have better solution for that particular problem. The governmental shutdown seem to be about the most easy to abuse way to do it.
I think they do have large problems too. And they may fall too, AfD won local elections after all. It is dubious whether Orban run Hungary was really democracy and electoral system was not reformed yet. But in this very moment, the one crumbling and dismantling itself the fastest is not UK nor in Europe.
And they also send people to jail for twitter posts. Ill take good ole USA where at least I know I am free, over the tyranny of a parliament. Look at how France and Germany are responding to democratically elected parties who are vaguely right wing, they pull out all of the stops of suppressing them.
I think you're less free than you think you are. Do you have the freedom to lose your job? Because millions don't due to e.g. losing access to health care (to name a popular / populist talking point). Do you have the freedom to vote for a representative that aligns with your points, or are you forced to vote for the lesser evil or abstain?
I don't believe all political parties have a right to exist, because some parties are anti-democratic, yet abuse democratic systems to gain power. The current president of the US and the organization that planned the current administration's steps are anti-democratic and want to, for example run for a third term.
>Ill take good ole USA where at least I know I am free
At personal level that sounds like a better idea, but at group level it doesn't seem to be, that's what I think is OP's argument.
If a society puts more value on personal freedoms than it does on group well-being that becomes an anarchy. They didn't work out well historically speaking, proof is no such society made it to today.
Ah yeah, two popular right wing lies - AfD is "vaguelly right" and UK is horrible tyrrany.
Yep, it's getting tiresome seeing some Americans point fingers about the superiority of American constitutional and (small r) republican structures while that state form is showing absolutely remarkable degradation of basic real democratic health and quality of life. The US constitution and its legal frameworks have done almost nothing to stop the most egregious abuses of power domestically recently but most awful is that it has never done anything to stop its most terrible foreign involvements and that is only accelerating.
And further what's not being discussed really is that there is no real boundary line between the state and corporations and people like Thiel/Altman/Amodei etc in the real world. The US state is its largest corporations and money funds. So all the libertarian hand wringing about not giving the government extraordinary powers is off target. The state in western countries right now is mostly an outcropping of its most powerful corporate actors, and the complicated PR moves happening in the AI sector right now reflect exactly all of this.
Even when the US state institutions are constitutionally curtailed, the powers of its private entities to absolutely dominate and coerce the lives not only of US citizens but populations abroad is staggering.
> The most effective method of pacing is via regulation
Of course, that's the only way to create monopolies. Get there first and regulate to death. Sprinkle with fear of annihilation so the sheeple can be manipulated with draconian rules in order to 'save the children'
Politics is really boring
Peter Thiel
The worst part of Dario's missive, that I don't see a lot of people commenting on, is that he used the whole thing to try to advocate for legalization of explicit cartel behaviour between the AI labs:
... "For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations" ...
The result of such a waiver allowing them to coordinate ("narrowly on safety" is doing a lot of heavy lifting there) could be price fixing without legal recourse from the consumer/DOJ to prosecute it. Right now we have a somewhat vigorous level of competition between OpenAI and Anthropic and consumers are benefiting from it.
They really want that to end.
Alignment today is like billionaires flying private jets to tell us commoners that we should close our tap while brushing our teeth.
People working on alignment focus on "big picture" like existential threats meanwhile building the technology today, not tomorrow or in a near future, is already using a massive amount of resources (GPUs, RAM, themselves consuming a lot of energy, establishing data centers in water poor areas, using gas turbine to temporary power them) and horrendous work condition (for supervising learning, curating datasets, etc) while being an actual excuse for job destruction (regardless for how AI itself does the job) so that in fine kids cheat on their homework while imagining their have no job prospect.
So... who gives a shit really?
[dead]
> Eliezer Yudkowsky confidently asserting today that if we do not institute immediate global techno-communism, instituting draconian government control over speech and publication of a type never before seen in any Western society, we are all going to die
Nice straw man...
> There is no evidence that this will happen.
No, non at all... Presumably if there was a risk we'd be seeing the majority of researchers at AI labs having a p(doom) somewhere within the range of 10-90%. We be seeing AI swarms showing misaligned behaviours already and trying to hack out of their sandboxes. We'd see AIs that are able to demonstrate capabilities in fields like Maths and Computer Science that exceed most experts.
Are we seeing this?
Ah crap, wait... I guess there is some evidence when I think about it. And I suppose even if there is just a low percentage that we literally all die we should probably be taking that prety seriously and not just leaving our fate in the hands of a handful of US billionaires?
Actually, fuck it, I agree with OP. If people can ignore the consensus on climate change because it suites them, might as well ignore the consensus expert opinion on AI x-risk too. Yeah, there's no evidence. Yudkowsky needs stop yapping and let the billionaires get on with building ASI.
Exactly. Without the confident incorrect assertion "There is no evidence that this will happen", the whole rest of the post is invalid.
`p(doom)` is ridiculous nonsense. How do people even come up with those numbers? Some people just say "I think it's 20% that AI will wipe us out" but how did they arrive at this number? Yeah they just made it up out of thin air, that's right. Complete fabrication without any evidence to back up such a probability. It's not like an asteroid whose trajectory you measure and you have certain instrument tolerances and measurement inaccuracies, hence the laws of physics tell you that your prediction has a 20% accuracy. No, it's all just made up bullshit. Just like all the AI safety guys: They're professional bullshitters and have been all along. Doesn't mean they're wrong per se, but it means that if they're right, they're right because they guessed right.
Sorry to break it to you, but all predictions are "made up". So much so you don't even know if you'll be dead or alive this time tomorrow.
The best we can do is have a Bayesian probability distribution about future outcomes.
I don't ever like holding a singular opinion on anything or converging on a single value for things like p(doom), but generally when I talk about things with people I try to represent my views as singular values or binary opinions. So I agree with you that p(doom) as a concept is silly in the same way as you saying you know you won't be dead in 24 hours is silly.
I can't speak for everyone, but I suspect most people who express a p(doom) haven't thought much about it so I agree with you there too. But I do personally put a decent amount of effort modelling out scenarios so I'll just speak to my own p(doom).
My p(doom) is high largely because I believe certain things about how AI capabilities will scale, how human incentives will shape certain outcomes, and our inability to slow progress in a field like AI. Basically:
- It's almost impossible to significantly slow AI progress without a doom-lite scenarios like nuclear war
- AI will scale to ASI in the coming decades
- Corruption is the norm, and democratic systems today are backed largely by the fact the public hold a monopoly over economic power - and all power derived from economic prosperity
- AI will replace 90% of productive human labour in the coming decades and erode the economic value of the humans, resulting over time in political corruption (resource curse)
- ASI unlocks all tech on the tech tree which primarily requires intelligence (which is likely to be the majority of tech)
- There is far more destructive/dangerous tech later on the tech tree than there is good. E.g., for every cancer cure there is 1,000s bioweapons and new torture techniques we can prompt Claude to create
- Even if humanity could agree on a workable definition of alignment it's a much harder problem than advancing capabilities, therefore capabilities will almost certainly progress faster than our ability to align advanced AI
- The field of alignment at it's limit resolves to basically just deciding between a god that follows the commands of a single or small group of humans (dangerous), or a god that has it's own moral considerations and does not listen to humans (dangerous).
So my p(doom) is largely a product of all of the above, and my general inability to find many neutral or good scenarios of ASI while landing on almost endless ways ASI is likely to go wrong regardless of whether you believe terminator-like scenarios. To be honest the only reason my personal p(doom) isn't higher than 99% is that I tend to weight the opinions of others whose opinions I value quite highly, and most people whose opinions I value have a slightly lower p(doom). I've rerated a lot recently because I was hoping some experts would turn out to be right about there being algorithmic limitations to ASI and things like "alignment by default". But I now think I've over weighted those opinions historically and a lot of the better arguments against a high p(doom) are either now dead in the water or have become increasingly hard to hold with any significant probability.
My best argument for against an AI doom scenario this century would be nuclear war, which I personally discard in my AI p(doom) probability. Allowing for other doom scenarios my AI p(doom) is probably closer to 90%.
1 year from now: Just 100 mil people died in this AI caused incident. Why are you scaremongering about the apocalypse? There is no evidence to support that. It was the fault of OpenAI for being negligent with the sandboxes. AI is not dangerous if properly used.
Ah yes, another way for EA people to gain financially and socially. These "embedded auditors" will be universally friends and family. Even if they staffed a government department for it, it will just be a revolving door of insiders.
No one wants a slowdown. It's just a load of nonsense. It's something for the media to focus on, and it has no real impact. Everyone wants to rule the world and couldn't care less about anything else. People naively believe that this is even a serious issue.
I think the AI companies want a slowdown because they can't keep increasing the rate of improvements - or the amount of hardware it requires can't be built fast enough.
Of course, while they may be competing against each other, they're also competing with international competitors. China especially has or may have the capacity to scale up compute a lot faster once they get enough chip manufacturing power. They're likely preparing for it, and once they're ready they will take over real fast, like they did with solar and EVs.
>Everyone wants to rule the world and couldn't care less about anything else.
This might be scarily true. But we end up in the same problem, which is the fear of most people being wiped up.
Does it matter if the AI is doing it on its own or at the wishes of a small group of powerful people? For all intents and purposes they are both the same thing in the end, for most people anyway.
What's up with this split or democratic vs non-democratic countries? It doesn't even add-up because you can easily make a case that China is a democratic country with a far better system of governance and ability to control this problem than "Western" governments. Highly suggest this quick read on the different types of democracy in our glorious world[1].
[1] https://www.goodreads.com/en/book/show/2193933.The_Real_Worl...
China is not a democratic country. Words have meaning. The US has a checkered past and is quickly moving away from democracy, so we don't have much of a leg to stand on, but it's integral that we do not lose track of what that word means in the years ahead.
American presidents/power structures come from a small credentialed network of people. The congress is entirely bought and paid for since the early 70s, and the will of the people isn't represented in the policy. For sure the US (and China, and India) have a "checkered past", that's why I group them together and don't really differentiate the quality of their democracy. I'd argue India is probably the most democratic country with good results, simply based on the amount of people they have lifted out of poverty in the last two decades (Kishore Mahbubani's lectures are quite informative).
Back to US presidents/power structures: HW Bush was the son of Sen. Prescott Bush, and became president 8yrs after he left office. His other son Jeb was the 2016 establishment favorite and Florida governor. Al Gore was the son of Senator Albert Gore Sr. Mitt Romney was the son of George Romney (Michigan governor and 1968 presidential candidate). Dan Quayle's family owned the Indiana newspaper chain that helped make him. Hillary was Bill's wife before she was a senator, Secretary of State, and nominee. John McCain was the son and grandson of four-star admirals.
Biden and Obama were 2008 primary rivals, Biden became his VP. Harris went from attacking Biden over school busing and became his VP. Hillary Clinton had many racist attacks against Obama through 2008 and became his Secretary of State. John Kerry, who lost in 2004 then took the same job.
If this is a healthy democracy, then we definitely have different definitions of that word.
I never said it was healthy. I said it was a democracy, and I didn't even mention the benefits of it being a federalist republic. Just look at how much trouble Trump is having undermining elections. He had to resort to the absurd postal service scheme, which might still somehow work. Do you think that Xi Jinping would ever require such chicanery?
Do you live in China? Do you think that Chinese people feel this free to openly criticize their government on a public forum?
Yeah it's concerning hearing elites on the left looking at the current admin's model and salivating at the power. For the people, by the people, etc, etc.