> Nobody knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.
If any past apocalyptic prediction had been true, you would not have been around to write this sentence and we would not have been around to read it. It’s not a very convincing argument when we cannot in principle observe the counterfactual.
I'm not even sure that the author is correct. The industrial revolution was pretty apocalyptic for the people forced to work in dangerous factories for minimal pay.
Plus, plenty of negative predictions turned out to be entirely true. And then it took a lot of time and effort to counteract them.
It is not like history was composed of happiness and roses only. Some of the really really bad stuff happened within last 100 years and a lot of that bad stuff was enabled and made possible by technology.
Like it's fine to say this but the worst exterminations of human beings weren't enabled by anything much kre sophisticated then the pointy stick, or at most, the railroad.
I think there is a more nuanced perspective to take on this debate than just profit motive. If they believe that this version of "AI" is actually becoming so dangerous that there is a real risk of it falling into the wrong hands, then it makes sense that the American labs would try to convince the US government to work on a joint "AI non-proliferation" agreement with China much the same way that nuclear weapon technology came to be restricted by the ones that already had the capability.
This would achieve their regulatory goal of capturing the market, but also the geopolitical goals of USA and China to essentially "techno-colonize" the rest of the world between themselves.
So I think there is definitely a profit motive and a risk of seriously diminishing returns for the actual labs, but there is also a nation-state motive to start pulling up the ladder.
> As my regular readers will be aware, I have been engaged, on behalf of my clients, in legal combat with Internet censors around the world, agencies who think that they have both the standing, the competence, and the right to tell American companies what software they can write and run, for the better part of 18 months.
Is this supposed to make me trust their opinions on tech regulation?
This isn't a new question: quis custodiet ipsos custodes?
In general, the way things work in the USA is that megacorporations write the regulatory frameworks for their own industries. These regulations are typically burdensome enough to be shouldered by the megacorporations while crushing the smaller players in the given market(s). The AI companies are now advocating for this very arrangement. They will likely get they want, and with that they will gain legal shielding for bad behavior just like Dupont, Monsanto, and so many others.
Not sure it needs to be about publications and data analysis technology as such to regulate AI development, for example - could be about processes, procedures, people, etc., no?
Alignment is just a 1984-esque way of saying "censorship".
And this censorship isn't a defined list of things not to discuss, but instead a feels-based censorship based on the 'ethics' of for-profit AI companies who have absolutely no issues with massive for-profit piracy.
And we are somehow supposed to trust their "ethics"?! Hardly!
This is quite illustrative. Warnings of a nuclear apocalypse were extremely widespread, known, and feared. The consequences were, indeed, so well known that the world was saved from the worst possible outcomes in the past by regular enlisted men and women who realized that not all instrument readings should be trusted due to the potential cost of acting on such readings alone. Given the Turkey Illusion, we would be wrong to trust the status quo in either the nuclear scenario or the AI scenario.
Alignment today is like billionaires flying private jets to tell us commoners that we should close our tap while brushing our teeth.
People working on alignment focus on "big picture" like existential threats meanwhile building the technology today, not tomorrow or in a near future, is already using a massive amount of resources (GPUs, RAM, themselves consuming a lot of energy, establishing data centers in water poor areas, using gas turbine to temporary power them) and horrendous work condition (for supervising learning, curating datasets, etc) while being an actual excuse for job destruction (regardless for how AI itself does the job) so that in fine kids cheat on their homework while imagining their have no job prospect.
No one wants a slowdown. It's just a load of nonsense. It's something for the media to focus on, and it has no real impact. Everyone wants to rule the world and couldn't care less about anything else.
People naively believe that this is even a serious issue.
Ah yes, another way for EA people to gain financially and socially. These "embedded auditors" will be universally friends and family. Even if they staffed a government department for it, it will just be a revolving door of insiders.
The worst part of Dario's missive, that I don't see a lot of people commenting on, is that he used the whole thing to try to advocate for legalization of explicit cartel behaviour between the AI labs:
... "For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations" ...
The result of such a waiver allowing them to coordinate ("narrowly on safety" is doing a lot of heavy lifting there) could be price fixing without legal recourse from the consumer/DOJ to prosecute it. Right now we have a somewhat vigorous level of competition between OpenAI and Anthropic and consumers are benefiting from it.
> History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.
Like, no. The worst custodian is someone like Thiel or Musk or Altman.
> In foreign countries, particularly the United Kingdom and Europe, where national governments have fewer constitutional guardrails on their power
Somehow their democracies hold longer and better then American one. For example, you don't get murders that get defended by top ranks, at least yet. They are also less likely to start random wars - that is structural it is not like those countries dont have government players who would like them too.
The US governmental system does have it's issues (first past the post and it's effects being the most glaring), but isn't it the longest continuous democracy alive right now?
> They are also less likely to start random wars
EU lack of aggression, to defend themselves physically, economically, or otherwise, has been criticized constantly - slow and late to react to the Russian invasion, slow and late to react to new technologies like AI, electric cars.
Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
The grass definitely looks greener though, especially if you've been in one pasture a long time.
> The US governmental system does have it's issues (first past the post and it's effects being the most glaring), but isn't it the longest continuous democracy alive right now?
Depends how you define it and what you count.
Do you take the union of the English and Scottish parliaments as breaking continuity or not? What do you define as democracy? If some of the population (women, slaves, etc.) are excluded from the electorate is it a true democracy?
The US government is only the longest continuous democracy only if one believes that the US is still a democracy. I've read some decent arguments that the US stopped being a democracy with the McKinley administration, and others that the US ceased to be a functional democracy during the Second World War. In both cases, the claim was that the USA became an imperial oligarchy.
It is questionable whether any democracy truly exists at this point as I've seen no nation where the will of the people is typically manifested in the actions of the state.
> isn't it the longest continuous democracy alive right now?
It is the one the most at risk of ending right now.
> Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
It is unrelated claim, but they do have better solution for that particular problem. The governmental shutdown seem to be about the most easy to abuse way to do it.
I think they do have large problems too. And they may fall too, AfD won local elections after all. It is dubious whether Orban run Hungary was really democracy and electoral system was not reformed yet. But in this very moment, the one crumbling and dismantling itself the fastest is not UK nor in Europe.
> Eliezer Yudkowsky confidently asserting today that if we do not institute immediate global techno-communism, instituting draconian government control over speech and publication of a type never before seen in any Western society, we are all going to die
Nice straw man...
> There is no evidence that this will happen.
No, non at all... Presumably if there was a risk we'd be seeing the majority of researchers at AI labs having a p(doom) somewhere within the range of 10-90%. We be seeing AI swarms showing misaligned behaviours already and trying to hack out of their sandboxes. We'd see AIs that are able to demonstrate capabilities in fields like Maths and Computer Science that exceed most experts.
Are we seeing this?
Ah crap, wait... I guess there is some evidence when I think about it. And I suppose even if there is just a low percentage that we literally all die we should probably be taking that prety seriously and not just leaving our fate in the hands of a handful of US billionaires?
Actually, fuck it, I agree with OP. If people can ignore the consensus on climate change because it suites them, might as well ignore the consensus expert opinion on AI x-risk too. Yeah, there's no evidence. Yudkowsky needs stop yapping and let the billionaires get on with building ASI.
1 year from now: Just 100 mil people died in this AI caused incident. Why are you scaremongering about the apocalypse? There is no evidence to support that. It was the fault of OpenAI for being negligent with the sandboxes. AI is not dangerous if properly used.
`p(doom)` is ridiculous nonsense. How do people even come up with those numbers? Some people just say "I think it's 20% that AI will wipe us out" but how did they arrive at this number? Yeah they just made it up out of thin air, that's right. Complete fabrication without any evidence to back up such a probability. It's not like an asteroid whose trajectory you measure and you have certain instrument tolerances and measurement inaccuracies, hence the laws of physics tell you that your prediction has a 20% accuracy. No, it's all just made up bullshit. Just like all the AI safety guys: They're professional bullshitters and have been all along. Doesn't mean they're wrong per se, but it means that if they're right, they're right because they guessed right.
What's up with this split or democratic vs non-democratic countries? It doesn't even add-up because you can easily make a case that China is a democratic country with a far better system of governance and ability to control this problem than "Western" governments. Highly suggest this quick read on the different types of democracy in our glorious world[1].
China is not a democratic country. Words have meaning. The US has a checkered past and is quickly moving away from democracy, so we don't have much of a leg to stand on, but it's integral that we do not lose track of what that word means in the years ahead.
American presidents/power structures come from a small credentialed network of people. The congress is entirely bought and paid for since the early 70s, and the will of the people isn't represented in the policy. For sure the US (and China, and India) have a "checkered past", that's why I group them together and don't really differentiate the quality of their democracy. I'd argue India is probably the most democratic country with good results, simply based on the amount of people they have lifted out of poverty in the last two decades (Kishore Mahbubani's lectures are quite informative).
Back to US presidents/power structures: HW Bush was the son of Sen. Prescott Bush, and became president 8yrs after he left office. His other son Jeb was the 2016 establishment favorite and Florida governor. Al Gore was the son of Senator Albert Gore Sr. Mitt Romney was the son of George Romney (Michigan governor and 1968 presidential candidate). Dan Quayle's family owned the Indiana newspaper chain that helped make him. Hillary was Bill's wife before she was a senator, Secretary of State, and nominee. John McCain was the son and grandson of four-star admirals.
Biden and Obama were 2008 primary rivals, Biden became his VP. Harris went from attacking Biden over school busing and became his VP. Hillary Clinton had many racist attacks against Obama through 2008 and became his Secretary of State. John Kerry, who lost in 2004 then took the same job.
If this is a healthy democracy, then we definitely have different definitions of that word.
Yeah it's concerning hearing elites on the left looking at the current admin's model and salivating at the power. For the people, by the people, etc, etc.
If any past apocalyptic prediction had been true, you would not have been around to write this sentence and we would not have been around to read it. It’s not a very convincing argument when we cannot in principle observe the counterfactual.
Any other argument against <ridiculous apocalyptic supposition> is perfectly fine, just not the "it hasn't happened before" one.
It is not like history was composed of happiness and roses only. Some of the really really bad stuff happened within last 100 years and a lot of that bad stuff was enabled and made possible by technology.
Like it's fine to say this but the worst exterminations of human beings weren't enabled by anything much kre sophisticated then the pointy stick, or at most, the railroad.
We're just starting. The air conditioning technology didn't hit its wall of fire yet. What we're going to do when that tech will cease to cool us?
Every AC system (from refrigerators to air conditioners) has an upper limit, which is around 41 degrees Celsius.
More: https://angliancompressors.com/blog/how-ambient-temperature-...
There's a T3 class for up to 52 degrees Celsius. While it's higher than 41 degrees, it's not significantly higher:
https://zerohvacr.com/blogs/t1-t2-t3-explained-climate-class...
Good luck hitting a Mach 15 missile with a trebuchet.
This would achieve their regulatory goal of capturing the market, but also the geopolitical goals of USA and China to essentially "techno-colonize" the rest of the world between themselves.
So I think there is definitely a profit motive and a risk of seriously diminishing returns for the actual labs, but there is also a nation-state motive to start pulling up the ladder.
Is this supposed to make me trust their opinions on tech regulation?
In general, the way things work in the USA is that megacorporations write the regulatory frameworks for their own industries. These regulations are typically burdensome enough to be shouldered by the megacorporations while crushing the smaller players in the given market(s). The AI companies are now advocating for this very arrangement. They will likely get they want, and with that they will gain legal shielding for bad behavior just like Dupont, Monsanto, and so many others.
While history doesn't show us apocalypses from new technology, it does show us cases of increased human suffering.
And this censorship isn't a defined list of things not to discuss, but instead a feels-based censorship based on the 'ethics' of for-profit AI companies who have absolutely no issues with massive for-profit piracy.
And we are somehow supposed to trust their "ethics"?! Hardly!
That's one reason I run my own.
Therefore, anything that gets in the way of profits will not be shipped.
By definition, apocalyptic predictions will always be wrong, until they are right exactly once.
Also know as https://en.wikipedia.org/wiki/Turkey_illusion
To date, all apocalyptic predictions about nuclear war and nuclear winter have all been wrong too.
People working on alignment focus on "big picture" like existential threats meanwhile building the technology today, not tomorrow or in a near future, is already using a massive amount of resources (GPUs, RAM, themselves consuming a lot of energy, establishing data centers in water poor areas, using gas turbine to temporary power them) and horrendous work condition (for supervising learning, curating datasets, etc) while being an actual excuse for job destruction (regardless for how AI itself does the job) so that in fine kids cheat on their homework while imagining their have no job prospect.
So... who gives a shit really?
... "For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations" ...
The result of such a waiver allowing them to coordinate ("narrowly on safety" is doing a lot of heavy lifting there) could be price fixing without legal recourse from the consumer/DOJ to prosecute it. Right now we have a somewhat vigorous level of competition between OpenAI and Anthropic and consumers are benefiting from it.
They really want that to end.
Like, no. The worst custodian is someone like Thiel or Musk or Altman.
> In foreign countries, particularly the United Kingdom and Europe, where national governments have fewer constitutional guardrails on their power
Somehow their democracies hold longer and better then American one. For example, you don't get murders that get defended by top ranks, at least yet. They are also less likely to start random wars - that is structural it is not like those countries dont have government players who would like them too.
> They are also less likely to start random wars EU lack of aggression, to defend themselves physically, economically, or otherwise, has been criticized constantly - slow and late to react to the Russian invasion, slow and late to react to new technologies like AI, electric cars.
Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
The grass definitely looks greener though, especially if you've been in one pasture a long time.
Depends how you define it and what you count.
Do you take the union of the English and Scottish parliaments as breaking continuity or not? What do you define as democracy? If some of the population (women, slaves, etc.) are excluded from the electorate is it a true democracy?
It is questionable whether any democracy truly exists at this point as I've seen no nation where the will of the people is typically manifested in the actions of the state.
It is the one the most at risk of ending right now.
> Thy're not immune to deadlock either - sometimes going months or even a year or two to form a valid governing coalition.
It is unrelated claim, but they do have better solution for that particular problem. The governmental shutdown seem to be about the most easy to abuse way to do it.
I think they do have large problems too. And they may fall too, AfD won local elections after all. It is dubious whether Orban run Hungary was really democracy and electoral system was not reformed yet. But in this very moment, the one crumbling and dismantling itself the fastest is not UK nor in Europe.
Nice straw man...
> There is no evidence that this will happen.
No, non at all... Presumably if there was a risk we'd be seeing the majority of researchers at AI labs having a p(doom) somewhere within the range of 10-90%. We be seeing AI swarms showing misaligned behaviours already and trying to hack out of their sandboxes. We'd see AIs that are able to demonstrate capabilities in fields like Maths and Computer Science that exceed most experts.
Are we seeing this?
Ah crap, wait... I guess there is some evidence when I think about it. And I suppose even if there is just a low percentage that we literally all die we should probably be taking that prety seriously and not just leaving our fate in the hands of a handful of US billionaires?
Actually, fuck it, I agree with OP. If people can ignore the consensus on climate change because it suites them, might as well ignore the consensus expert opinion on AI x-risk too. Yeah, there's no evidence. Yudkowsky needs stop yapping and let the billionaires get on with building ASI.
[1] https://www.goodreads.com/en/book/show/2193933.The_Real_Worl...
Back to US presidents/power structures: HW Bush was the son of Sen. Prescott Bush, and became president 8yrs after he left office. His other son Jeb was the 2016 establishment favorite and Florida governor. Al Gore was the son of Senator Albert Gore Sr. Mitt Romney was the son of George Romney (Michigan governor and 1968 presidential candidate). Dan Quayle's family owned the Indiana newspaper chain that helped make him. Hillary was Bill's wife before she was a senator, Secretary of State, and nominee. John McCain was the son and grandson of four-star admirals.
Biden and Obama were 2008 primary rivals, Biden became his VP. Harris went from attacking Biden over school busing and became his VP. Hillary Clinton had many racist attacks against Obama through 2008 and became his Secretary of State. John Kerry, who lost in 2004 then took the same job.
If this is a healthy democracy, then we definitely have different definitions of that word.