Viewing a single comment thread. View all comments

Mrkvitko t1_je8ajsn wrote

Most people mention air attacks on the datacenters as the most controversial point, and miss the paragraph just below. > Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.

That is downright insane. The ASI might kill billions, assuming:

  1. it is possible for us to create it
  2. we will actually create it
  3. it will be initially unaligned
  4. it will want to kill us all (either by choice or by accident)
  5. it will be able to gain resources to do so
  6. we won't be able to stop it

Failure at any of these steps means nobody is going to die. And we don't know how big is the probability of each of the steps succeeding or failing.

We however know that nuclear exchange will certainly kill billions. We know the weapon amounts and yields, we know their effect on human bodies.

If you argue it's better to certainly kill billions and destroy (likely permanently) human civilization over the hypothetical that you will kill billions and destroy human civilization, you're at best deranged lunatic, and evil psychopath at worst.

19

Spire_Citron t1_je8hn6c wrote

Especially since AI has the potential to make incredible positive contributions to the world. Nuclear war, not so much.

5

Mrkvitko t1_je8im10 wrote

Nuclear war is probably extinction event for all / most life on earth in the long term anyways. Modern society will very likely fall apart. Because post-war society will no longer have cheap energy and resources available (we already mined those easily accessible), it won't be able to reach technological level comparable to ours.

Then all it takes is one rogue asteroid, or supervolcano eruption. Advanced society might be able to prevent it. Middle-ages one? Not so much.

1

monsieurpooh t1_je95k4w wrote

You don't need ASI for an AI extinction scenario. Probably skynet from terminator can be reenacted with something that's not quite AGI combined with a few bad humans

3

blueSGL t1_je8q3lm wrote

> 3. it will be initially unaligned

if we had:

  1. a provable mathematical solve for alignment...

  2. the ability to directly reach into the shogoths brain, watch it thinking, know what it's thinking and prevent eventualities that people consider negative outputs...

...that worked 100% on existing models. I'd be a lot happier about our chances right now.

As in the fact that the current models cannot be controlled or explained in fine grain enough detail (the problem is being worked on but it's still very early stages) what makes you think making larger models will make them easier to analyze or control.

The current 'safety' measures are bashing at a near infinite whack-a-mole board whenever it outputs something deemed wrong.

As has been shown. OpenAI has not found all the ways in which to coax out negative outputs. The internet contains far more people than OpenAI's alignment researches, and those internet denizens will be more driven to find flaws.

Basically until the AI 'brain' can be exposed and interpreted and safety check added at that level we have no way of preventing some clever sod working out a way to break the safety protocols imposed on the surface level.

1

Shack-app t1_je8rcb6 wrote

Who’s to say this will ever happen?

2

blueSGL t1_je8saz1 wrote

what will ever happen? Interpretability? it's being worked on right now, there are already some interesting results. It's just an early field that will need time and money and researches put into it. Alignment as a whole needs more time money and researchers.

1

GorgeousMoron OP t1_je8jurv wrote

"Willing to run some risk" and "calling for" are not equivalent. There are actually some pretty strong arguments being made by academics that ASI will now likely than not fail to work out in our favor in perhaps spectacular fashion.

−3