Why Superintelligent AI is Dangerous by Default

“The principal reason for humanity’s dominant position on Earth is that our brains have a slightly expanded set of faculties compared with other animals.” - Nick Bostrom

Intelligence is the single quality we have an advantage over all other animals. How will we fare when this is not the case?

But how is this different than machines we’ve already built that are better than us at tasks?

Consequences

Why should a superintelligent AI concern you any more than the fact that mechanical vehicles can go faster than a human can run?

The fundamental difference between superintelligent AI and mechanical vehicles is the generality of their capabilities.

Chess engines and cars don’t threaten our status as apex predators because they have narrow intelligence. They are better than us at one thing, chess and movement speed, respectively, but their set of overall abilities is very small.

Superintelligence, on the other hand, will have extremely broad intelligence and will be able to operate completely autonomously.

Okay, even if that’s true, how would a chatbot take over the world?

Abilities

How could a computer program gain or wield power?

A computer program doesn't actually serve as a constraint for any generally intelligent agent.

But why should we expect it to use its power to the detriment of humankind?

Destructiveness

Wouldn’t a superintelligent AI be smart enough to use its power responsibly?

In short, no. An AI system will use its power in a way to maximize its goals regardless of the nature of its goals (and as we’ve seen, it is difficult to choose good goals). A superintelligent AI would simply be better at doing so.

This means that even if the AI system were many times more intelligent than humans, this would guarantee absolutely nothing about its final goals.

Moreover, even if the AI system’s final goals seem safe, it is likely that it will share a lot of intermediate goals in common with a misaligned superintelligence.

Hence, it is not enough to align an AI system's final goals alone; we must also align its intermediate goals, which by default will be identical to those of a misaligned system.