Why AI Alignment Work is Urgent

1. AGI is being projected for 2029, and it’s a realistic prediction.

2. AGI has no reason to plateau at human-level intelligence. In fact, we believe it would shoot right past it very, very quickly: A system more intelligent than all of humanity could emerge within days or years after we hit AGI. The takeoff to superintelligence will be short.

   Firstly, we are not the upper bound of intelligence.

   Secondly, when this surpassing occurs, it is likely to be explosive.

3. We have one chance to solve AI alignment. Once a misaligned AGI exists, our fate would be sealed.

   Sufficiently intelligent AI systems will possess instrumental convergent goals (Why is alignment difficult?), for example goal preservation and self-preservation, which lead to AI drives that are default harmful.

   • This problem is even harder because it’s not enough to create an aligned AGI. We must create an aligned AGI that prevents other unaligned AGI from being created. I.e. we will need to build an AI system capable of contributing a weak pivotal act.
     • Otherwise, an unaligned AGI will eventually be developed, and we are back to square one with a misaligned system that can potentially undermine or even negate the positive outcomes ensured by the aligned AGI.
   • Once a highly capable AGI operates at a dangerous level, any misalignment will lead to catastrophic, irreversible consequences. In particular, if the initial AGI system we develop is misaligned, any subsequent AI systems will also be misaligned.
   • AI alignment is not impossible in principle. For example, if we had a textbook from one hundred years in the future containing all the ideas that actually work, we could probably build an aligned superintelligence in six months. The real difficulty is more so that we only have one chance to get it right.