Preface to the Computational Agent Theory and ASI Alignment Sequence
Provides theoretical foundation for researchers working on superintelligence safety and alignment problems.
AI Summary
Preface introduces computational agent theory framework for analyzing ASI risks and existential threats.
Excerpt
The idea of “artificial superintelligence [1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001 [2] , and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Be
