Nick Bostrom

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence.

Paths, Dangers, Strategies

Superintelligence

里森

In the worst-case scenario, all teams have equal levels of capability. The winner is then determined exclusively by investment in safety: the team that took the fewest safety precautions wins.

For most of our species’ existence, macro-structural development was slower than it is now. Fifty thousand years ago, an entire millennium might have elapsed without a single significant technological invention, without any noticeable increase in human knowledge and understanding, and without any globally meaningful political change.

The ground for preferring superintelligence to come before other potentially dangerous technologies, such as nanotechnology, is that superintelligence would reduce the existential risks from nanotechnology but not vice versa.

[Coherent Extrapolated Volition] can thus be seen as an application of the principle of epistemic deference.

A future superintelligence occupies an epistemically superior vantage point: its beliefs are (probably, on most topics) more likely than ours to be true. We should therefore defer to the superintelligence’s opinion whenever feasible.

In one informal experiment, the role of AI was played by an intelligent human. Another individual played the role of gatekeeper and was tasked with not letting the AI out of the box. The AI could communicate with the gatekeeper only by text and was given two hours to persuade the gatekeeper to let it out. In three cases out of five, with different individuals playing the gatekeeper, the AI escaped (Yudkowsky 2002). What a human can do, a superintelligence can do too.

[When] dumb, smarter is safer; yet when smart, smarter is more dangerous.

[The] first superintelligence may shape the future of Earth-originating life, could easily have non-anthropomorphic goals, and would likely have instrumental reasons to pursue open-ended resource acquisition.

While survival in the sense of instance-continuity may not.

A population of [artificially generally intelligent] agents […] might be better individuated as 

, based on their values, rather than on the basis of bodies, personalities, memories, or abilities. In such scenarios, goal-continuity might be said to 

Example of a system with non-linear recalcitrance.

The process of solving a jigsaw puzzle starts out simple—it is easy to find the corners and edges. Then recalcitrance goes up as subsequent pieces are harder to fit. But as the puzzle nears completion, the search space collapses and the process gets easier again.

Agents with large mental speedups who want to converse extensively might find it advantageous to move near one another. Extremely fast minds with need for frequent interaction (such as members of a work team) may take up residence in computers located in the same building to avoid frustrating latencies.

To such a fast mind, events in the external world appear to unfold in slow motion. Suppose your mind ran at 10,000x. If your fleshy friend should happen to drop his teacup, you could watch the porcelain slowly descend toward the carpet over the course of several hours, like a comet silently gliding through space toward an assignation with a far-off planet; and, as the anticipation of the coming crash tardily propagates through the folds of your friend's gray matter and from thence out into his peripheral nervous system, you could observe his body gradually assuming the aspect of a frozen oops—enough time for you not only to order a replacement cup but also to read a couple of scientific papers and take a nap.

Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization--a niche we filled because we got there first, not because we are in any sense optimally adapted to it. 

No fundamental conceptual or theoretical breakthrough is needed for whole brain emulation to succeed. 

At best, the evolution of intelligent life places an upper bound on the intrinsic difficulty of designing intelligence. 

That didn’t age well: Deep Blue beats Kasparov in '97. Hofstadter continues: “That may be the kind of dialogue you could have with a program that could beat anyone. That is because real intelligence inevitably depends on a total overview capacity—that is, a programmed ability to “jump out of the system”, so to speak—at least roughly to the extent that we have that ability. Once that is present, you can’t contain the program; it’s gone beyond that certain critical point, and you just have to face the facts of what you’ve wrought.” (Hofstadter, D. (1999). p. 678). Turns out MinMax search does the trick.

In 1979, Douglas Hofstadter opined in his Pulitzer-winning Gödel, Escher, Bach: “Question: Will there be chess programs that can beat anyone? Speculation: No. There may be programs that can beat anyone at chess, but they will not be exclusively chess programs. They will be programs of general intelligence, and they will be just as temperamental as people. Do you want to play chess?' No, I'm bored with chess. Let's talk about poetry” (Hofstadter [1979] 1999, 678).

As soon as certain technologies have practical use, nobody thinks of them as dystopian anymore. People roll their eyes at “science fiction nonsense,” then unlock their phones by having thousands of infrared dots projected onto their faces so they can proceed to use an infinite scroll app to have their dopaminergic system exploited by algorithms and to read bot content they can't reliably tell from human content…

In the view of several experts in the late fifties: “If one could devise a successful chess machine, one would seem to have penetrated to the core of human intellectual endeavor." This no longer seems so. One sympathizes with John McCarthy, who lamented: ”As soon as it works, no one calls it AI anymore.”

Accordingly, one can view artificial intelligence as a quest to find shortcuts: ways of tractably approxi- mating the Bayesian ideal by sacrificing some optimality or generality while pre- serving enough to get high performance in the actual domains of interest

For example, many types of artificial neural network can be viewed as classifiers that perform a particular kind of statistical calculation (maximum likelihood estimation). This perspective allows neural nets to be compared with a larger class of algorithms for learning classifiers from examples—"decision trees," "logistic regression models," "support vector machines," "naive Bayes, "k-nearest-neighbors regression," among others." In a similar manner, genetic algorithms can be viewed as performing stochastic hill-climbing, which is again a subset of a wider class of algorithms for optimization.

 Nets lacking hidden layers had previously been shown to have severely limited functionl (Minsky and Papert 1969).

For many applications, however, the learning that takes place in a neural network is litle difer- ent from the learning that takes place in linear regression, a statistical technique developed by Adrien-Marie Legendre and Carl Friedrich Gauss in the early 1800s.

For instance, to prove a theorem that has a 5-line long proof in a deduction system with one inference rule and 5 axioms, one could simply enumerate the 3,125 possible combinations and check each one to see if it delivers the intended conclusion. Exhaustive search would also work for 6- and 7-line proofs. But as the task becomes more difficult, the method of exhaustive search soon runs into trouble. Proving a theorem with a 50-line proof does not take ten times longer than proving a theorem that has a 5-line proof: rather, if one uses exhaustive search, it requires combing through 5^5 ≈ 8.9 x 10^34 possible sequences—which is computationally infeasible even with the fastest supercomputers.

Let an ultraintelligent machine be defined as a machine that can far surpass all the inte!- lectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion" and the inteligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

The economist Robin Hanson estimates, based on historical economic and population data, a char- acteristic world economy doubling time for Pleistocene hunter-gatherer soci- ety of 224,000 years; for farming society, 909 years; and for industrial society, 6.3 years.

Richu A Kuttikattu

Hated reading it, absolutely love having read it.

mimi nguyen

Frederik Creemers

Sometimes, a book, or something someone says, changes the way you think. This book did just that for me, regarding AI. It made me see the flawed logic in thinking that superintelligence would never exist.

Hernando

Just a few annotations:

"A big breakthrough in artificial intelligence. It seems somewhat likely that it will happen sometime in this century, but we don't know for sure".

-I don't think so, but I truly hope to see that with my eyes.

"The idea of a coming technological singularity has been popularized, starting with Vernor Vinge's seminal essay and continuing with the writings of Ray Kurzweil and others. The term 'Singularity,' however, has been used confusedly in many disparate senses and has accreted an unholy aura of techno-utopian connotations".

-Agree.

"Machines matching humans in general intelligence have been expected since the invention of computers in the 1940s"

-Add another century to it.

"The fact that the best performance at one time is attained through a complicated mechanism does not mean that no simple mechanism could do the job as well or better. It might simply be that nobody has yet found the simpler alternative".

-That's my bet for developing a SI.

"How far are we currently from achieving a human whole brain emulation?. One recent assesment presented a technical roadmap and concluded that the prerequisite capabilities might be available around mid-century, though with a large uncertainty level".

- I also doubt a WBE in the near future. Probably in 2120?.

"Final goal: 'make us smile'.
Perverse instantiation: 'paralyze human facial musculatures into constant beaming smiles".

- I laughed on this because it makes sense.

"...These observations make it plausible that any type of entity that developed a much greater than human level intelligence would be potentially extremely powerful. Such entities could accumulate content much faster than us and invent new technologies on a much shorter timescale. They could also use their intelligence to strategize more effectively than we can".

- I liked this book a lot specially because of the Realistic Pessimism about an SI. I consider there must be a real concern to think about that level of intelligence.

Gavin

Like a lot of great philosophy, Superintelligence acts as a space elevator: you make many small, reasonable, careful movements - and you suddenly find yourself in outer space, home comforts far below. It is more rigorous about a topic which doesn't exist than you would think possible. 

I didn't find it hard to read, but I have been marinating in tech rationalism for a few years and have absorbed much of Bostrom secondhand so YMMV.

I loved this:

  Many of the points made in this book are probably wrong. It is also likely that there are considerations of critical importance that I fail to take into account, thereby invalidating some or all of my conclusions. I have gone to some length to indicate nuances and degrees of uncertainty throughout the text — encumbering it with an unsightly smudge of “possibly,” “might,” “may,” “could well,” “it seems,” “probably,” “very likely,” “almost certainly.” Each qualifier has been placed where it is carefully and deliberately. Yet these topical applications of epistemic modesty are not enough; they must be supplemented here by a systemic admission of uncertainty and fallibility. This is not false modesty: for while I believe that my book is likely to be seriously wrong and misleading, I think that the alternative views that have been presented in the literature are substantially worse - including the default view, according to which we can for the time being reasonably ignore the prospect of superintelligence.
 
Bostrom introduces dozens of neologisms and many arguments. Here is the main scary apriori one though:

1. Just being intelligent doesn't imply being benign; intelligence and goals can be independent. (the orthogonality thesis.)
2. Any agent which seeks resources and lacks explicit moral programming would default to dangerous behaviour. You are made of things it can use; hate is superfluous. (Instrumental convergence.) 
3. It is conceivable that AIs might gain capability very rapidly through recursive self-improvement. (Non-negligible possibility of a hard takeoff.)
4. Since AIs will not be automatically nice, would by default do harmful things, and could obtain a lot of power very quickly*, AI safety is morally significant, deserving public funding, serious research, and international scrutiny.

Of far broader interest than its title (and that argument) might suggest to you. In particular, it is the best introduction I've seen to the new, shining decision sciences - an undervalued reinterpretation of old, vague ideas which, until recently, you only got to see if you read statistics, and economics, and the crunchier side of psychology. It is also a history of humanity, a thoughtful treatment of psychometrics v genetics, and a rare objective estimate of the worth of large organisations, past and future.

Superintelligence's main purpose is moral: he wants us to worry and act urgently about hypotheticals; given this rhetorical burden, his tone too is a triumph. 

  For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens. Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the firmament. Nor is there a grown-up in sight... 

This is not a prescription of fanaticism. The intelligence explosion might still be many decades off in the future. Moreover, the challenge we face is, in part, to hold on to our humanity: to maintain our groundedness, common sense, and goodhumored decency even in the teeth of this most unnatural and inhuman problem. We need to bring all human resourcefulness to bear on its solution. 


I don't donate to AI safety orgs, despite caring about the best way to improve the world and despite having no argument against it better than "that's not how software has worked so far" and despite the concern of smart experts. This sober, kindly book made me realise this was more to do with fear of sneering than noble scepticism or empathy.

[EDIT 2019: Reader, I married this cause.]


* People sometimes choke on this point, but note that the first intelligence to obtain half a billion dollars virtually, anonymously, purely via mastery of maths occurred... just now. Robin Hanson chokes eloquently here and for god's sake let's hope he's right.

Jimmy Cerone

More Philosophy than Tech 

I loved the big questions this book asked as I think it revealed some important issues with Superintelligence that I hadn’t considered. That said, this book could have been 50 pages shorter. The author tried to cover all his bases and it was utterly overwhelming - I barely finished the book.

Ben Jenkins

Relevant, calmly rigorous, terrifying. Accessible but not a casual read.

Yasser Yahia

The book is full of assumptions, theories, alternatives, and premises, but it analyzes each of them and tries to explore all the paths. The author's bias is clear (which is something that I prefer), yet he is using logic and examples from our current reality or past history to make his point.

Overall I enjoyed the book and it was aligned with a lot of what I believe is possible.

Danté

A sober take on the rise and rise of AI. 

Main takeaways:
- Different types of AI (neuromorphic - mimicing human brain activity and computational AI that is unique to computers). 
- It's very difficult to encode morality into a machine because our morality is vague. There are different approaches though, including one saying that we should behave as our higher intellectual selves would want us to behave (but that sidesteps the question). 
- Ramp up speed for an AI is a key factor - depending on how fast it happens we might not have time to react.
- Generally Bostrom views superintelligent AI as a bad thing for humanity; a critical event that would only have to happen once to signal our downfall.

Michael Pejin

As someone who wasn't overly familiar with AI to begin with, this book was a rather dense read. 

There were many ideas expressed in this that gave me a lot to think about, and truly admire. Bostram's in depth look at neurological structure and referencing that to the expression of super computers/intelligence was awe inspiring and has left me with wanting to look into the subject further. It is clear he is passionate about the topic, and put a great deal of effort in making sure the information was well researched and thoroughly expressed.

At times though, Bostram's writing got quite clunky, and was filled with terms and concepts that required a lot of referencing to really get the most out of his ideas. It's difficult for me to say whether that is my failing due to my limited understanding of the topic, or if it is a lack in clarity of the prose.

Bryan Alexander

Superintelligence is a grand-scale book, written at the level of human destiny. Nick Bostrom invites us to think at a cosmic scale, while contemplating a vast range of possible transhuman futures.

The focus of the book is the ways an artificial intelligence could grow into something greatly surpassing human intellect and control. It's a classic science fiction theme (think HAL, Colossus, Skynet, and of course the origin in Frankenstein); what Bostrom adds is considering the problem philosophically. That approach is Anglo-American analytical philosophy, not continental, which is a pleasant change of pace for me. This means many considerations of ethics, frequent definitional explorations, and many divisions of concepts into subcategories.

Frustratingly, Bostrom's discussion feels at times fruitless. When he breaks down superintelligence takeoff rates, for example, the reader might shrug, given the huge dependence on so many variables we don't know now and which the author doesn't settle. "It depends" seems to be the implicit conclusion of too many chapters.

This is a book rich with ideas. Superintelligence tosses off concepts like a speed-addled science fiction writer: AIs turning humans into paperclips, or transforming planets and stars into computational substrates based on varying information architectures. I appreciated the many forms meta-human intelligence could take.

Bostrom parallels his exploration of superinteligent AI with related human structures. He considers the possibility of massively augmenting human intelligence throughout the book, while pondering human organization in like manner. For example, a later passage posits a human singleton in order to counter an AI singleton, "a global superintelligent Leviathan" (182). Bostrom's discussions of how to uplift humanity is breathtaking, and also chilling.

Yet Bostrom leavens his reflections with very entertaining, sometimes visionary or disturbing passages.
The bouillon cubes of discrete human-like intellects thus melt into an algorithmic soup. (172)
And so we boldly go - into the whirling knives. (118)
A mere line in the sand, backed by the clout of a nonexistent simulator, could prove a stronger restraint than a two-foot-thick solid steel door. (135) (from a chapter positing the fun idea of "anthropic capture")
The universe then gets filled not with exultingly heaving hedonium but with computational processes that are unconscious and completely worthless - the equivalent of a smiley-face stricker xeroxed trillions upon trillions of times and plastered across the galaxies.(140) ("exultingly heaving hedonium"!)


It's hard to issue a recommendation for this book. It really appeals to a very narrow set of readers, people interested in transhumanism and willing to work through British-style philosophical discourse. For those people it's a rewarding read.
It may also be productive for science fiction writers, hunting ideas.

For the general public, eh, this leans too much in the specialist direction.

Superintelligence Paths, Dangers, Strategies

Reviews

Highlights