Nick Bostrom

Paths, Dangers, Strategies

Superintelligence

里森

In what sense is humanity a dominant species on Earth? Ecologically speaking, humans are the most common large (~50 kg) animal, but the total human dry biomass (~100 billion kg) is not so impressive compared with that of ants, the family Formicidae (300 billion–3,000 billion kg). Humans and human utility organisms form a very small part (<0.001) of total global biomass. However, croplands and pastures are now among the largest ecosystems on the planet, covering about 35% of the ice-free land surface (Foley et al. 2007). And we appropriate nearly a quarter of net primary productivity according to a typical assessment (Haberl et al. 2007), though estimates range from 3% to over 50% depending mainly on varying definitions of the relevant terms (Haberl et al. 2013). Humans also have the largest geographic coverage of any animal species and top the largest number of different food chains.

Anthropomorphic frames encourage unfounded expectations about the growth trajectory of a seed AI and about the psychology, motivations, and capabilities of a mature superintelligence.

It is no part of the argument in this book that we are on the threshold of a big breakthrough in artificial intelligence, or that we can predict with any precision when such a development might occur. 

The economist Robin Hanson estimates, based on historical economic and population data, a char- acteristic world economy doubling time for Pleistocene hunter-gatherer soci- ety of 224,000 years; for farming society, 909 years; and for industrial society, 6.3 years.

Let an ultraintelligent machine be defined as a machine that can far surpass all the inte!- lectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion" and the inteligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

For instance, to prove a theorem that has a 5-line long proof in a deduction system with one inference rule and 5 axioms, one could simply enumerate the 3,125 possible combinations and check each one to see if it delivers the intended conclusion. Exhaustive search would also work for 6- and 7-line proofs. But as the task becomes more difficult, the method of exhaustive search soon runs into trouble. Proving a theorem with a 50-line proof does not take ten times longer than proving a theorem that has a 5-line proof: rather, if one uses exhaustive search, it requires combing through 5^5 ≈ 8.9 x 10^34 possible sequences—which is computationally infeasible even with the fastest supercomputers.

For many applications, however, the learning that takes place in a neural network is litle difer- ent from the learning that takes place in linear regression, a statistical technique developed by Adrien-Marie Legendre and Carl Friedrich Gauss in the early 1800s.

 Nets lacking hidden layers had previously been shown to have severely limited functionl (Minsky and Papert 1969).

For example, many types of artificial neural network can be viewed as classifiers that perform a particular kind of statistical calculation (maximum likelihood estimation). This perspective allows neural nets to be compared with a larger class of algorithms for learning classifiers from examples—"decision trees," "logistic regression models," "support vector machines," "naive Bayes, "k-nearest-neighbors regression," among others." In a similar manner, genetic algorithms can be viewed as performing stochastic hill-climbing, which is again a subset of a wider class of algorithms for optimization.

Accordingly, one can view artificial intelligence as a quest to find shortcuts: ways of tractably approxi- mating the Bayesian ideal by sacrificing some optimality or generality while pre- serving enough to get high performance in the actual domains of interest

Superintelligence quotes

Join the club