Transcript Document

?
8th
European conference on Computing And Philosophy—ECAP, 4 – 6 Oct. 2010, Munich, Germany
?
?
?
?
SingularityHypothesis.blogspot.com
?
?
?
The Singularity Hypothesis
?
?
?
?
?
?
?
?
?
Amnon H. Eden
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
School of Computer Science & Electronic Engineering
University of Essex
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
1
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
http://singularityhypothesis.blogspot.com/p/central-questions.html? ?
CENTRAL QUESTIONS
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
?
???
??
??
?
??
??
?
?
??
??
?
?
??
??
??
?
?
?
?
?
?
?
?
?
?
2
?
?
Questions
?
?
?
1. What is the [technological] singularity hypothesis? What
exactly is being claimed?
?
?
?
2. What is the empirical content of this conjecture? Can the it be
refuted or corroborated, and if so, how?
?
?
?
?
?
3. What exactly is the nature of a singularity: Is it a discontinuity on
?
?
a par with phase transition or a process on a par with Toffler’s? ?
?
?
'wave'? Is the term singularity appropriate?
?
?
?
?
?
?
4. What evidence, taken for example from the history of technology
?
?
?
?
?
and economic theories, suggest the advent of some form
of
??
?
??
?
singularity by 2050?
?
??
?
??
??
??
5. What, if anything, can be said to be accelerating?
Which
??
?
??
?
???
? ? its existence? Which
?
evidence can reliably be said to support
?
???
? ??
?
?
???
metrics support the idea that
'progress'
is indeed accelerating?
?? ?
?
?
?
????
??? ??
?? ? ? ????
?? ?? ??? ?? ?? ?? ?? ??
??? ?????
? ? ??? ?
????
? ? ??? ?
? ? ???
???
3
?
?
Questions (cont.)
?
?
?
6. What are the most likely milestones ('major paradigm shifts') in the
countdown to the singularity?
?
?
7. Is the so called Moore's Law on a par the laws of thermodynamics? ?
?
How about the Law of Accelerating Returns? What exactly is the ?
?
nature of the change they purport to measure?
?
?
?
?
8. What are the necessary and sufficient conditions for an intelligence
?
?
?
explosion (a runway effect)? What is the actual likelihood of such
?
?
?
an event?
?
?
?
?
?
9. What evidence support the claim that machine intelligence
? has been
?
?
rising? Can this evidence be extrapolated reliably? ??? ??
?
?
??
??? ?? ?? ??
??
?
10. What are the necessary and sufficient conditions
machine
?
? for
??
?
??
??
intelligence to be considered to be on a par
with
that of humans'?
?
???
?
?
??
???
?
That is, what would it take for the???"general
educated opinion [to]
?
??
?
?
? ???
? ? ?? will be able to speak of machines
have altered so much? ?that
one
?
?
?
? ??? ?
???
?
?
?
?
?
?? ? ?
thinking
expecting to be contradicted" (Turing 1950)?
???? ? ? ?
?? ? ? ? ?without
??
?
?
??
??
??
??? ?? ?? ??
4
?
?
?
Questions (cont.)
?
?
?
11. What does it mean to claim that biological evolution will be replaced by technological ?
evolution? What exactly can be the expected effects of augmentation and enhancement, in
?
particular over our cognitive abilities? To which extent can be expect our transhuman?and
?
posthuman descendants to be different from us?
?
?
12. What evidence support the claim that humankind's intelligence quotient has been rising
?
("Flynn effect")? How these evidence relate to a more general claim about a rise in? the
'intelligence' of carbon-based life? Can these evidence be extrapolated reliably? ??
?
?
13. What are the necessary and sufficient conditions for a functioning whole brain
?
?
?
emulation (WBE) of a human? At which level exactly must the brain be emulated?
What
?
?
?
will be the conscious experience of a WBE? To which extent can they be ?said
to be
?
?
human?
?
?
?
??
?
14. What may be the consequences of the singularity? What may be its
effect on society, e.g.
??
?
?
in ethics, politics, economics, warfare, medicine, culture, arts,??the
? ? humanities, and
?
??
?
religion?
??
?
??
??
?
15. Is it meaningful to refer to past events as singularities?
If so, what can be learned from
???
??
?
?
???
them? Is it meaningful to claim a wider interpretation
of singularity as the emergence of
? ??
?
?
??
?
?
? ? ? ? in some specific domain of activity, eg, a
super human-level artificial intelligence
? ? ??
?
?
?
? ??? ?
? ? ? ? ? in face recognition, in car driving, etc?
?
?
?
?
?
singularity?in
chess
playing,
?? ? ?
5
???? ? ? ?
??? ?? ?? ?? ??? ?? ?? ??
?? ?? ?? ? ? ????
???
?
?
Call for chapters
?
?
?
The Singularity Hypothesis:
A Scientific and Philosophical Assessment
?
?
?
?
?
?
?
?
?
?
• Abstracts: 15 Jan. 2011
?
?
• Full chapter manuscripts: Sep. 2011
• Publication: Springer, 2012 (est.)
• More details:
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
SingularityHypothesis.blogspot.com
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
6
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
“SINGULARITY”
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
7
?
?
von Neumann: Singularity
?
?
?
The ever-accelerating progress of technology
... gives the appearance of approaching some
essential singularity in the history of the race
beyond which human affairs, as we know
them, could not continue [in Ulam 1958]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
8
?
?
Vinge: Singularities
?
?
?
A change comparable to the rise of human
life on Earth [“The coming technological
singularity” 1993]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
9
?
?
Kurzweil: Singularity
?
?
?
A future period during which the pace of
technological change will be so rapid, its
impact so deep, that human life will be
irreversibly transformed... so that the
changes [that will be brought] about will
appear to rupture the fabric of human
history [The Singularity Is Near, 2006]
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
10
?
?
Toffler: Wave
?
?
?
What is occurring now is, in all likelihood,
bigger, deeper, and more important than the
industrial revolution. ... nothing less than the
second great divide in human history, the
shift from barbarism to civilization. [Future
Shock, 1970]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
11
?
?
Toffler: Waves
?
?
?
The first wave of change—the Agricultural
Revolution—took thousands of years to play
itself out. The second wave—the [Industrial
Revolution], took a mere three hundred
years. Today history is even more
accelerative, and it is likely that the Third
Wave will sweep across history and complete
itself in a few decades. [The Third Wave,
1980]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
12
?
?
Hanson: Singularities
?
?
?
We call this transition the Industrial
Revolution, but that does not mean we
understand it well or even know precisely
how and why it arose. But whatever the
Industrial Revolution was, clearly it was an
event worthy of the name “singularity.”
[“Economics of the singularity,” IEEE
Spectrum 45:6, 2008]
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
13
?
?
Solomonoff: Singularity/outcome
?
?
?
While I hesitate to guess as to just when the
singularity will occur, I would be much
surprised if it took as much as 20 years. As
for the next 50 years of A.I., I feel that
predicting what will happen after this
singularity is much like guessing how the
universe was before the Big Bang—It’s a real
discontinuity! [“Machine Learning—Past
and Future,” 2006]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
14
?
?
Kurzweil: Singularity/outcome
?
?
?
Some would say that we cannot comprehend
[the singularity], at least with our current
level of understanding. For that reason, we
cannot look past its event horizon and make
complete sense of what lies beyond. This is
one reason we call this transformation the
Singularity. [The Singularity is Near 2006]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
15
?
?
Moore: No singularity
?
?
[The] Singularity will never occur… I am a
skeptic. I don't believe this kind of thing is
likely to happen, at least for a long time. And
I don't know why I feel that way. The
development of humans, what evolution has
come up with, involves a lot more than just
the intellectual capability. [IEEE Spectrum
2008]
?
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
16
?
?
Chalmers: Singularity hypothesis
?
?
?
One might think that the singularity would be of
great interest to academic philosophers, cognitive
scientists, and artificial intelligence researchers.
…
?
?
?
?
?
?
The argument for a singularity is one that we
should take seriously. And the questions
surrounding the singularity are of enormous
practical and philosophical concern. [“The
Singularity: A Philosophical Analysis,” 2010]
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
17
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
ACCELERATION
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
18
?
?
Toffler: Acceleration
?
?
?
Western society for the past 300 years has been caught up in a fire
storm of change. This storm, far from abating, now appears to be
gathering force. Change sweeps through the highly industrialized
countries with waves of ever accelerating speed and unprecedented
impact.
?
?
?
?
?
?
?
?
?
Much of what now strikes us as incomprehensible would be far less so??
?
if we took a fresh look at the racing rate that makes reality seem like?? a
?
?
kaleidoscope wild run. For the acceleration of change ... is a concrete
?
?
?
?
force.
?
?
??
?
?
The disturbing fact is that the vast majority of people, including
?
??
?
?
??
educated and otherwise sophisticated people who understand
?
?
??
?
??
intellectually that change is accelerating, have not
? ?internalized that
?
??
??
?
?
?
knowledge. [Future Shock, 1970]
???
???
?? ??
??? ?? ?? ?? ??? ?? ?? ??
?? ? ? ???? ? ?
??
?? ? ? ?
???? ? ? ?
??? ???
?? ? ? ?
? ??
?? ? ?
? ?? ?
???
??
???
???
??
19
?
?
Kurzweil: Acceleration
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
[The Singularity Is Near, 2006]
?
?
?
?
20
?
?
Kurzweil: Acceleration (2)
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
[The Singularity Is Near, 2006]
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
21
?
?
Sagan et al.: Acceleration
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
? ???
“Cosmic Calendar” [The
Dragons
of Eden: Speculations on the
?? ? ?
?
?
?
?
???
?
?
?
?
?
?? ? ?
???? ? ? ?
? ? ? ?Human
??
??
?
?
??
Evolution
of
Intelligence 1986]
??
??
??? ?? ?? ?? ??? ?? ?? ??
?
?
?
22
?
?
Hanson: Acceleration
?
?
Now look at the data for world product over the past 7,000
years, estimated by Bradford DeLong, an economic
historian at the University of California, Berkeley. ... For
most of that time, growth proceeded at a relatively steady
exponential rate, with a doubling of output about every
900 years. But within the past few centuries, something
dramatic happened: output began doubling faster and
faster, approaching a new steady doubling time of about
15 years. That's about 60 times as fast as it had been in the
previous seven millennia. [“Economics of the singularity”,
2008]
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
23
?
?
?
?
?
?
?
?
?
Toffler: Acceleration/outcome
?
?
?
In the three short decades between now and the 21st century, millions of ordinary,
psychologically normal people will face an abrupt collision with the future.
Citizen’s of the world’s richest and most technologically advanced nations, will
find it increasingly painful to keep up with the incessant demand for change that
characterizes our time. For them, the future will have arrived too soon. ....
?
?
?
?
?
?
?
?
Take an individual out of his own culture and set him down suddenly in an
?
environment sharply different from his own, with a different set of cues to react ??
?
?
to…then cut him off from any hope of retreat to a more familiar social landscape,
?
?
and the dislocation he suffers is doubly sever. Moreover, if this new culture ?is? itself
?
? the
in constant turmoil, and if—worse yet—its values are incessantly changing,
?
?
?
sense of disorientation will be still further intensified. ... The victim may
? become a
?
?
?
?
hazard to himself and others.
??
?
??
?
?
Now imagine not merely an individual but an entire society, an
? ?entire generation—
?
?
??
?
?
including its weakest ... members—suddenly transported? ?into
this new world. The
?
?
?
? ? ? scale.
result is mass disorientation, future shock on a grand
??
?
?
?
??
???
???
?
??
This is the prospect that man now faces.
Shock, 1970, p. 12]
? ?[Future
?? ? ?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
24
?
?
Modis: Acceleration?
?
?
?
All exponential curves that represent a real
growth process constitute part of some logistic
curve. [in R.U. Ayres “Book review: Ray
Kurzweil, The Singularity is Near”, 2006]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
25
?
?
Modis: Acceleration? (2)
?
?
All the data for the graphs of Chapter One [of The
Singularity is Near]… come from two articles of mine. The
data consist of fourteen sets of milestones in the evolution
of the universe, which I researched. But while I strived for
the data to come from independent sources I did not
succeed very well. … [Kurzweil] augments the number of
data sets by one adding the set from my second
publication—which is the average of 13 of the previous
data sets—and thus boasts evidence from 15 independent
sources! [in Ayres, 2006]
?
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
26
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
SUPERINTELLIGENCE
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
27
?
?
Descartes: Superintelligence
?
?
(i) Language: “[Machines] could never use
words or other signs arranged in such a
manner as is competent to us in order to
declare our thoughts to others.
(ii) Complexity: “it must be morally
impossible that there should exist in any
machine a diversity of organs sufficient to
enable it to act … in the way in which our
reason enables us to act.” [in Colburn
2003, p. 23]
?
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
28
?
?
Turing: Superintelligence
?
?
?
I believe that at the end of the century the use
of words and general educated opinion will
have altered so much that one will be able to
speak of machines thinking without expecting
to be contradicted. [“Computing Machinery
and Intelligence”, 1950]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
29
?
?
Dijkstra: Superintelligence
?
?
?
Alan Turing thought about criteria to settle
the question of whether machines can think, a
question of which we now know that it is
about as relevant as the question of whether
submarines can swim. [ACN South Central
Regional Conf., 1984]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
30
?
?
Searle: Superintelligence
?
?
Because we do not understand the brain very well we are
constantly tempted to use the latest technology as a model
for trying to understand it. In my childhood we were
always assured that the brain was a telephone
switchboard. ('What else could it be?') I was amused to see
that Sherrington, the great British neuroscientist, thought
that the brain worked like a telegraph system. Freud often
compared the brain to hydraulic and electro-magnetic
systems. Leibniz compared it to a mill, and I am told some
of the ancient Greeks thought the brain functions like a
catapult. At present, obviously, the metaphor is the digital
computer. [Minds, Brains and Science, p 44]
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
31
?
?
?
?
?
?
?
?
?
Bostrom: Superintelligence/outcome
?
?
When we create the first superintelligent entity, we
might make a mistake and give it goals that lead it to
annihilate humankind, assuming its enormous
intellectual advantage gives it the power to do so.
For example, we could mistakenly elevate a subgoal
to the status of a supergoal. We tell it to solve a
mathematical problem, and it complies by turning all
the matter in the solar system into a giant calculating
device, in the process killing the person who asked
the question. [“Existential Risks”, 2002]
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
32
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
INTELLIGENCE EXPLOSION
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
33
?
?
Good: Intelligence explosion
?
?
Let an ultraintelligent machine be defined as a machine
that can far surpass all the intellectual activities of any
man however clever. Since the design of machines is one of
these intellectual activities, an ultraintelligent machine
could design even better machines; there would then
unquestionably be an intelligence explosion, and the
intelligence of man would be left far behind. Thus the first
ultraintelligent machine is the last invention that man need
ever make ...
?
?
?
?
?
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
It is more probable ... that, within the twentieth century, an
ultraintelligent machine will be built and that it will be the
last invention that man need make. [“Speculations
Concerning the First Ultraintelligent Machine”, 1965] 34
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
?
?
?
?
?
?
?
?
Turing: Intelligence explosion
?
?
There would be plenty to do i.e. in trying to keep
one's intelligence up to the standards set by the
machines, for it seems probable that once the
machine thinking method has started, it would not
take long to outstrip our feeble powers. ... At
some stage therefore we should have to expect the
machines to take control, in the way that is
mentioned in Samuel Butler's 'Erewhon'.
[“Intelligent Machinery, A Heretical Theory,”
1951]
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
35
?
?
?
?
?
?
?
?
?
Vinge: Intelligence explosion
?
?
Good has captured the essence of the runaway,
but does not pursue its most disturbing
consequences. Any intelligent machine of the sort
he describes would not be humankind's "tool" —
any more than humans are the tools of rabbits or
robins or chimpanzees. [“The coming
technological singularity” 1993]
“Will computers ever be as smart as humans?”
“Yes, but only briefly”. [“Signs of the
Singularity,” 2008]
?
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
36
?
?
Chalmers: Intelligence explosion
?
?
Although there is a self-amplifying cognitive capacity G, either
we or our successors might not manifest our capacity to create
systems with higher values of G (or with higher values of a
cognitive correlated capacity H): ... the motivational defeaters in
which an absence of motivation or a contrary motivation prevents
capacities from being manifested.
?
?
?
?
?
?
?
?
Overall, I think that there is at least a good case that absent
defeaters, a number of interesting cognitive capacities will
explode. I think the most likely defeaters are motivational. But I
think that it is far from obvious that there will be defeaters. [“The
Singularity: A Philosophical Analysis”, 2010]
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
37
?
?
?
?
?
?
?
?
?
Hawking: Intelligence explosion
?
?
?
it seems to me that if very complicated chemical
molecules can operate in humans to make them
intelligent then equally complicated electronic
circuits can also make computers act in an
intelligent way. And if they are intelligent they
can presumably design computers that have even
greater complexity and intelligence. [“Science in
the Next Millennium: Remarks by Stephen
Hawking”, 1998]
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
38
?
?
Bostrom: Intelligence explosion
?
?
Once artificial intelligence reaches human level,
there will be a positive feedback loop that will
give the development a further boost. AIs would
help constructing better AIs, which in turn would
help building better AIs, and so forth. Even if no
further software development took place and the
AIs did not accumulate new skills through selflearning, the AIs would still get smarter if
processor speed continued to increase. [“How
Long Before Superintelligence?”, 1997/2008]
?
?
?
?
?
?
?
?
?
?
?
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
39
?
?
AAAI: intelligence explosion
?
?
The panel of experts was overall skeptical of the radical views
expressed by futurists and science-fiction authors. ... There was
overall skepticism about the prospect of an intelligence explosion
as well as of a “coming singularity,” and also about the largescale loss of control of intelligent systems.
?
?
?
?
?
?
?
?
?
?
?
Nevertheless, there was a shared sense that additional research
would be valuable on methods for understanding and verifying
the range of behaviors of complex computational systems to
minimize unexpected outcomes. [“AAAI Presidential Panel on
Long-Term AI Futures ('Asilomar Meeting'), 2009].
?
?
??? ?? ?? ?? ??? ?? ?? ??
??? ????
?? ?? ?? ? ? ????
? ? ? ??? ?
? ? ???
? ? ??
?????
? ? ? ??
??
?? ?
?? ? ?
???
???
??
???
? ??
??
??
??
?
?
??
??
?
?
??
??
?
?
??
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
40
?
?
References
?
AAAI. (2009, August). AAAI Presidential Panel on Long-Term AI Futures ('Asilomar Meeting'), Interim
?
Report from the Panel Chairs. Retrieved May 29, 2010, from http://research.microsoft.com/enus/um/people/horvitz/AAAI_Presidential_Panel_2008-2009.htm
?
Ayres, R. U. (2006). Book review: Ray Kurzweil, The Singularity is Near: When Humans Transcend
?
Biology, Viking Penguin, New York (2005) 602 pages plus index; $29.95. Technological Forecasting and ?
Social Change, 73(2), 95-127.
?
Bostrom, N. (1998). How long before superintelligence? International Journal of Futures Studies, 2.
?
?
Chalmers, D. J. (2010). The Singularity: A Philosophical Analysis. Retrieved from
?
http://consc.net/papers/singularity.pdf
?
?
Good, I. J. (1965). Speculations Concerning the First Ultraintelligent Machine. In F. L. Alt & M. ?
Rubinoff (Eds.), Advances in Computers (Vol. 6, pp. 31–88). Academic Press.
?
?
Hanson, R. (1994). What If Uploads Come First: The Crack of a Future Dawn. Extropy, 6(2).
?
?
?
Toffler, A. (1970). Future shock. Random House.
?
?
Toffler, A. (1980). The third wave (1st ed.). New York: Morrow.
?
?
?
?
Toffler, A. (2006). Revolutionary wealth : shaping tomorrow's way of life. New York: Knopf.
?
?
?
Turing, A. M. (1950). Computing Machinery and Intelligence. Mind, 59(236). Retrieved
from
??
?
http://www.jstor.org/stable/2251299
?
??
?
Turing, A. M. (1951). Intelligent Machinery, A Heretical Theory. The '51 Society.
? ? BBC. Retrieved from
??
?
http://philmat.oxfordjournals.org/cgi/content/citation/4/3/256
?
??
?
?
??
Ulam, S. M. (1958). Tribute to John von Neumann. Bulletin of the American
Mathematical Society,
??
?
?
?
?
64(3), 1–49.
??
? ??
?
?
?
Vinge, V. (1993). The Coming Technological Singularity:
to Survive in the Post-human Era. In
? ?How
???
?
?
?
?
Proc. Vision 21: Interdisciplinary Science and Engineering
in
the
Era of Cyberspace (pp. 11–22).
?
?? ??
?
?
?
Presented at the Vision 21: Interdisciplinary
? ? ? ? ? Science and Engineering in the Era of Cyberspace, Lewis
? ? ? ??
?
?
?
?
Research Center: NASA.
Retrieved
from http://adsabs.harvard.edu/abs/1993vise.nasa...11V
? ? ???
? ? ? ??? ?
?? ??
??? ?? ?? ?? ??? ?? ?? ??
?? ? ? ???? ? ?
? ????
41
?