Existential-Risk | The Oakland Futurist

I have always been dubious of the assumption that unfriendly AI is the most likely outcome for our future. The Singularity Institute refers skeptics like myself to Eliezer Yudkowsky’s paper: Complex Value Systems are Required to Realize Valuable Futures. I just reread Yudkowsky’s argument and contrasted it with Alexander Kruel’s counterpoint in H+ magazine. H+ seems to have several articles that take exception with SI’s positions. The 2012 H+ conference in San Francisco should be interesing. I wonder how it much it will contrast with the Singularity Summit.

One thing that bothers me about Yudkowsky’s argument is that on the one hand he insists that AI will always do exactly what we tell it to do, not what we mean for it to do, but somehow this rigid instruction set could be flexible enough to outsmart all of humanity and tile the solar system with smiley faces. There is something inconsistent in this position. How can something be so smart that it can figure out nanotechnology but so stupid that it thinks smiley faces are a good outcome? It’s sort of a grey goo argument.

It seems ridiculous to even try constraining something with superhuman intelligence. Consider this Nutrient Gradient analogy:

Bacteria value nutrient gradients.
Humans evolved from bacteria achieving a comparable IQ increase to that which a superhuman AI might achieve as it evolves.
A superhuman AI might look upon human values the same way we view bacterial interest in nutrient gradients. The AI would understand why we think human values are important, but it would see a much broader picture of reality.

Of course this sets aside the problem that humans don’t really have many universally shared values. Only Western values are cool. All the rest suck.

And this entire premise that an algorithm can maximize for X doesn’t really hold water when applied to a complex reflexive system like a human smile. I mean how do you code that? There is a vigorous amount of hand waving involved there. I can see detecting a smile, but how to you code for all the stuff needed to create change in the world? A program that can create molecular smiley faces by spraying bits out to the internet? Really? But then I just don’t buy recursively self-improving AI in the first place.

Not that I am against the Singularity Institute like some people are, far from it. Givewell.org doesn’t think that SI is a good charity to invest in, but I agree with my friend David S. that they are poorly equipped to even evaluate existential risk (Karnofsky admits existential risk analysis is only a GiveWell Lab project). I for one am very happy that the Singularity Institute exists. I disagree that their action might be more dangerous than their inaction. I would much rather live in the universe where their benevolent AI God rules than one where the DARPA funded AI God rules. Worse yet would be a Chinese AI implementing a galaxy wide police state.

This friendliness question is in some ways a political question. How should we be ruled? I was talking with one of the SI related people at Foresight a couple of years ago and they were commenting about how much respect they were developing for the US Constitution. The balance of powers between the Executive, Legislative, and Judiciary is cool. It might actually serve as good blueprint for friendly AI. Monarchists (and AI singleton subscribers) rightly point out that a good dictator can achieve more good than a democracy can with all it’s bickering. But a democracy is more fault tolerant, at least to the degree that it avoids the problem of one bad dictator screwing things up. Of course Lessig would point out our other problems. But politics is messy, similar to all human cultural artifacts. So again, good luck coding for that.

I did break down and actually attend a couple of talks at the Singularity Summit this year: Vernor Vinge and Peter Norvig.

Peter Norvig gave a talk that would have satisfied any generic group of AI developers. Google is making some frightening progress. This Deep Learning project is the most interesting aspect of his presentation from an AI architecture point of view. It’s impressive that Google can pair two top-level researchers in the field (Andrew Ng and Geoffrey Hinton) with parallel processing expert Jeff Dean and scale up academic models onto a functional 1000 node cluster. Boom, you are identifying cats and faces from unlabeled YouTube videos. It must be sickening to anyone who wants to compete with Google in the AI space.

But he never really mentioned friendliness. I was hoping he would trot out some more theory behind this big data approach. He gave a similar talk to Monica Anderson’s AI meetup a couple of years ago. I was there for that and it was pretty cool to see him present to such a small crowd.

At the Singularity Summit this year, he also talked about Google’s translation service which basically derives translations by mapping many many identical documents written in multiple languages. I was hoping to ask him what happens when the algorithm starts consuming translations that were actually created by Google Translate. It’s bound to screw them up if that happens. But then I realized that Google probably saves every translated document and checks new documents checksums against previous translations before using them to build mappings. That’s hard to picture though. They manage: A. Mind. Crushingly. Large. Amount. Of. Data.

Vernor Vinge outlined some outcomes that he sees for the singularity. One crazy idea he puts forth is a digital gaia where the world is minutely ornamented with digital sensors coupled to processors and actuators. One day they all spontaneously “wake up” and all hell breaks loose. He describes a reality with all the stability and permanence of the financial markets. I had a vision of my SmartLivingRoom(tm) suddenly reconfiguring itself into a nightmare of yawning jaws and oozing orifices. But in reality, we might just see wild fluctuations in the functionality of computationally enhanced environments; from smart to dumb to incomprehensible.

Next up: Augmented intelligence, a neo-neo-cortex provided by technology. This is his preferred scenario. Crowdsourcing is cool, yada-yada. Vinge imagines a UI so extreme that accessing it would be as convenient as the supported cognitive features. I used to like this idea until I started thinking about the security implications. I don’t want my brain hacked.

He did make one amazingly succinct point about human computer synergy. Computers can give us instantaneous recall and amazing processing speed, humans can provide that which we are best at: wanting things.

Humans want things. For me this cuts to the very heart of the AI question. I always complain that none of these AI geniuses can show us an algorithm to define problems. (No, CEV doesn’t count.) Algorithmic problem definition is just another way to say algorithmic desire definition Good luck with that one.

All simple human desires seem to arise from biological imperatives. Maybe artificial life could give you that. More complex desires are interpersonal and might be impossible to reduce back to metabolic processes. You may want fame for the status, but the specific type of fame depends on which person or group you are trying to impress. And that changes throughout your life.

And if we do build Artificial Life, it may well be that it can only function with similar constraints as, uh, non-artificial life. In fact, Terrence Deacon may well be right and constraints are the key to everything. Ahh, the warm fuzzies of embodiment are seeping over me now.

But seriously, SingInst, where is this algorithmic desire going to come from? And once you get that, how the hell are you going to constrain the actions of GodLikeAI again? I know, I know, Gandi would never change himself into an anti-Gandi. But we may be like bacteria saying that our distant offspring would never neglect the all encompassing wisdom of nutrient gradients.