Satisficing is Safer Than Maximizing

Before I begin, let me just say that if you haven’t read Bostrom’s SuperIntelligence and you haven’t read much about the AI Alignment problem, then you will probably find this post confusing and annoying. If you agree with Bostrom, you will DEFINITELY find my views annoying. This is just the sort of post my ex-girlfriend used to forbid me to write, so in honor of her good sense, I WILL try to state my claims as simply as possible and avoid jargon as much as I can.

[Epistemic Status: less confident in the hardest interpretations of “satisficing is safer,” more confident that maximization strategies are continually smuggled into the debate of AI safety and that acknowledging this will improve communication.]

Let me also say that I THINK AI ALIGNMENT IS AN IMPORTANT TOPIC THAT SHOULD BE STUDIED. My main disagreement with most people studying AI safety is that they seem to be focusing more on AI becoming god-like and destroying all living things forever and less on tool AI becoming a super weapon that China, Russia, and the West direct at each other. Well, that’s not really true, we tend to differ on whether intelligence is fundamentally social and embodied or not and a bunch of other things really, but I do truly love the rationalist community even though we drink different brands of kool-aid.

So ok, I know G is reading this and already writing angry comments criticizing me for all the jargon. So let me just clarify what I mean by a few of these terms. The “AI Alignment” problem is the idea that we might be able to create an Artificial Intelligence that takes actions that are not aligned with human values. Now one may say, well most humans take actions that are not aligned with the values of other humans. The only universal human value that I acknowledge is the will to persist in the environment. But for the sake of argument, let’s say that AI might decide that humans SHOULDN’T persist in the environment. That would sort of suck. Unless the AI just upgraded all of us to super-transhumans with xray vision and stuff. That would be cool I guess.

So then Eliezer, err, Nick Bostrom writes this book SuperIntelligence outlining how we are all fucked unless we figure out how to make AI safe and (nearly) all the nerds who thought AI safety might not matter much read it and decided “holy shit, it really matters!” And so I’m stuck arguing this shit every time I get within 10 yards of a rationalist. One thing I noticed is that rationalists tend to be maximizers. They want to optimize the fuck out of everything. Perfectionism is another word for this. Cost insensitivity is another word for it in my book.

So people who tend toward a maximizing strategy always fall in love with this classic thought experiment: the paper clip maximizer. Suppose you create an AI and tell it to make paper clips. Well what is to stop this AI from converting all matter in the solar system, galaxy, or even light cone into paperclips? To a lot of people, this just seems stupid. “Well that wouldn’t make sense, why would a superintelligent thing value paperclips?” To which the rationalist smugly replies “Orthogonality theory,” which states that there is NO correlation between intelligence and values. So you could be stupid and value world peace or a super-genius and value paper clips. And although I AM sympathetic to the layman who wants to believe that intelligence implies benevolence, I’m not entirely convinced of this. I’m sure we have some intelligent psychopaths laying around here somewhere.

But a better response might be. “Wow, unbounded maximizing algorithms could be sort of dangerous, huh? How about just telling the AI to create 100 paper clips? That should work fine, right?” This is called satisficing. Just work till you reach a predefined limit and stop.

I am quite fond of this concept myself. The first 20% of effort yields 80% of value in nearly every domain. So the final 80% of effort is required to wring out that final 20% of value. Now in some domains like design, I can see the value of this. 5 mediocre products aren’t as cool as one super product, and this is one reason I think Apple has captured so much profit historically. But even Jobs wasn’t a total maximizer, “Real artists ship.”

But, I’m not a designer, I’m an IT guy who dropped out of highschool. So I’m biased, and I think satisficing is awesome. I can get 80% of the value out of like five different domains for the same amount of effort that a maximizer invests in achieving total mastery of just one domain. But then Bostrom throws cold water on the satisficing idea in Superintelligence. He basically says that the satisficing AI will eat up all available resources in the universe checking and rechecking their work to ensure that they really created exactly 100 paper clips. Because “the AI, if reasonable, never assigns exactly zero probability to it having failed to achieve its goal.” (kindle loc 2960) Which seems very unreasonable really, and if a human spent all their time rechecking their work, we would call this OCD or something.

This idea doesn’t even make sense unless we just assume that Bostrom equates “reasonable” with maximizing confidence. So he is basically saying that maximizing strategies are bad, but satisficing strategies are also bad because there is always a maximizing strategy that could sneak in. As though maximizing strategies were some sort of logical fungus that spread through computer code of their own accord. Then Bostrom goes on to suggest, well, maybe a satisficer could be told to quit after a 95% probability of success. And there is some convoluted logic that I can’t follow exactly, but he basically says, well suppose the satisficing AI comes up with a maximizing strategy on its own that will guarantee 95% probability of success. Boom, Universe tiled with paper clips. Uh, how about a rule that checks for maximizing strategies? They get smuggled into books on AI a lot easier than they get spontaneously generated by computer programs.

I sort of feel that maximizers have a mental filter which assumes that maximizing is the default way to accomplish anything in the world. But in fact, we all have to settle in the real world. Maximizing is cost insensitive.  In fact, I might just be saying that cost insensitivity itself is what’s dangerous. Yeah, we could make things perfect if we could suck up all the resources in the light cone, but at what cost? And really, it would be pretty tricky for AI to gobble up resources that quickly too. There are a lot of agents keeping a close eye on resources. But that’s another question.

My main point is that the AI Alignment debate should include more explicit recognition that maximization run amok is dangerous <cough>as in modern capitalism<cough> and that pure satisficing strategies are much safer as long as you don’t tie them to unbounded maximizing routines. Bostrom’s entire argument against the safety of satisficing agents is that it they might include insane maximizing routines.  And that is a weak argument.

Ok, now I feel better. That was just one small point, I know, but I feel that Bostrom’s entire thesis is a house of cards built on flimsy premises such as this. See my rebuttal to the idea that human values are fragile or Omohundro’s basic AI drives.  Also, see Ben Goetzel’s very civil rebuttal to Superintelligence.  Even MIRI seems to agree that some version of satisficing should be pursued.

I am no great Bayesian myself, but if anyone cares to show me the error of my ways in the comment section, I will do my best to bite the bullet and update my beliefs.

If You Use Tools, Then You’re “Transhuman”


I’m reading Nexus, by Ramez Naam. I like the book so far, except for the early chapters, which are a little creepy and repetitive. But one of the themes is that there will be a war between transhumans and unaugmented humans. This idea has gained popularity and it’s starting to annoy me.

When I saw Ramez Naam speak at the H+ conference in 2012, he scoffed at the very term “transhuman.” He made the point that ALL humans augment, and the term “transhuman” seems to imply a mythical, non-augmenting human. His examples of transhumans were people who have pacemakers or who use birth control. Look at professional athletes. Does anyone really think those are unaugmented humans? They’re on steroids, practically mutants. Cyclists are using blood doping until their blood is so packed with red blood cells that it sometimes stops flowing through their veins and kills them. Or consider how new prosthetic legs make amputee runners superhuman.

I would go so far as to say that using a sharpened stick is as transhuman as using a brain-computer interface. So transhumanism is a misnomer. Transhumanism is really just extreme tool use. And politics being what they are, the elite will always control the wielders of technology, just as kings controlled the knights of medieval times.

But writers keep setting up conflicts between tool users and non-tool users. Zoltan Istvan has previously called for transhumans to deliberately create conflicts with religious people, who he imagines don’t like tools. Yuval Harari in Homo Deus suggests that transhumans could dominate humans as 19th century Europeans dominated Africans. In a recent Forbes article, Jeff Stibel warns that brain-computer interfaces could destroy humanity and calls for ethicists and philosophers to guide us. And, you know what, they are all correct (aside from the ethicists guiding us idea, that’s utterly laughable bullshit), but they are missing a key point.

Tool using populations destroy non-tool using populations. In the past, farmers used crazy technology to create food on demand and then consolidated resources and crushed the hunter-gatherers around them. Zoltan needn’t call for deliberate conflict. The wielders of the most advanced technology INEVITABLY overwhelm or convert those without it. Transhumanism will be no different. Harari might take a moment to note how the technology rich Global North dominates the poor Global South TODAY. Wealthy Westerners are already transhuman compared to the poorest in the world. We have longer lives and amazing influence.

This competition is very essentially human, and I’m not even sure that it’s entirely bad. As I’ve said before, cooperative groups turn out to be more competitive. Compassion is an evolved superweapon. Stibel is deluded. There is no stopping this process. He can hold back his patents for BrainGate all he wants. The physical world will continually yield up its mechanisms of action to the prying minds of restless humans. His discoveries will be reproduced. Cultures that seek to repress technology will be surpassed and dominated by cultures that don’t. That’s just the way the world works. Partly because ethics aren’t universal. Some players have legitimately diverging interests.

But in the broader scheme of things, we aren’t meant to stop and rest. Life has been evolving for billions of years. It will keep evolving. That’s physics. Entropy must be maximized and the negative entropy of more and more complex living things must fulfill the requirement of the physics engine running our universe. What if life had stopped evolving at bacteria? From a human’s perspective, that would have sucked. How can we begrudge the post-humans their place? Hint, we don’t get to. On the plus side, bacteria ARE still around and we need them to survive. On the negative side, neanderthals only exist as DNA remnants. I hope that the jump to the next level of evolution will be so extreme that, to the next generation, humans are more like bacteria and less like neanderthals. There’s a strange toast. Cheers!

Is the Military-Industrial Complex Functioning as Designed?

military general drinking whiskey and holding a gunThe Operators, by Michael Hastings

I went to a poetry reading in North Beach last week and then out drinking with the poets and their friends afterwards. This fulfilled an old Beatnik fantasy of mine. As I teenager, I venerated the Beat generation writers like Kerouac and Ginsberg. Their jazz-fueled, drug-laden epiphanies, wandering barefoot through the city at dawn, seemed a far cry from my suburban ennui, where the shopping mall was the hottest spot for us teens to gather. But there I was, last week, sitting in an old North Beach bar, arguing about Hemingway with a bunch of old hippie communist poets in the very place that Ginsberg, Kerouac, and Cassady might have had the same disagreement. It was like a dream come true.

Communists that they are, these poets blamed all of modernism’s ills on capitalism exclusively, it’s all about the money for them. Marx himself was a modernist in my own view, and the systems inspired by his visions have fared no better than those driven by capital in terms of real human flourishing, but I’m grinding a different axe today. See, the commies are convinced that it’s all about the money. Why has the US been venturing into the Middle East for so many years, wasting money and lives, as we clumsily sow chaos throughout the region? It must be for the money. “No Blood For Oil” read the protest signs. And it’s not an exclusively leftist position to take. Eisenhower himself coined the phrase “military-industrial complex” to warn of these powerful vested interests, and he was technically a Republican, though no Republican today would tolerate his views, I’m sure.

I hate to say this, postmodernist that I am, but there’s a part of me that thinks, if the US needs oil, it will grab oil. Our country is a system, competing with other systems for resources. I’d rather live under our system than under the Chinese or Russians, so I grit my teeth and bear it. But if you look at the details of the Iraq war, for example, huge Iraqi oil fields were ruined with water when they were disrupted by the war. Maybe some US oil companies benefitted temporarily from transient spikes in oil prices, but it’s not like the US is literally pumping the oil and taking it away. The US isn’t really benefitting from Iraqi oil. We spent way, way more on that war than we got back in free oil. One might say, well, it’s these multinational oil corporations that benefited. But it’s not at all clear why multinationals would care which regime they got their oil from. Saddam Hussein couldn’t pump the stuff himself.

And then there’s this whole line of reasoning that Hussein was threatening the US petrodollar by accepting euros for oil. And that seems like a decent argument, and would actually be aligned with US interests. Sure, we want the dollar to be propped up by the fact that it’s a global reserve currency. That’s fine, in a sense. Like it or hate it, if the purpose of the military-industrial complex is to preserve western dominance, as Chomsky might say, then this is just the sort of thing we should EXPECT it to do, and crying about it isn’t going to change the realpolitik of the situation. Powerful systems crush weak systems, end of story.

But then I ask myself, what if these systems AREN’T functioning to preserve US dominance? It may well be that US interests have actually been harmed by our Middle East adventures. We’ve certainly spilled plenty of blood and cash in Afghanistan for no apparent benefit. No oil there. Some pipeline theories float around, or maybe there’s a huge cache of rare earth metals we can grab, but, based on our track record, the US will probably fail to profit from either rare earth metals or pipelines.

It doesn’t seem like our military-industrial complex is being guided by the principle of advancing US interests. And that’s actually a bigger problem than if it were. If we were just bullies stomping on weak nations to make ourselves stronger, that wouldn’t be so bad, really. . . But if the military-industrial complex ISN’T guided by principles . . . if it fails to advance US interests, then it could destroy the US.

It doesn’t seem like our military-industrial complex is being guided by the principle of advancing US interests. And that’s actually a bigger problem than if it were. If we were just bullies stomping on weak nations to make ourselves stronger, that wouldn’t be so bad, really. As Pinker asserts in Better Angels, we have a long history in the West of becoming more and more civilized. If we’re bullies, we can learn to be gentle. But if the military-industrial complex ISN’T guided by principles, then it runs the risk of killing the goose that laid the golden egg. If it fails to advance US interests, then it could destroy the US, and there could be no more US to suck money out of at some point.

We saw a similar thing play out with the bank bailouts. A proper capitalism with accountability for making bad bets removes stupid strategies from the marketplace and should sustainably generate wealth for many. A crony capitalism that offloads risk onto the taxpayers threatens to break the very system from which its wealth is built.

What if these systems that we depend upon are broken? What if we didn’t go to war with Iraq to get the oil, we just went to fulfill the contracts of the military contractors? That just seems so crazy. Surely there should be someone at the wheel, guiding this whole thing, who would see a problem with that? But maybe not.

I read The Operators, by Michael Hastings. . . The big takeaway for me was that the military seems to be driven by a cult of bloodlust. . . The officers aren’t fighting for their country, they’re fighting to spill blood and to risk their own troop’s blood, in and of itself. . . Hastings describes war as a drug, the ultimate adrenaline high. And that paints an ugly picture. The actual foot soldiers think they’re fighting for their country, but they’re actually fighting out of loyalty to their fellow soldiers. The Pentagon starts a war simply to deploy assets and make sure that the defense industry gets paid, and the officers execute it because spilling blood is such a fucking RUSH, man! Holy shit, we’re fucked. . . This is a terrible system.

I had sort of accepted that defense contract spending was driving US military intervention, but then I read The Operators, by Michael Hastings. There’s a lot to say about this excellent book and its unfortunate author, but the big takeaway for me was that the military seems to be driven by a cult of bloodlust. On some level, the officers aren’t fighting for their country, they’re fighting to spill blood and to risk their own troop’s blood, in and of itself. It’s not even clear if they see these war theaters as proving grounds for their character, which would have some virtue, I guess. Hastings describes war as a drug, the ultimate adrenaline high. And that paints an ugly picture. The actual foot soldiers think they’re fighting for their country, but they’re actually fighting out of loyalty to their fellow soldiers. The Pentagon starts a war simply to deploy assets and make sure that the defense industry gets paid, and the officers execute it because spilling blood is such a fucking RUSH, man! Holy shit, we’re fucked. The wheels cannot help but come off of this system. Maybe we don’t have to outrun the bear, maybe we just need to outrun the other guys running from the bear, but come on. This is a terrible system.

To my conservative friends, my pals who defend modernism and think that it’s us postmodernists who have dismantled the system: Take a closer look at our systems. We postmodernists are doing you a favor by pointing out the flaws. We haven’t come up with any sustainable solutions, granted, but we didn’t CREATE these flaws. And now it seems to be up to ALL of us, modernists, traditionalists, postmodernists, whatever, to figure out a post-postmodernism that builds sustainable systems guided by principles and not just lust for cash and blood. God help us.