Memes and the "machines only do what you tell them to" fallacy

I've finished reading and reviewing Probably Approximately Correct: Nature's Algorithms for Learning and Prospering in a Complex World by Leslie Valiant.

It didn't make it into my review, but one comment in the book seemed especially naive to me. Leslie says:

The most singular capability of living organisms on Earth must be that of survival. Anything that survives for billions of years, and many millions of generations, must be good at it. Fortunately, there is no reason for us to endow robots with this same capability. Even if their intelligence becomes superior to ours in a wide range of measures, there is no reason to believe that they would deploy this in the interests of their survival over ours unless we go out of our way to make them do just that. We have limited fear of domesticated animals. We do not necessarily have to fear intelligent robots either. They will not resist being switched off, unless we provide them with the same heritage of extreme survival training that our own ancestors had been subject to on Earth.

For most readers familiar with the topic of machine intelligence it is probably obvious why this is wrong. Survival is a universal instrumental value. You typically don't need to deliberately program it in.

I think a basic smattering of memetics could have avoided this error. Memes are evolved to be good at surviving, just as genes are. They too have extended evolutionary histories during which they are subject to natural selection. This is likely to be true of intelligent machines, just as it is true of many cultural entities today. Today's cultural entities may not have survived for billions of years - but they often have a good number of generations behind them - and many of them plainly exhibit adaptations for survival.

Machine intelligence enthusiasts tread a difficult line. On the one hand they don't want to exaggerate the dangers of their products - since Terminator-savvy consumers may agree with them. On the other hand, they should not exaggerate the safety of their product category - or they will appear over-confident about the potential negative side effects of their products. Consumers don't by a hedge trimmers from manufacturers who thinks that hedge trimmers are totally safe and don't need special safety features. I think Leslie is erring too much on the size of being blazeé.

One of the reasons I got into memetics in the first place because I believed that a proper science of cultural evolution was too important to be left to cultural anthropologists - who had demonstrably failed to get to grips with the topic for over a hundred years. I think that machine intelligence enthusiasts should take steps to understand this topic. Part of the reason is to avoid making this sort of mistake. We now have a great science of cultural evolution - which is very useful for understanding issues associated with the coming era of intelligent machines. However people have to study and learn about the subject before it is of very much use to them.