The scientific neglect of filtering and sorting

In computer science, filtering and sorting are big topics. Donald Knuth devoted volume 3 of his epic The Art Of Computer Programming to these two topics. That's a reasonable indication of their importance in computer science.

However, science in general took a different route. Filtering is dealt with partly as "selection" - which is covered most comprehensively by evolutionary biology. However filtering is a much broader topic, which extends well beyond biology. As a result, the science of selection is fragmented:

• Anthropic reasoning - a form of observation selection.
• Survivorship bias
• Selection bias
• Reporting bias
• Confirmation bias
• Funding bias
• Selective perception
• Selective attention
• Selective recall

Because the science involved is fragmented there are also a number of areas where it could be applied, but currently isn't - because its influence is not understood or recognised. When small coins accumuate in your wallet, that's a type of selection. Similarly selection results in unpalatable goods accumulating in your refridgerator. Selection is also important in many common physical phenomena, such as erosion, crack propagation, catalysis, crystal growth and electrical discharges. However, its influence often goes unrecognised there as well.

In 1971, George Price called for a theory of selection, writing:

A model that unifies all types of selection (chemical, sociological, genetical, and every other kind of selection) may open the way to develop a general ‘Mathematical Theory of Selection’ analogous to communication theory.

Price continues with:

Selection has been studied mainly in genetics, but of course there is much more to selection than just genetical selection. In psychology, for example, trial-and-error learning is simply learning by selection. In chemistry, selection operates in a recrystallisation under equilibrium conditions, with impure and irregular crystals dissolving and pure, well-formed crystals growing. In palaeontology and archaeology, selection especially favours stones, pottery, and teeth, and greatly increases the frequency of mandibles among the bones of the hominid skeleton. In linguistics, selection unceasingly shapes and reshapes phonetics, grammar, and vocabulary. In history we see political selection in the rise of Macedonia, Rome, and Muscovy. Similarly, economic selection in private enterprise systems causes the rise and fall of firms and products. And science itself is shaped in part by selection, with experimental tests and other criteria selecting among rival hypotheses.

If the situation with filtering in science is bad, the situation with sorting is surely worse. At least selection is championed by evolutionary biologists. Sorting is also very common. You can see its results while looking at stones on a beach or clouds in the sky. Shaking your breakfast cerial makes the biggest lumps rise to the top - a simple sorting operation. I have talked about "natural sorting " before - but most people have never heard of it. If the science of filtering is fragmented, the science of sorting is positively obscure.

There have been efforts to build a general science of selection. Proponents of universal Darwinism have been working on it. There's Zukav's "Without Miracles". There's Hull's "Science and selection". There's Fog's "Towards a universal theory of competition and selection". There's Campbell's "Epistemological roles for selection theory". It is probably fair to say that most of the pieces are out there, but the topic is far from penetrating the scientific mainstream. It seems as though more work in the area remains to be done.