However, recently, there are some signs that this trend has abated. In particular Neanderthals had larger brains than modern humans.
Is this consistent with the idea that big brains are meme nests? Surely memes have been on the up-and-up - while brain size has not.
The modern brain shrinkage corresponds to the rise of modern agriculture and increased population densities. Here's a quote from a 2010 article on the topic:
Bailey and Geary found population density did indeed track closely with brain size, but in a surprising way. When population numbers were low, as was the case for most of our evolution, the cranium kept getting bigger. But as population went from sparse to dense in a given area, cranial size declined, highlighted by a sudden 3 to 4 percent drop in EQ starting around 15,000 to 10,000 years ago. “We saw that trend in Europe, China, Africa, Malaysia—everywhere we looked,” Geary says.
Through most of human history the meme pool had to fit in a single brain. There wasn't much in the way of specialization - all the tribe members played similar roles (except perhaps for the doctor). There was probably some gender-based specialization, but that was about it - the meme pool mostly had to fit in a single mind.
With the advent of agriculture, large populations and exchange and specialization - and more advanced language - all this radically changed. Memes were liberated from the confines of a single mind, and the meme pool was able to expand enormously. Towns could support far more memes that any hunter-gatherer tribe could manage. The process of meme liberation eventually led to writing - another major move to liberate memes from the human skull.
The meme pool not being effectively confined to a single mind would have massively reduced the selection pressure on minds to grow to accommodate more memes. Now minds only had to accommodate the memes associated with a given specialization.
Other theories may have something going for them too. Modern humans have been domesticated by their institutions - and domestication often results in smaller brains - since the domesticated creatures have their defensive and foraging needs supplied for them. Also agriculture led to poorer diet - and that might have had a negative effect on brain size too (though re-feeding modern humans doesn't give them much bigger brains).
However, the idea of big brains as meme nests is at least consistent with the modern cerebral downturn. The modern cranial shrinkage corresponds to the liberation of the meme pool from the mind of a single hunter-gatherer mind. The division of labor that came with large populations would have meant that each specialization had its own, much smaller and largely-independent meme pool. The pressure on the brain to grow to accommodate the entire meme pool of the human race was off - and stupider humans could thrive.
The liberation of memes led to a removal of the size limit on cumulative cultural evolution. Now that they were no longer effectively confined to a single mind, the modern meme explosion began to gather speed.