If we are to suppose that brain size is correlated with intelligence, then we should be forced to conclude that men are smarter than women, on average.

Furthermore, we would be forced to conclude that Neanderthals were smarter than homo sapiens. Again, a hypothesis that is generally unsupported by the available evidence.

Here's an alternative hypothesis: brain shrinkage is correlated with agricultural development. Maybe our diets have changed to allow for smaller, more efficient brains. After all, there is certainly evolutionary pressure to have smaller, more efficient organs that consume less energy to perform th same tasks.