The way milk was regarded during ancient times is an illustration of how simple geographical differences get turned into xenophobia. The Roman Empire had plenty of reasons to distrust the people it conquered (and vice-versa), but milk-drinking now seems an odd thing to complain about. It was just another custom that separated the "civilized" Romans from everyone else, and it disgusted them.
During a visit to conquered Britain, Julius Caesar was appalled by how much milk the northerners consumed. Strabo, a philosopher, geographer, and historian of Ancient Rome, disparaged the Celts for excessive milk drinking. And Tacitus, a Roman senator and historian, described the German diet as crude and tasteless by singling out their fondness for “curdled milk.”
The Romans often commented on the inferiority of other cultures, and they took excessive milk drinking as evidence of barbarism. Similarly, butter was a useful ointment for burns; it was not a suitable food. As Pliny the Elder bluntly put it, butter is “the choicest food among barbarian tribes.”
Romans didn't drink milk because they couldn't keep it fresh in the Mediterranean heat. Northern tribes did because they could. Yet milk gave the Romans another reason to feel superior to their colonies. Cheese? That was another story entirely. Even after the fall of the Empire, the controversy over milk remained and eventually went global. Read about barbarians and their milk at Atlas Obscura.