Yeah, that's pretty much a question you known the answer to: no, and not many of them ever did. But the cliche came from somewhere, and Slate explains.
Before Communists, the first “card-carrying” groups tended to be members of labor unions, and often members used the cards openly to claim union jobs. Perhaps the earliest mention of “card-carrying” union members came in a 1912 Daily People article, which described how “ ‘Union-card’ carrying members” of certain labor unions were occasionally able to get elected to Congress. A 1918 New York Tribune article described members of the Industrial Workers of the World as “red-card-carrying ‘wobblies,’ ” referring to the members by their popular nickname and red membership cards. In 1927, all “card-carrying members” of the secret organization United Farmers reportedly received discounts from local merchants.