Libertarians would like to eliminate all authority except the unfettered, sovereign right to property, which in fact, makes them despotic governors of their own property. Anarchists would eliminate not only centralized government, but also sovereign property rights.

Both systems, though, require some way of enforcing the rules, which mandates the need for a government, otherwise you end up with a simple might-makes-right form of anarchy, which is not really what either side wants.