Rancors are native to the planet Dathomir, grow to be five meters tall or more, and see most humanoids as nothing more than snack food. So how did Jabba's Rancor end up in that pit beneath his palace?
It was given to Jabba as a gift from his henchmen when it was but a wee Rancy Wancy, and the Rancor's keeper Malakili learned to love that big nasty beastie like one of his own...until Luke smashed its head in.
But don't worry, according to this comic by Chris Gugliotti the story of Malakili and the Rancor has a happy ending, and despite the large dent in Rancy's head he's doing fine and eating solid foods again!
-Via Geeks Are Sexy