Theodore Roosevelt was the only U.S. President sworn in without a Bible.
Ansley Wilcox, who owned the Wilcox Mansion where Roosevelt was inaugurated, wrote: "According to my best recollection no Bible was used, but President Roosevelt was sworn in with uplifted hand." (The Presidents and Their Wives)
Franklin Pierce was the only president who affirmed the oath of office, and was not sworn in at all. (More Presidential Inauguration Trivia here: Link)