The English language has a multitude of terms for insanity, although few of them are diagnostic terms for mental illness. Slang terms for "crazy" are used when a person has lost control or just doesn't make sense to those around him. These terms aren't confined to people, since a wild party or an unconventional building or a nonsensical movie can all be "bananas." But bananas, crackers, and nuts are the only food terms that came to mean crazy. Let's learn how that happened.
The Routledge Dictionary of Modern American Slang and Unconventional English traces the idea of bananas relating to craziness only back to the late 1910s; The Concise New Partridge Dictionary of Slang and Unconventional English roughly agrees, with its own etymology going back to 1924. These sources claim that the crazy banana meaning comes from the phrase “banana oil,” which, in flapper slang, meant “nonsense.”
Typically, the substance referred to as banana oil is amyl acetate, which is, insanely, used as both a paint solvent and as a flavoring. (It smells strongly, but not precisely, of bananas, which is partly why banana-flavored candies are distinct but not really banana-flavored.) Why did flappers like to use the phrase “banana oil” to mean “nonsense”? Possibly it’s related to “snake oil,” but I suspect it’s also because the word “banana,” coming from a language that has not given English very many words, sounds unusual to the ears of English speakers.
"Banana oil" fell out of use, but came back later, oil free. And what about nuts and crackers? Both terms mean things other than food, which makes their evolution into slang all the more difficult to trace. You can read the history of the food terms used for craziness at Atlas Obscura.