Hi, I'm a Bluetooth Security researcher. My primary focus is on BLE for which I built a highly robust sniffer on the Ubertooth platform. I have experience in other aspects of Bluetooth.
TL;DR: Classic Bluetooth is very secure, BLE is secure under some circumstances. Even if you leave your Bluetooth on in discoverable mode, there isn't much an attacker can do to harm you barring bugs in your Bluetooth stack.
Bluetooth is a well-designed protocol stack that takes security seriously in its design. Implementation quality (and bugs therein) varies from stack to stack. It's always a good idea to disable Bluetooth if you aren't using it, as is the case with any other remotely accessible feature.
Classic Bluetooth has used Secure Simple Pairing (SSP) since 2.1 in 2007. This pairing mechanism is based on ECDH to provide perfect forward secrecy and is highly secure. There was one weakness discovered in the numeric entry pin mode in 2008 by Andrew Lindell. This mode is not commonly used in older devices and more recent devices do not implement it. It's effectively impossible for an attacker to sniff any data sent over Bluetooth with SSP.
BLE has major weaknesses in its pairing protocol that I spoke about at BlackHat USA 2013 and other venues. For the most recent video see my presentation at USENIX WOOT 13.
In BLE, a passive eavesdropper who is present during pairing can recover the secret key used to encrypt all communications. This effectively makes the security worthless. However, if the attacker is not present during pairing then the encryption is very effective. It uses AES-CCM and doesn't have any major flaws in the design. AES-CCM is used in WPA2-AES; it's well-established and has no major shortcomings.
Finally, some Bluetooth stack implementations have bugs. I've found remote bugs in one major vendor's stack.