Not coins have value because they have useful properties of money that facilitate trade. They can not easily be created/mined. They are easily divisible, easily transported, they don't rot or wear out. So they value for the same reason gold has value: their intrinsic properties are close to that of ideal money.
Tabasco sauce can easily be made, will go off, is annoying to divide and measure, and must be transported physically in a container. It is terrible for use as money.-
Money is meant to be something that can store wealth, in that people will accept it for wealth and give it when given wealth.
They are not banks, they are exchanges. A bit coin is basically a secret long number, you can store it however you want. Most people store them in a common file format that is encrypted called a wallet.