Bitcoins are stored in adresses which have a private key associated with them. These address keypairs can be made on the fly by bitcoin wallets or wallet generators. Once you transfer coins into an address and the transaction is confirmed as legitimate it is recorded in the blockchain and from that moment on it is a matter of public record that address X contains Y bitcoins. If you want to transfer the coins out of the address you import the private key into wallet software and then you can spend it.
That address-key pair can be stored in an electronic wallet or be written down on a piece of paper. Paper wallet software does this for you and can conveniently add QR codes which saves you from typing in 50+ character keys. Those pieces of paper (or fancy metal tokens with a piece of paper glued on them) can be traded like collectible trading cards.
Everyone with access to the address can confirm that the bitcoins are in there, but only those with the private key can transfer money out.