So you are the reason that a lot of stores have a minimum charge amount for credit / debit charges. The transaction fees charged to merchants are ridiculous and so are ATM fees. Until these fees are reduced, you will never see a truly cashless society. And that doesn't include those that have less trust of banks than they do of governments.
Why blame the man working within the system for charges applied by someone completely different? Either the merchant should be happy to absorb the transaction cost, the merchant should specify the minimum cost, or the bank shouldn't charge the fees. But it most definitely is NOT the fault of the person simply buying something.
Now let's flip the thing around. For the few cents per transaction that end up going to the banks for small purchases how much could be potentially saved by not tallying up the register, not storing float offsite or managing a safe, not having to train staff to manage cash securely, not having to bank your earnings at the end of the day (that's a good expensive one there), and above all when the cash register disagrees with the paperwork not spending an hour trying to figure out where the money went.
There is a cost of doing business in cash. You just don't see it and point to credit fees instead. I for one pay an accountant to do my taxes because he's faster and cheaper than the time I'd spend doing it, so why not pay a bank to manage the money (if we went cashless).