So if they can't enforce a fine, then what happens if you don't pay the straffavgift?
I doubt that it would end up in court. There is an agency called "Kronofogdemyndigheten" (Swedish Enforcement Authority) who's responsibility it is to collect debts no one has been able to collect. If you have an unpaid debt that end up with them you get something called "betalningsanmärkning" (erm... note of payment default perhaps). Something no-one in Sweden wants as it makes it pretty much impossible to get loans, telephone contracts, bank accounts, etc, etc for five years. It's a pretty strong incentive to pay your debts and bills.
It sounds like they don't have any authority to actually run things. While I appreciate the enforcement of privacy, does that also apply to businesses? Are they not allowed to keep track of who shop-lifted or passed bad forms of payment or otherwise...
Storing personal information in Sweden is thoroughly regulated with progressively tougher demands depending on the sensitivity of the information. It's doubtful you would be allowed to store any information about the behaviour of a person in a personally identifiable manner. Companies that give credits are allowed to store information about the payment history of individuals, but only for a limited time and with strict requirements on how that data can be used and shared.
...caused problems and they don't want to let into their business again?
You are not yourself allowed to hinder or detain someone unless they have commited a crime, only the Police or licenced security guards are allowed to do that. Which causes a bit of a headache for the company running the subway actually as their personell aren't allowed to hinder anyone from leaving the premises instead of showing their ticket without backup from the police or a guard (IIRC).