Yeah, my FIL is a clothing salesman. He sells five states, about twelve lines of clothes, to women's boutiques. The boutiques pay about half of "retail" cost for their clothes, and people say, "well, that's all they cost, why do you charge double?" Because they have rent, they have employees, they have to have stock. You can run it really cheap - if you order by the truckload. But I don't think anyone wants 1000 of any shirt, especially if they can't return it.
The whole distribution process isn't free. You're paying for just this size, just this amount, just this color... the manufacturer has to guess that months in advance. Retailers take their own gambles, and in that market, a lot of them don't even make much money - they are vanity projects for businessmen's wives. Running a retail business that nets $50k/yr is dumb. If you have the skills to do that, you can make far more working for someone else if you need money.