If you can't keep a consistant bare min. sustained speed of 35mph...then, yes...you are holding up traffic on about 98% of the roads most driver use...and are an impediment.
As I said, if there were cars piled up behind me and they couldn't pass I would just pull over but this has never been a problem because most of the streets around here are either two lanes or wide enough that I can ride to the right and cars just pass me if they need to. Honestly, in downtown traffic I'm sometimes moving faster than traffic and more often than not it seems like we're all racing just to wait at the next light anyway.
And sure, some people live in what I'd almost have to list as a micro-climate...where it is consistently good enough weather to ride a bicycle to work on a consistent basis.
I guess my area is a bit more temperate than many but it is going to hit 100F today. That's not a problem though because it's not hot until I'm already at work for the day. And I don't care if I sweat on the way home. Yes it's going to rain during the winter but that's what rain gear is for.
not even considering if you need to bring your laptop with you...possibly gym clothes if you like to hit the gym after work for some weight training or swimming....and work materials (ref. books,etc).
Between the various bags that can be mounted on a bicycle and the option of wearing a backpack you can carry a lot more than you'd think. I personally have a relatively small bag on top of my rack which is plenty to carry my lunch in the morning and the extra layers I don't need on the ride home. I keep several articles of nicer clothing at the office and wear comfortable and practical clothing for the actual commute.