Anecdotal - Daily ear bud user here who has been wearing them for 6-10 hours a day at work. My company has mandatory annual hearing tests. In the ten years I've been with the company, my hearing has not changed significantly*.
A few provisos though..
1. I do not have the volume very high. I can hear and understand people with the music on and could carry a conversation if I had to.
2. I'm a sample size of 1 giving my personal experience without providing supporting evidence.
My belief is that they are safe for extended use if the volume is controlled to normal levels. Of course they can more readily do damage since you're essentially funneling sound directly to your ear drums, just don't crank it.
* - My levels usually fluctuate slightly. Some values go up, some go down and vice-versa the next test.