I think it's pretty obvious: your boss has never worked a solid day in his life.
I'm not a programmer, I do ceramic tile during the day, photography in my off hours. Since I shoot raw, many evenings are filled with post-processing. I'll tell ya, I'd DIE, without any music while working.
These two things, I think, are very similar to what you do: a little bit creative, a little bit technical, coming together to make a unique whole. Music probably gives your thoughts rhythm and a creative spark to solve problems that absolute silence, or distracting background banter in the office would quench.
Without music, you probably feel more...autonomoton than productive employee.
In fact, in every job site I've worked, with few exceptions, others on the site need something to listen to. Without it, work is drudgery.
I think your boss is just jealous because you're productive, and he (she?) doesn't know what they're supposed to do. :)