On the contrary... I think PE should stay, but it should not suck. Have PE where the teacher actually, ZOMG, teaches the students how to play sports and do other physical things... all the rules, various techniques for throwing, running, jumping, hitting, kicking, etc. I was incredibly averse to any physical activity we might have been exposed to in school for years, because they would just give everyone a basketball and say, "OK, for the next 45 minutes, you play basketball. Go!" Not knowing the intricacies of the rules, and not having been instructed in how to do things properly, and not having picked any of that up on my own, I quite naturally sucked at it, and ended up hating it. In retrospect, had the teacher attempted to show us the joy of physical activity rather than just providing outdoor babysitting in contrast to the rest of the day's indoor babysitting, it may have done some good.
I say keep PE and music, and cut out the couple of hours you wasted every day watching videos, listening to a book get read at you, outlining criminally dry history books filled largely with anecdotal and questionably accurate information. And for the love of the children, cut out the standardized testing!! I remember my K-8 as having a ridiculously high concentration of hours of extreme boredom. Yes, I was a smart kid, but the, with all due respect, dumb kids were also bored out of their mind and did nothing most of the time.
If anything, the school day can be made shorter, so the kids can spend less time being talked at and more time learning and exploring on their own. Kids WOULD be inclined to exercise on their own if they weren't kept locked indoors during the most beautiful hours of the day and given mindless work to do once they get out.
So I partially agree with you, but don't see PE, music class, and healthy lunches as being mutually exclusive solutions.