I agree with your sentiment.
I'll veer off into the woods a bit. At my company, we recently switched to scrum and it gives clear visibility into who is adding all the extra, unnecessary work and scope creep that comes with people not quite doing what they are supposed to be doing. What I really like is that it also gives developers a tool to push back with when they are asked to switch contexts, re-prioritize what they are doing or take on new work via scope creep.
Team structure seems to play a larger part in things than I hear people talk about. Trying to find a balance between inexperienced and experienced developers is difficult, and we've pretty much always come up on the wrong side of it. One of the other fundamental imbalances I've seen (at least in our organisation) is that, within the context of a sprint, the developers are usually capable of doing everyone else's job- but the reverse is not true. I've seen this turn into situations where developers end up doing BA work, QA work and supporting production issues while trying to keep up with the development tasks that are on the board. Meanwhile, everyone else is just sort of sitting around, trying to look busy. In my opinion. this is a management failing more than anything, but it is still an imbalance.
This could just be how we apply (or misapply) things, but what I really wrestle with is the idea that ultimate accountability seems to rest with the developers, but others don't seem to be held to the same standard. Agile (I can only speak to scrum) seems to provide a framework that offers to give developers more power in exchange for more accountability- but none of the processes that give the developers what they really need to deliver on what is required seem to fall under that same framework. Scrum seems to declare things like figuring out specific requirements for what you're trying to develop outside it's scope. This means that when other areas of the business don't do what they are supposed to, the developers end up having to deal with it in some capacity. Since they are the ones on the hook, that usually means doing someone else's job. Developers are motivated by the sheer number of eyes focused on them, but where is the motivation for everyone else involved in the process? Even this book review seems to only talk about rewarding (or not) developers. There isn't mention of the other groups that feed the process.
I'd also go waaaay out on a limb and argue that misapplied agile seems to have the potential to kill vision within a company. By forcing people to spend the majority of their time thinking only about what they can deal with in terms of this (or the next) sprint, it seems long-term vision with respect to a product has the potential to get lost. In other words, if there isn't a cohesive vision of what you're trying to produce, you could be reduced to a team of developers who only have the ability to react to situations and stare at the technical debt you've accumulated by switching directions so many times. That's probably a bit melodramatic- maybe that potential exists regardless, but scrum makes it easy to see how that could happen.
Ok, enough ranting.