The main answer, at least in my own career, is this:
Lack of formal architecture training.
I'm self taught, as many web devs are.
Now, I was considered a pretty good Web Dev, by Web Dev standards, but I noticed that the projects my team was working on ran into common problems.
No matter what fancy programming methodologies and delivery tools we were using, we always reached a point where the solution ossified and became difficult to develop for. This meant that in projects with complex requirements we always overran.
Also, while we could make the client happy on delivery, we struggled to *keep* them happy.
We tried 'Agile', we tried 'Extreme', we tried a thousand different buzzwords, but nothing worked.
As a Web Dev, when you see this happening, there are two ways you can go.
You can:
A: Accept this as a reality of the field, and carry on as before. You can then have a breakdown at thirty and never work again.
B: Refuse to accept crappy software and look for a better way. If you're thoughtful, and not too vain to learn the lessons of the sages, you discover the Gang of Four, Design Patterns, etc, and achieve enlightenment.
I went for B and finally, after ten-odd years as a professional, am finally writing good software that *stays* good.