"Brogramming" (horrible term) and highly engineered programming are both great solutions to different sorts of problems. The key in choosing which is how well mapped out is your destination. If you are building a banking system where customers will engage in known transactions resulting in a known data stored in the database then the backend of this system obviously demands a very carefully engineered solution. But for that same solution the front end should be pretty damn freeform. People should try a bit of this and a bit of that feeling their way to an awesome UI. The back end engineering will dictate what data needs to come in from the UI and what data needs to go out but a great UI comes from a combination of requirements, gut feelings, fiddling, artistic balance, etc.
Then after the UI is ready for polishing you might go back to a more engineering approach and try AB testing where you watch the speed at which a user uses the system along with other measures such as number of mistakes.
Personally I find that people who hate the free form programing tend to be those managers who just don't trust their employees. They want to lay everything out in a design document that then locks everyone into a my-way-or-the-highway approach. This is a great way to get your best programmers to find another company to work for. Also my best programming has come from those times that I went down 5 dead ends and the 6th was really cool. But the 6th naturally evolved from what I learned in the first 5. There is no way that it would have ever have been conceived in a design phase.
There are many things that can cause inertia that are not directly related to the code. A simple example would be unit testing. (I love unit testing) but if you are going to completely redo your system then much of your unit testing goes out the door. Your carefully written documentation is garbage. Your design documents are all garbage, and any work you have done in planning version 2 or more is trashed. This makes drastic alterations much more costly than just the programming. But the reality is that you should never produce a bad product because the paperwork got in the way of switching it to a good or great product. This is sad because often if all that needs alteration is the UI where a well engineered code base should have fairly good UI abstraction and thus a new UI should involve little fundamental/programming work.
Really which is used and when it is used comes down to great managers. They will focus the freeform programming on organically finding cool ideas while not chasing rainbows and at the same time making sure that the fundamentals are well engineered. Within any team of programmers there are usually those who prefer one or the other anyway.