With the advancements in modeling, especially BIM, a team could look at the completed design and associated model as the product. These models are now getting advanced enough that the client could provide constructive feedback after using the model. With the current waterfall method, there is very little scope to feed back these issues to the design team, as they will move on to the next project immediately after handover.
Or even better, the architects don't just throw the building over the wall to the structural engineers, but used Agile to work collaboratively with the structural, M&E services and other stakeholders to produce the final design/model.
But the fundamental issue is these models and processes take more time up front, which means much higher design fees. Many clients will struggle to justify the expense. Until a financial payback is found to justify using Agile, I'm not sure if it will be viable.