Working in isolation can sometimes be more productive. But if the work is moving on quickly, people away from the office have latency. Also, certain spontaneous things just cannot happen. I remember rewriting a critical internal component of a rather core library since the original version, my rewrite, the other guy's rewrite and my re-rewrite had reached the limits of their flexibility, generality and performance. Like I said---a critical part. So I wandered over to discuss it with someone, and started some pair programming. A third person overheard and joind us. We probably ended up arguing back and forth oover the design on every line of code.
The end result hasn't needed to be rewritten or even modified in 7 years.
And honestly, it's a piece of code I'm proud of.
I've also found it harder to work with genuinely remote workers and I've found that the team spirit or whatever it is is harder to achieve. I've also worked at a Very Large Orgnisation, where we were all on-site but the critical people (for me) were within walking distance, so I could go over to their office. That was probably the best of both worlds in some ways, but I wouldn't like to rely on such a setup since that project was run by someone who was an utter genius at project managment, so it probably would have worked whatever happened.
Also on a final note, I would point you at:
search for "door closed". The piece is a very well written peice (and not very nice) about research careers, but I think that section has enough in common with programming. After all in both, you have to move with the times almost brutally and continously solve new problems and learn new things to stay on top.