Doesn't it depend a lot on what you refactor, when and how?
I have 3 year old code that I would like to refactor because I've since switched framework (from CodeIgniter to Symfony 2) and it would bring it in line with all my other projects, allowing me more easy code-reuse and not maintaining two frameworks both on servers and in my mind. But it's largely a convenience factor and I would agree that it will probably not improve code quality very much.
But I also have 12+ year old code written in plain PHP with my own simple database abstraction layer. I'm quite certain that refactoring that would do a world of good.