I once took over 30,000 lines of code that had been written by a subcontractor and trimmed it to around 4000 LOC. And you better believe it ran faster! Not because refactoring is magic, but because once all the mind-numbing almost-repetition was mucked out you could actually see what the code was doing and notice that a lot of it wasn't really necessary. Ever since then I have always maintained that coders should never ever copy and paste code. I've had people disagree, saying that a little bit of copying and pasting won't hurt, but I say if it's really such a little bit then you shouldn't mind re-typing it. Of course if you do that very soon you start putting more effort into devising ways to stop repeating yourself, which is exactly the point. Repeating yourself should be painful.
That's I think a reliable litmus test for whether you should refactor a piece of software. If it's an area of code that's been receiving a lot of maintenance, and you think you can reduce the size significantly (say by 1/3 or more) without loss of features or generality you should do it. If it's an area of code that's not taking up any maintenance time, or if you're adding speculative features nobody is asked for and the code will get larger or remain the same size, then you should leave it alone. It's almost common sense.
I don't see why anyone would think that refactoring for its own sake would necessarily improve anything. If an automotive engineer on a lark decided to redesign a transmission you wouldn't expect it to get magically better just because he fiddled with it. But if he had a specific and reasonable objective in the redesign that's a different situation. If you have a specific and sensible objective for reorganizing a piece of code, then it's reasonable to consider doing it.