Not quite. The virtual pair has a net energy of 0, and therefore isn't really "extracting" anything from the vacuum. If the pair weren't straddling a black hole, they'd recombine and disappear and nothing would happen. However, when one member of the pair is sucked into a black hole while the other particle escapes, the escaping particle must have a greater-than-zero amount of energy. Because of that, and the requirement that the two particles balance each other out, the black hole has necessarily absorbed a negative-energy (not negatively-charged, mind you, actually negative energy) particle, causing the black hole to shrink ever so slightly. Basically, while the particles originally came from vacuum, the energy was taken from the black hole.
Also, to an external observer this process looks exactly the same as if the black hole itself was slowly emitting particles and shrinking away. And if the black hole is emitting particles, we can use that to determine something about its internal state.