I had a very similar idea, and it will work. Really. By the way, the poster above, Bysmuth, is dead wrong, labs and all. Feel free to contact me (Bill Cox - waywardgeek@gmail.com) if you need me as a reference to support this idea.
One of my contributions to open source and the blind community has been improving speech speedup algorithms. I listen at > 600 wpm, and have a blind friend who listens at double that. As part of this, I've done numerous A/B tests on many subjects (friends, family and acquaintances), trying to figure out what works for them. Here's what I found. First, anyone who is already a high speed reader also very rapidly becomes able to listen at high speed. This is 100% correlated, after maybe 100-ish tests. I found no counter examples, and the strength of listening speed ability increases with the subject's reading speed. While some speed readers do not hear a voice while reading, it must still using the speech centers in their brain, because high speed readers are already prepared for speed listening, whether they claim to vocalise or not. There are other contributing factors, most notably age. I am the only non-blind person I know who learned to be comfortable speed-listening after the age of 40, though I do have a strong central vision loss issue. Every test I did on with anyone over 40 backed up the fact that speed readers are also naturally speed listeners, but the > 40 crowd is almost violently opposed to speed listening, while the under 40 crowd thinks it's cool. I know... that's such an objective scientific observation :-)
Also, I found that non-blind listeners who force themselves to learn to speed listen (including me), discover that their regular reading speed increases naturally. People can argue all day long about vocalisation being good/bad while reading, but the fact is that the same centers in the brain are used regardless. If you train to listen fast, your reading speed will increase, and vise-versa. This is the single most obvious conclusion I have been able to draw. It's a very real effect.
Another interesting point is that young people will, given a chance, naturally turn up the audio speed over time while listening to good books, very much like we see kids reading faster as they read a good series.
Reading a story both visually and audibly in parallel should enable a reader (whether mostly using their eyes or ears) to focus on the story the way that is more natural for him, and as he goes faster over time, his regular reading speed will increase, regardless of his preference for audio or printed text.