No one has said it yet. Lower the settings for call quality in your Vonage settings a bit till you reach a happy place. You already are giving top billing to the adapter so by doing that it would probably be all you need to do.
I know when I first got my vonage i had a lot of delay issues while gaming or downloading, till I lowered the quality then it was all fine. Still sounds great and no lagging issues.