Don't they have a right to an opinion.
Everybody has a right to their own opinion. What they don't have is a right to their own facts.
Sure you need to be on the VPN to see your e-mail (that notorious destroyer of productivity), but there's a lot of stuff you can be doing offline (or at least off the VPN) that is still productive work. For example, if I'm writing code, it's not always the case that I have to be on the company network to do it.
Also, my VPN software seems to be the only common element in the rare blue-screen crashes I get on my work laptop - so it's usually a lot less frustrating to leave it off.
In fact, if I am goofing off, I'm much more likely to log into the VPN and open my e-mail so that others can see that I'm "online" and working. I like to sit my laptop next to my gaming desktop while I do this!