VPNs can be useful. They should be used to forcibly encrypting traffic as well as only restricting traffic to known sources, destinations, ports, etc.
Some PHBs think encryption like https is good enough and that simply isn't the case. Systems that have no business connecting to the Internet should be explicitly blocked from doing so. Systems that need to transmit data over the Internet for B2B traffic should do it over a VPN connection whenever possible with restrictions. I have seen too many systems granted open Internet access to download a single file from a business partner.