I've been running an openvpn link from my home to our colo for years. I also have it set up on all my devices so I can use it while traveling. Some of our DFly devs also use it when they are traveling. Here's my cumulative wisdom on the matter:
Generally speaking it works quite well. I use a medium-numbered port but I also have a server running on port 443 because the many weird networks one runs through when traveling often block most parts, but usually leave the https port open.
* Use UDP for the transport when running openvpn over a broadband link. This provides the most consistent experience.
* Use TCP for the transport for connections from mobile devices. This provides the most consistent experience. There are several reasons for this not the least of which being that the telco infrastructure seems to devalue UDP by a lot verses other traffic. TCP is also a lot easier to run on the server-side if you potentially have many devices connecting in, because you can run one server instance.
* Configure a smaller mss, I use 1300, so the encapsulation doesn't get fragmented by the transport. This is very important.
* Configure a relatively frequent keepalive in openvpn over a WAN link (I use 1sec/10sec), but a less frequent one over mobile (I use 20sec/120sec). This is particularly important on mobile because cell tower switches can cause long disruptions. You don't want to drop the VPN link in such circumstances if you can help it. DO NOT DISABLE THE KEEPALIVE. Always have an openvpn keepalive setup, particularly over TCP, because the TCP connection backoff can prevent your sessions from recovering or cause them to take a long time to recover if one or the other direction is not actively sending data (such as with most web connections, downloads, streaming, etc).
I personally like 'OpenVPN Connect' on IOS (which I use to connect to our project colo). And of course I run openvpn on all the DragonFly boxes including my laptop.
Reliability of the VPN depends entirely on the path between your location and the VPN server. The packet must travel this path in addition to the path from the VPN server to the nominal destination, and even in the best of circumstances it will double the chances of something going wrong.
I've had a number outages at home where my cable link is still operational but the cable company's path to the VPN server is having problems. Also, recovery times are longer because not only does the dead network have to revive, but the openvpn setup has to reconnect and renegotiate.
Commercial services are going to be hit or miss. VPN'ing your broadband link might be problematic and you have no real visibility into what the commercial service is doing with your data. That said, they are probably going to be a lot better than trusting your data to the telco and wifi hot-spots you connect from when you are mobile.
Netflix and other video streaming providers will often block-out commercial VPN IPs from the service. Generally speaking, using a commercial service for high-bandwidth connections is really hit-or-miss. You are using their bandwidth as well as your own.
When using a VPN, you are bypassing any special deals your broadband provider has made with the likes of YouTube, Netflix, etc. Remember that if the cell bandwidth is supposed to be free, because it won't be over the VPN.
In terms of security, its a mixed bag. The VPN will secure your traffic from your immediately ISP/Telco (aka Comcast, AT&T), and that's actually very important. However, you are not anonymous and once your traffic reaches the egress point its up for grabs by any network it flows through and, in particular, the target web page or whatever might be doing its own data collection.
But the telco data collection is MUCH more valuable to third parties than target data collection, and the VPN link at least protects you from that.
The VPN will not do a whole lot for your internal network security. If someone breaks into an IOT device on your home network you are pretty much screwed. The best defenses here are (A) to not use IOT devices in your home - disable their internet access for the most part, and (b) is to have a router inbetween your cable modem / U-verse device and your home network:
cable modem home router home network + WIFI router
I run all the NAT and openvpn stuff on my home router, so a compromised cable modem has no access to my home network. I also segregate the wired ethernet's IP space from the wireless router's IP space, and firewall the IPs, so nothing on the wireless router can fake my wired IPs.
More on the IOT devices. Obviously things like a printer or AppleTV have to be on the wifi network. But your refrigerator, 'smart' TV, Blueray player, receiver, and other junk does not. And you can further segregate the wifi devices by running several different WIFI SSIDs with different passwords. I don't quite go that far even though my printer is almost certainly vulnerable to a LAN hack.