exactly. it doesn't really matter if you are there or not. eventually something is going to break in a new and interesting way that can't be fixed without a significant amount of work.
generally we try to have at least three systems for any production service so that we can still have redundancy while doing maintenance.
that said, I rarely come in for patching anymore. I just make sure I'm available in case something doesn't come up afterwards. (no binge drinking on patch nights!)
redundancy and proper monitoring make life much, much nicer.