If that's a major chore for you, may I suggest investing in a test suite, so that you don't have to go through manually and test all your patches?
Yes, that would help with testing, but also multiplies the work: now you have patches and the new test suite to maintain and shepherd through upgrades.
Testing patches actually wasn't the main obstacle. The far bigger time sink was when some patched piece of functionality moved over to a different location in the codebase. Then the engineer had to go and become familiar with their changes, find how things got rearranged, and adopt the patch to the new code. That's a lot of unnecessary work just to maintain the status quo.
As for whether we needed to upgrade so frequently - yeah, we actually did. Hopefully the OP doesn't, though.