If you're in a small company but still managing 50 servers, what is your role on the CAB? You should advocate to be part of the CAB, at the very least, so that you can coordinate processes to streamline critical and security patches, and keep management informed of the process. If you are approaching this with a hostile or obstructive attitude, it doesn't reflect well on you and it injures your ability to get management to listen to you when it counts. A CAB can be a rubber stamp, for the most part, that ensures that there is at least documentation and a modicum of thought going into the maintenance of the company's infrastructure. The creation of a CAB is pretty reasonable, but the key is to be involved in its creation and ongoing existence as well so that you can eliminate the red tape while still documenting the process to CYA in case something goes wrong. And something, sometime, will go wrong.
It's possible that moving to a more validated and controlled environment will increase your workload, but again, documenting the volume of changes, etc. is to your benefit. That is the data you need in order to provide management the justification for getting a second admin hired. This is also the data that you can use to justify a big raise or promotion at your next review, instead of people wondering what exactly you do all day.
Also, as mentioned previously, you really should have WSUS or a similar solution to deploy patches across your organization, both to make administration easier and to help enforce a consistent environment across all the systems. If you patch everything piecemeal, it's quite difficult to tell whether a patch will break one system with a different set of patches than this other one.