[...] because those files are binary and very large
No VCS is meant to do this, neither Git, SVN and certainly not CVS. Those files don't belong in a VCS because you cannot make a diff between them. Small binary files (e.g. icons in a website) are a small nuisance, but there is no point in storing large binary blobs in a VCS regularly. What you need is a backup system, not a version-control system.
(2) permanently delete those files that I know I will no longer need
SVN allows to do this with svndumpfilter (and I was unaware CVS had any way to do this). And no it should not be made any easier, no one should be allowed to monkey around with the repository history with any less than admin rights. If you find yourself regularly removing files from a VCS, it means you have been adding too many useless files. Again, you want backup for this, not VCS.