No, if you violate the GPL, you violate the GPL. You do not necessarily violate copyright.
The GPL says that you do not have to accept its terms, and can simply abide by normal copyright rules instead. So, unless you're doing something that would otherwise violate copyright, it doesn't even apply. And you can't violate the GPL when it doesn't even apply!
So that only leaves cases where 1. you're violating what copyright law would allow, but following the GPL (which is fine) or 2. violating what copyright law would allow and violating the GPL. Thus, if you're violating the GPL, you're violating copyright law.
It's really that simple.
Specifically, you may modify the source code but you must publish your modifications if you re-distribute.
That's not a restriction. Copyright law doesn't allow you to redistribute in the first place, so that's a merely a limited grant of permission. You can redistribute (which copyright law doesn't allow) if and only if you do X. That doesn't make X a restriction. It makes X a contingent condition on the permission you wouldn't otherwise have. The sum total is still more permissions than you would have had otherwise. Even if you don't like the specific conditions.
After all, if you don't like the GPL's conditions, you can ignore them and follow copyright law instead. So how can that possibly be a restriction of any sort? The only immutable restrictions are those which are not allowed by copyright law or by the GPL. And the only reason you have to obey those restrictions is because they're part of copyright law. The GPL doesn't restrict you at all. Copyright law does all the restricting. The GPL simply outlines the very specific terms under which you can ignore the normal restrictions of copyright law.