No that's backwards, the GPL tells you what you can do; it doesn't take away any rights you don't have without it.
The way I see it is, by default, if I give you the source for some software I own the copyright for, you're not allowed to distribute it, or works derived from it, at all. I'm the only person with a right to copy and distribute...the 'copyright'. You can do whatever you like with it on your own (unless I made you sign a EULA where you gave up some of your rights before you use it), but when it comes to distributing it yourself - forget about it.
So say for whatever reason I want to allow other people to distribute my code or works derived from it, I'd have to either relinquish my copyright, or grant them a license to distribute it. I might want to grant them a very permissive license saying "hey, distribute this as you like, I don't care" which is pretty much what the BSD licenses do. On the other hand, if I wanted to guarantee that all the end users have the freedoms espoused by the FSF, I might only grant you a right to distribute it under certain conditions (which is what the GPL says).
If you want it in a nutshell, I think the GPL was created specifically to allow rights holders to grant recipients of their work the right to re-distribute it without allowing them to withold rights from the people they re-distribute it to.