Other parts of the gun are often serialized, but the serial on the receiver is the one that is considered the serial number of the gun for legal purposes. For collectors, matching numbers on all parts still fetch a premium, especially for antique or just old firearms, such as WW2 rifles.
Speaking of old guns, this is one other interesting side effect of treating the receiver as the gun, and all other components as appendages. Federal law defines a category of firearms called "antique", which is any firearm manufactured on or before 1899. Those are basically completely out of the scope of all existing gun control legislation on both federal and state level (I believe Hawaii is the only exception), unless they fall under NFA (full auto, short barrels etc). You can order them online and have UPS deliver them to their doorstep, no background checks, nothing. If you're a felon, you can still legally own one. And so on.
Now remember that when we say "gun", we really mean "receiver" here. This means that it's perfectly legal to take a receiver from an antique gun, replace the rest with newly manufactured components, and the result will still legally be considered antique. And receiver is the part that normally gets least wear on a firearm (because, on one hand, it's usually metal, and on the other hand, it's not the part that gets most mechanical stress, unlike the bolt assembly and the barrel).
Here is one example of such a thing: an Imperial Russian Mosin-Nagant receiver from 1896, which ended up in the stocks of the Finnish arsenals (quite possibly captured during Winter War or Continuation War, or else it could come from the arsenals they inherited from the Russian Empire when they declared independence), and was then remade into a new Mosin rifle sometime in the 60s, with new barrel, bolt, stock, and possibly the trigger group as well. The resulting rifle shoots just as well as any other modern bolt-action rifle, and ammo is cheap and plentiful.
It's not limited to just bolt-action rifles, either, though those are the most plentiful. But there is a number of antique Mauser C96 handguns around, as well - not quite on par with modern handguns, obviously, what with a fixed 10-round magazine loaded from stripper clips, but still a very capable firearm in its own right. There are plenty of antique revolvers, too.