Guns, knives, fireworks, blowtorches, and chainsaws are dangerous by the very nature of what they are intended to do. Even small children immediately understand their capacity to destroy things. The CPSC does not ban chainsaws because they cut or blowtorches because they burn. But it does ban toys when they tend to cause harm in totaly unexpected ways.
Children do not immediately understand the capacity of these things to destroy. There are plenty of unfortunate accidents involving children and firearms. Responsible parents understand the capacity of these things to destroy and teach their children about them accordingly. The problem with rare earth magnets is some people fail to take heed to the hazard these can potentially cause despite multiple warnings on the packaging. If you can understand why a child shouldn't play with a firearm you should also be capable of understanding the warning label on that desk toy you just bought yourself. The warning label is there for a reason.