The same way it is being done for drivers licenses. There are set of rules of what causes suspension of your driver license due to mental illnesses and there is a group of people who rule on a case by case basis. The same way is also being done for the courts. Somehow the courts can rule if you are sane or insane. This isn't a problem that hasn't been solved before.
Stringent background checks means less desirable people are given guns, which means fewer guns. It is all about limiting the amount of guns in circulation and having only responsible people own guns.
For example you could require by law that all gun owners keep their guns locked in a gun cabinet which the police will enforce by inspection of your home. That would limit the relatives of the gun owner of aquiring the weapons.