A working Hiroshima style fission bomb isn't that hard to build, which is why the Little Boy bomb had that particular design. But getting the 64 kg of fissile materials you need to build one is a bitch. Unlike as depicted in many bad post-apocalyptic novels, you can't produce bomb grade nuclear fuel in a basement or a small cave. It requires massive industrial facilities and leaves evidence which is impossible to hide. That's why non-state actors have never been able to produce a nuclear weapon, despite the fact that such as weapon is highly desirable and some of them are well-funded. And there's getting the uranium in the first place; you need a theoretical minimum of about nine tons of purified but unenriched uranium to start, and in practice quite a bit more if you don't have forever to do it.
For a private actor, obtaining a nuclear weapon is extremely unlikely, barring some breakthrough in physics or chemistry.
Dropping a big space rock, on the other hand, is limited not by physics, as you suggest, but by economics. At present it's economically impossible, but if there were private space activities such as near Earth asteroid mining, everything you'd need to do it would be there, so the only missing piece would be intent. As for using lunar materials, the same applies, it's just farther off because the cost of getting stuff out of the Moon's gravity well means lunar mining is only attractive for materials destined for space use.
And note that even if obtaining a nuclear weapon were considerably easier than it is today, you still couldn't rule out an opportunistic space attack. It could be the guy running the mining scow, or even hacking an automated vehicle's software. That's something well within the capability of a wealthy individual, not to mention state actors.
At some point we're going to have to deal seriously with the space rock attack scenario. But that's decades off. When someone starts a project to move dense masses in space on the order of a metric ton or so, that's the time that governments need to step in with oversight. Right now it's still in the realm of sci fi.