Speculation yes, blind no. It's true that neither Musk nor Gates are AI academics/engineers, but they are both deeply involved in setting up and funding AI research, but the clincher is that they have huge wallets and have a long held
personally interested in the subject. As such they almost certainly have a better grasp on AI and it's potential usest than the proverbial Joe Sixpack.
It follows that, with or without fame, they are both "experts" relative to the general population. As such their opinion ranks as an "educated guess" and is preferable to that of the majority of "Joe Sixpack's" who base their concerns on Terminator, Bender or an obscure passage from an ancient religious text. In other words, those with knowledge of what is possible today are more informed about what that may lead to tomorrow. In the same vein I have an (old fashioned 1990-ish) degree in computer science which included an AI component, I have written numerous AI toys over the last 30yrs, and recently sat through the 2010 online MIT course* for AI (just for fun). None of this means that Joe is a moron, Joe is quite likely to be an "expert" in other fields compared to you or I.
Here's the rub with "AI" (or any complex and controversial issue). Due to a messy divorce I hadn't really been paying attention to what had been going on in AI during the 2000's, it blew me away, I showed my 'wife' who happens to be a (sought after) business 'expert' who's lectures attract large audiences. She shrugged and said "The voice thing is neat, but what's the big deal, it's
just looking up the answers on the internet, right?". To this day she simply does not have sufficient knowledge to recognize the problem. Just about everyone I have shown (other than fellow AI geeks) has a similar reaction. Not only don't they "get it", they don't even recognize "it" when "it" is talking to them. There's no offense intended when such an opinion is deemed "uninformed", it's actually a plea for Joe to familiarise himself with the problem before offering an opinion.
Personally, I'm not afraid of AI suddenly turning hostile, but "knowledge is power" so I am definitely concerned about what the "known behavior of the only high intelligence we've ever met" may do with such a tool/weapon. Given the track record of our species I don't think that is an unreasonable concern, in fact the last line of your post would seem to agree with it.
Now, if you actually take a few moments to (randomly)
listen to what these people are saying about AI in their speeches and interviews, you may find that their concerns are not that different to yours and mine and that the "SkyNet" hype is just the MSM doing their thing to "sex up" conservative (and unoriginal) speculation about the human tendency to use tools in every endeavor, including our inhumane endeavours.
* For anyone wanting to sharpen their existing AI knowledge (and make smarter toys), I highly recommend
MIT's online AI course. It took me about a month to watch and absorb all the lectures, I approached it as a "refresher" but also learnt some new tricks that were not available 25yrs ago. The guy running the show has trouble keeping his pants hitched up but he is simultaneously entertaining, intelligent, and down to earth, when I finished the series I wanted to sit down and talk "cabbages and kings" with him...