It cannot even actually explain it's answers. Instead it treats a request to explain as a request to come up with a plausible explanation.
There is an increasing body of evidence that this is also how human reasoning normally works. Usually, we make decisions first and only afterwards (if necessary) engage our reasoning ability to generate an explanation of our decision. Experiments show that we're actually just as good at explaining decisions that were the opposite of the ones we made, assuming we can first be tricked into believing that they were our decisions.
Of course there are cases where we engage in lengthy, logic-based sequences of reasoning to derive a conclusion, especially if we've spent many years in higher education that trains us to think that way. This is particularly true when we reason in groups, exchanging ideas and critiquing others' reasoning and conclusions. It appears that for most of human history this is probably the primary way we employed reason, though in such social reasoning the goal is less to ensure accurate results than it is for the most skilled and smartest debaters to get their way (thereby improving their genes' probability of survival)... though it's still the case that making better arguments is the best way to "win" and that better arguments are more likely to be at least directionally correct. I'm sure you've noticed that your own reasoning gets better when you attempt to order your thoughts to potentially present them to a critical listener. This even works quite well when the "listener" isn't able to be critical; hence the utility of "rubber duck debugging". Of course, discussing with an intelligent and listener who is engaged in trying to critique your reasoning and offers their own counterpoints for your critique works even better.
It shouldn't surprise us that AI exhibits similar characteristics. Their core structure is at least notionally modeled on the neural networks of our own brains, and they're trained on large bodies of our text. Their evolution over the last few years also models our thought processes. The first hugely-successful LLMs just produced output without any sort of "thought process". That quickly ran into limitations, then someone came up with the idea of feeding their output back in and enabling them to engage in "self-talk" during which they could evaluate, criticize and improve their conclusions. This "reasoning overlay" looks at least superficially similar to the stream-of-thought self-talk that we engage in when thinking our way logically through a problem. And again, much like us, when LLMs do engage in a longer chain of thought, they actually can explain their conclusions by looking at the sequence of steps they went through.
There are also ways in which human and AI cognition differs substantially, of course, and in many ways AI thinking is far inferior to human cognition. Some of this is based on their different approach to context. An AI with a 1M token context window can keep at its "fingertips" far more precise context then we can -- we struggle to remember more than about 7 numbers for more than a few seconds -- but we're much better at maintaining a "big picture", which we achieve not by remembering everything but by being better at selectively extracting the most salient bits (I expect that improving AI ability to do this will be one of the next large steps forward in capability).
Any way, the point is that AI "thinking" is far more similar to ours than you credit it to be.
We do not make a real AI, it will not try to kill us.
No one can really define what "real AI" is. As for whether it will try to kill us... it is utterly impossible for us to say what AI might or might not do as it continues to improve. Your point about its reward system is potentially valid, but with a little thought it's pretty easy to come up with ways in which a superintelligence might destroy us while trying to serve us. Even more to the point, we don't know what the "derived" motivation system inside the systems is. We can't look inside. Observing the self-talk stream-of-consciousness gives us some insight, insight that we actually can't have into humans' thought processes, but it's not definitive.
Do some reading about "instrumental" vs "final" goals. I highly recommend Nick Bostrom's book, "Superintelligence". It's pretty old and quite dated in a lot of ways, but it gives you the tools to think usefully about these questions.
