At the risk of an appeal to authority fallacy. I do have some background, as I used to write drivers for two kinds of deep learning / DNN accelerators. One for inference that can slice and rotate memory views for TensorRT, and one for training that transparently distributes virtual memory on a multinode cluster.
We humans can easily understand an inference engine. It's not hard, and I would recommend checking one out, even if just from an introductory YouTube video. Or diving into something like llama.cpp is worth doing if you want to see the underlying math. The ugly bits are where it hooks into various backends, but there is a core piece that anyone with a decent C++ background can understand. Without C++ background, I'd recommend looking into simple-llm.
Starting with a simple but useful model like TinyLlama is a good point of research. Low requirements and the model is small enough to experiment on.
The models are a blob of data. Introspection is very limited, there is no history, and the weights (tensors really) are interdependent and even circular. So adjusting one weight can rippled through the model with unintended consequences. There is very little that one can formally reason about in a general way with a model. Maybe we'll have that kind of math one day, but right now we are stuck treating models as a black box.
So my response to anyone that claims that humans already understand what an LLM does is: Then show me.