The difference, of course, is that we currently DO actually know EXACTLY how an LLM works. We can snapshot the model and seed the random number generator to make it generate exactly the same output from exactly the same input every single time. We can pause it, set breakpoints, inspect and dump data structures.
It IS simply a program running on a CPU, and using RAM.
Is it possible that's all humans are in the end? Sure its possible, I can't prove otherwise. But we are not remotely in a position to assert that its the case.
You invoke philosophy which is entirely appropriate. There are fairy tales for example of artists painting things so realistic that they come to life. And it poses an interesting question here: is there is a difference between a simulation and a real thing? Can a simulation of life, be "alive"? Or must it forever remain a simulation.
And a related, and perhaps ultimately simpler question is can a *turing machine simulation of life* be "alive".
A lovely illustration of the question:
https://xkcd.com/505/
Can what you and I perceive as our lives, the universe around us, and everything REALLY be underpinned by some guy in a desert pushing pebbles around in a big desert somewhere?
Can the arrangement of stones in a desert, and some guy updating moving them aorund, in some pattern he interprets as representing the information that describes our universe actually "BE" our universe?
Is is the pattern of rocks is JUST a pattern of rocks. Is the guy moving them around JUST moving them around. Is the interpretation of the pattern as a representation of the state of a universe, just that, a representation?
Or you truly think there is a galaxy with a planet with people on it having a conversation on slashdot,'frozen in time' waiting for some guy to move the rocks into the next pattern and that somehow results in the experience we are sharing right now?
Or put more succinctly - can an abstract representation of a thing be the thing? be it bits in a DRAM module memory or pebbles arranged in the sand? can it be the thing it represents? Can the painting of a zebra if its done skilfully enough be a zebra?