It produces token strings that are statistically likely to be a match for any given prompt based on text it's been fed. But the reason it gets things wrong is not the same reason we get things wrong. It gets things wrong because the probabilities matched but the actual meaning did not.

It sounds great that RAG stands for Recursively Assembled Grammar (in a discussion about LLMs), only it doesn't. The LLM generated text that even explained RAG as a recursive grammar applied in the context of LLM usage, but RAG stands for Request-Augmented Generation. When I pointed this out the system output "Oh, of course, that is also a thing and it means..." But the system had no awareness. Not of the meaning of the text it was producing, and not of any sort of self.