Yes, the problem would be due to attention being limited to the context. Reasoning requires constructing a model that can be referred to and updated. This is what CoT attempts to do but it is limited when expressed in natural language and being append only.