He is postulating that the way we deal with physical memory (an example of L2 and L3 caches is provided) demonstrates that, as we proceed with trying to form AGI out of our classical computer architectures, there are some fundamental problems for example larger caches are slower. With human intelligence, this doesn't always seem to be a problem for some humans. If you understand the "attention" part of recent developments in this field, he's saying that transformers are the most efficient way we got to achieve that and it's starting to look like a problem from a "physicality" standpoint as the author puts it.

The "physically" here is that larger caches are not just computationally larger banks of data but are actually physically larger and, by Euclidean distance, further away. Yet paradoxically the elephant nor blue whale is not the smartest brain on the planet, the distance from the center of my head to broca's region seems to have no effect on my elocution. Etc. Studying Einstein's brain doesn't do much (I guess the insulators are somewhat important?) for understanding Einstein's intelligence ... but that is 100% critical when understanding L2 and L3 caches on a die.

> Who is to say biological computers won't be built?

No one is saying that. We're just pretty sure it's not happening in our lifetimes.

I think you misunderstand the author's intent. He's not saying "The things I can't imagine are not going to happen". He's trying to argue that, "Look, the way things are going, the diminishing returns we are already seeing, the way our hardware works, this isn't going to get us to AGI." Of course, if you had some new architecture or weird "wetware" that somehow solved these problems I'm sure this article would concede that that's not the point.