My point is that physical reality cannot be made of something that doesn't exist physically, such as mathematical objects, abstract concepts or more generally ideas of any kind. Do you not agree that this must be true?
You lack the capability to judge what those things are, not because of some personal deficiency but because humans lack the ability at this level. If you dug in far enough you'd find your definition of what an abstract concept is circular anyhow, from the looks of it.
Proving the universe isn't made out of "mathematical objects" in particular is equivalent to the difficulty of proving it's not a "simulation". This is one of the red lines that tells you you've gone too far; you can't prove that. You can't even non-circularly define such a thing in this context anyhow, let alone prove anything.
It is true, but also kind of trivial. More subtly, one can say that we tend to call what is real using the same words that we use for mathematical objects that model them.
I don't know that we do that. For instance, physicists are able to describe motion in physical terms and don't insist that it literally is a vector. It only happens with quantum fields.