What about Wernicke's area and Broca's area? If damaged they produce causal changes to our use and processing of language. I agree it's not as simple "language brain" but there are physical structures in the brain that are directly related to language.