not a single super large language model has beaten the state of the art in the key NLP tasks (POS tag, dep tag, coreference, wsd, ner, etc)
They are always only used for higher level tasks, which is tragic.
Why is that tragic? Classic NLP tasks are IMHO kinda pointless. Nobody _actually_ cares about parse trees, etc. These things were useful when that was the best we could do with ML, because they allowed us to accomplish genuinely-useful NLP tasks by writing code that uses things like parse trees, NER, etc. But why bother with parse trees and junk like that if you can just get the model to answer the question you actually care about?
I would not discount NLP tasks just yet. In practice, they are still used to solve problems like spellcheck, autocomplete and even words and text rewrites.
I'd argue that every one of those tasks is better solved with a neural LM. The fact that people still use traditional techniques does not mean those old techniques are better. It just means those tasks haven't caught up to the modern era yet.
Or maybe it's because MLM are too expensive to run. But that's hardly an argument that MLM should solve traditional NLP tasks.
if you can just get the model to answer the question you actually care about?
LOL your level of optimism is over 9000. Language models are shit, they regularly utter nonsense and have an impressive number of limitations (e.g. no continual learning). A neuro-symbolic system on the other hand can be incrementally improved upon, has continual learning/a memory, and is interpretable.
