The shortcoming of most RAG-based approaches is the assumption that the question resembles the answer in a way that also jives with the embeddings model. Thus far, I’ve not seen strong evidence (or in my testing) that this is true or works well, but at least citation allows for better assessment. The problem seems to be that we don’t have a good feedback loop for ranking RAG retrieval, as we have for LLMs with things like DPO.
100%. This is why RAG and "classical search" will converge for non-trivial use cases. The folks who are doing RAG well still rely on many tried-and-true tricks of the trade: combining semantic search with keyword-based search, using graphs, doing re-ranking, etc. etc. Yet most discussions of RAG on the internet seem to promise consistently awesome query output by just jamming together some embeddings and an LLM, which doesn't pan out in practice.
