I am CEO and one of cofounders of Vectara. We are very proud of the release of this open source eval model. We certainly would like to add more LLMs to the scorecard, and would love to collaborate with others to make the evaluation model even more accurate. Please reach out to bader@ or simon@ if interested.
What are the limitations and challenges of Boomerang in terms of scalability to a large corpus with tens of millions off questions? (I know answer as I am one of the founders of Vectara, asking this for the benefit of others)
How does Boomerang handle the trade-off between speed and accuracy? Does it sacrifice the quality of the results for faster response time? (I know answer as I am one of the founders of Vectara, asking this for the benefit of others)
The metrics presented in the blog post are those of our production model. When designing Boomerang, we tried to balance latency and search relevance in a manner that strikes the right balance for most use cases.
On the other hand, GTR-XXL is an example of a research model that biases in favor of search relevance, at the expense of latency. It's not really practical to deploy in production environments as a result.
We, humans, are preconditioned to be linear in our extrapolation (as opposed to exponential) thanks to our hunter ancestors (and FPS games!). It is very clear that the rate of advancement of Large Language Models is super-linear, if not exponential.
Hence, I indeed predict that keyword search will be completely supplanted in the next 5 years as a mechanism for search.
Of course we will still need to do lookups for ISBNs and generic ids, but that isn't keyword search, that is index lookup functionality.
Case in point: take a look at Meta Research's Contriever model (https://github.com/facebookresearch/contriever), which already matches keyword techniques in efficacy without any supervision.
This is only the beginning, come build the future with us, we see it very clearly :)
