> 112. The integration of generative AI is perhaps the clearest example of competition advancing search quality. Google accelerated and launched its public piloting of Bard one day before Microsoft announced BingChat, the integration of ChatGPT’s generative AI technology into Bing to deliver answers to queries. Id. at 8272:4-7 (Reid); id. at 2670:10–2671:9 (Parakhin). (describing BingChat).
Perhaps a normative assertion on my part, but AI results have not "advanc[ed] search quality" by any metric that I am familiar with; in fact, AI results in Google mark the first time I have ever encountered incorrect or patently untrue information at the top of a Google query.
> This quality-reduction experiment correlated with only a 0.66–
0.99% decline in global search revenue. UPX1082 at 294. In short, this study demonstrates that a significant quality depreciation by Google would not result in a significant loss of revenues.
The gist of the story is the author wrote an article debunking the recipe myth that you only need 5 minutes. In their tests it was 25 - 45. Google snippet would report "about 5 minutes" and link to the article as a reference.
I think the information summaries at the top of searches have been wrong for a while.
It's certainly wrong, but its not out of place w/ the sheer number of recipes and blogs saying the same thing. It's parroting "common wisdom" which is wrong.
It's not as egregious as some of the generative AI results spewed out by the lowest-cost-inference LLMs they hooked up to Google search a few months back.
It was still a common occurrence for google to display incorrect results that weren't common wisdom.
I had tons of such experiences. One I remember was searching for "healthy body weight for x of age y" where x was man,woman and y was my age. Google said 50lbs higher than I expected. I clicked through to the article. The article said the average was this high number, not a healthy weight.
How familiar are you with the knowledge graph update process? There is actually a significant amount of spend going into reviewing the wrong answers to top queries and updating to fix/tweak them.
Your assumption basis is completely wrong, not only on when it came about. Bizarrely, Google spelled out that the AI answers weren't from an LLM in post-debacle PR, like it was a good thing
Google's main game is not serving better and better results: It's generating more clicks for advertisers on the first couple of results. The antithesis to Google's business model is someone asking an AI a question and receiving a bunch of direct answers for a monthly fee. Google's business model is people trying a bunch of queries (essentially shopping around), clicking on a whole set of links (each time costing the advertiser money without leading to a sale) until they get what they were looking for. It's the modern equivalent of confusing supermarket layouts with all the essential products stowed away in the back.
(So, disclosure that I work at Google, but nowhere near anything to do with search or ads.)
That is wildly distorting the way online advertisement works, mostly by a sideways denial that "advertising works" at all. The business model is that people who sell "X" want a link to their product displayed when potential customers consume content related to "X". There's nothing unclear or weird about this, and it's not fundamentally any different form print ads in the 1930's.
So to the extent that "Google's main game is not serving better results", that's true, in the same way that Apple's game is not selling a better mobile operating system. iOS and search both generate revenue by making the product they're actually selling more effective.
Totally agree with you about online ads, I worked in that space for years and that's not the goal with Ads. Years ago it became clear that Google was shifting towards taking over the lower level advertiser markets and providing services and automated systems to use Ads without requiring a marketing agency while providing higher levels of data analysis and tools for agencies to tap that market. Has nothing to do with stealing from advertisers, and a lot of Google's internal programs will likely drive more money to these smaller players.
> iOS and search both generate revenue by making the product they're actually selling more effective
In the abstract, sure.
The problem starts with separating financial incentives internally from product improvement and aligning it with short term revenue increases. Then you start separating the financial side from all other aspects of the business entirely, a la Jack Welch. That's where a lot of major US corporations are right now. When that happens, you end up with a conflict between what is best for the company and what is best for the people running the company, and you end up with product quality decreasing.
Right now from an outside perspective it looks like there was an internal conflict in the Search team at Google and the more product oriented leaders were pushed out in favor of the finance guys over the last year or two, with a side effect being lower search quality.
Except that the inscentives are totally different from an end user perspective. Apple is inscentivised to make the phone more attractive to the person using it, because they're the ones paying. With Google search, it's the advertiser that's paying. The only thing Google needs to worry about is keeping advertisers happy. That doesn't align with making search results better.
That's specious. Apple doesn't get paid by "consumers" either, they sell phones to retailers for the most part. "All they care about" is making retailers happy, right?
Obviously everyone who sells a product want to keep their end users (the "end" in "end users" is there for a reason!) happy, because if they don't they won't get paid. To argue that only your favorite is so incentivised seems silly.
The idea that a 0.66% decline in revenue would be insignificant made me snort. Increasing revenue that much would make many careers - it would justify the existence of a substantial origination. That’s billions of dollars annually.
In a very competitive market, reducing quality slightly could drastically reduce revenue.
E.g. if there were two brands of apple at the grocery store and one started putting a tiny worm in each apple, you would expect the worm apples to lose nearly 100% of their revenue.
That's an interesting example of a "slight" quality reduction. In apples, a slight quality reduction might be a bit of discoloration or a small bruise. The presence of a worm, on the other hand, causes the apple to be perceived as unsafe or inedible.
google had plenty of incorrect information at the top of the page before then, just instead of AI it was some algorithm pulling sentence fragments out of a page where the context was often the opposite of what you're asking
I think there is a clearly anti-competitive component in this, demonstrated by the recent AI training deal between Reddit and Google. The one that went hand in hand with preventing any search engine but Google from indexing Reddit: https://news.ycombinator.com/item?id=41057033
> 112. The integration of generative AI is perhaps the clearest example of competition advancing search quality. Google accelerated and launched its public piloting of Bard one day before Microsoft announced BingChat, the integration of ChatGPT’s generative AI technology into Bing to deliver answers to queries. Id. at 8272:4-7 (Reid); id. at 2670:10–2671:9 (Parakhin). (describing BingChat).
Perhaps a normative assertion on my part, but AI results have not "advanc[ed] search quality" by any metric that I am familiar with; in fact, AI results in Google mark the first time I have ever encountered incorrect or patently untrue information at the top of a Google query.
> This quality-reduction experiment correlated with only a 0.66– 0.99% decline in global search revenue. UPX1082 at 294. In short, this study demonstrates that a significant quality depreciation by Google would not result in a significant loss of revenues.