now imagine google has to check if every link it will show still exists. That's like billions of trillions of request per hour world-wide. And another thing is a privacy issue, I know, google doesnt give a shit about it, but some people do. Think that you search for 'smelly poo' and 10 different servers get HEAD request to articles that are top ranked in this keyword, potentially 9 of the servers won't get a human-made visit, 9 sysadmins or developers will find a way to abuse that metadata in the future.
I doubt Google account for 40% of Internet traffic. How would you even measure that !? What most likely happened was they saw a 40% drop in traffic on their servers while Google was down.
