Hacker Newsnew | past | comments | ask | show | jobs | submit | stall's commentslogin

Well, google published https://developers.google.com/search/docs/advanced/structure... which describes how it should be done


Many of these paywalls simply have either the full body in the response or will return it if you ask with the right user agent (e.g. the one that google uses). These plugins then either remove some overlays or impersonate the google bot.

Obviously that’s not how it should be implemented and that‘s also not what google suggests you to do as a publisher. Instead you should verify that the bot is actually google https://developers.google.com/search/docs/advanced/crawling/...


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: