Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

faustian bargain: you trade scrape ability for your website for one click buy of your product.


It's hardly faustian if your goal is to sell your product. If you have some weird hybrid sales/content-based ad revenue setup, then yeah, maybe it's faustian.


The document suggests unblocking OAI-SearchBot in robots.txt, and then clarifies:

> OAI-SearchBot is for search. OAI-SearchBot is used to link to and surface websites in search results in ChatGPT's search features. It is not used to crawl content to train OpenAI’s generative AI foundation models. Learn more (opens in a new window).


If you sell a product you want everybody to scrape as much as possible from your website.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: