Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You left out the part that says Perplexity-User generally ignores robots.txt because it's used for user requested actions.

> Since a user requested the fetch, this fetcher generally ignores robots.txt rules.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: