Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

When I first started it there was a couple of instances that my IP was blocked - despite being a residential IP behind CGNAT.

I then started randomising every aspect of the scraping process that I could: The order that I visited the links, the sleep duration between almost every action, etc.

As long as they don't implement a strict fingerprinting technique, that seems to be enough for now



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: