Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> It is now significantly easier to build a search engine with its own index and crawl the entirety of the Internet than it was ten years ago.

ReCAPTCHA, hCaptcha, CloudFlare definitely don't make it easier to crawl the web than it was ten years ago



Thankfully (1) there just aren't that many useful websites anymore. Most content lives in walled gardens. Negotiate a few deals to get around CloudFlare for those select few and you're set.

[1] I'm being sarcastic.


'Many a true word was said in jest' as the aphorism goes. IME useful information has become somewhat concentrated into nodes, instead of trawling blogs for info it's all on stackoverflow/Twitter/whatever (depending on subject).


For certain populations of data and authors, it's true that centralization aids in discovery, distribution, discourse, maintenance, etc.

Unfortunately many of these channels are hooked up to hyper-growth valuation needs and this necessitates volume over quality.

Look at Quora, Pinterest, Medium, and LinkedIn. And I haven't even mentioned the biggest offenders.

I don't think I'm better served in 2020 than I was in 2008.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: