Working on Strot - an AI agent that reverse-engineers website APIs for scraping.
Instead of DOM scraping, it intercepts AJAX calls and figures out which API endpoint gives you the data you need. Uses visual analysis + fuzzy matching to identify the right call.
The use case: scraping product reviews, paginated listing data (products), etc. Existing AI scrapers either didn't work or were very slow and costly. A product with 1000 reviews takes 10+ minutes with Playwright, costs $10 with LLM scrapers. With Strot? 10 seconds via direct API calls.
Being used in production by a couple of clients. Would love feedback!
Very neat - I imagine you could even use this as a web scanner to identify security misconfigurations in API implementations (e.g. broken access control)
Instead of DOM scraping, it intercepts AJAX calls and figures out which API endpoint gives you the data you need. Uses visual analysis + fuzzy matching to identify the right call.
The use case: scraping product reviews, paginated listing data (products), etc. Existing AI scrapers either didn't work or were very slow and costly. A product with 1000 reviews takes 10+ minutes with Playwright, costs $10 with LLM scrapers. With Strot? 10 seconds via direct API calls.
Being used in production by a couple of clients. Would love feedback!
Blog: https://blog.vertexcover.io/strot-is-a-api-scraper GitHub: https://github.com/vertexcover-io/strot