2025 · Full-Stack / Systems
22 Autonomous Apify Scraper Fleet
A fleet of 22 web scrapers orchestrated end-to-end with an AI agent supervisor inside a 900MB sandbox.
22 Scrapers · Agent-Orchestrated Build
The problem
Building and maintaining dozens of distinct scrapers manually against evolving website layouts and anti-bot protections is too slow. Hand-writing each crawler one by one was never going to scale.
The constraint
The scrapers had to operate inside a strict 900MB RAM server slice without memory leaks, cycle proxies to avoid rate limits, and abort cleanly without billing customers if a target website layout shifted.
What I built
An autonomous supervisor loop that mined search demand (targeting queries with demand_runs_30d >= 400), synthesized TypeScript / Crawlee scrapers, executed pre-flight dry runs, configured Pay-Per-Event pricing, and deployed to Apify cloud containers.
The outcome
22 production regulatory and compliance scrapers shipped and monitored, with automated canary tests preventing broken runs from ever billing users.