← All ideas

Robots: Allow longer runs and more pages for catalog crawls

NewRobotsIdea

Browse AI allows tasks up to 60 minutes on every paid plan (https://www.browse.ai/pricing); Apify runs are bounded only by the plan's compute. Robots caps a run at 180 seconds, 20 pages and 1,000 list rows (lib/robots-schema.ts), so a 30-page catalog or a detail-page crawl over 100 items cannot finish in one run. First scope: a long-run mode (up to 15 minutes, 200 pages, 10,000 rows) available to scheduled and API runs, with progress saved per page so a timeout keeps what was extracted. Priority: medium. Effort: M. Compared with Browse AI / Apify.

Comments 0

No comments yet.