← All ideas

Agents: Crawl a whole site or sitemap with include and exclude patterns

NewAgentsIdea

Chatbase crawls a website or sitemap and auto-resyncs weekly (https://www.chatbase.co/docs/user-guides/chatbot/data-sources.md); Voiceflow manages sitemaps with include/exclude patterns and a refresh schedule (https://www.voiceflow.com/docs/documentation/build/importing-data-sources.md). We allow 20 single URLs (AGENT_MAX_URLS) re-synced daily. A team moving from an old help center on another domain has hundreds of pages. First scope: add a site or sitemap URL, discover pages up to a cap (e.g. 500) with include/exclude path patterns, fetch through safeFetch with a crawl budget per cron tick, show the page list with status, and re-sync weekly.

Comments 0

No comments yet.