Skip to content
AskVault Docs
Try Free →

Full-site crawl

Written by , Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: 4 min read

When to use full-site crawl

Three patterns:

  1. Initial onboarding. Index your site without needing a sitemap.
  2. No sitemap available. Older sites or CMS without auto-sitemap.
  3. Discovery crawl. Find content you didn't know existed.

For known URL lists, prefer batch URL ingest. For sitemap-driven crawls, use sitemap crawl.

Setup

  1. Knowledge Hub > Add Source > URL Crawl > Full Site.
  2. Enter starting URL (typically homepage).
  3. Set max depth (1 to 5 hops from the seed URL; default 2).
  4. Set include patterns (e.g., /docs/*, /blog/*).
  5. Set exclude patterns (e.g., /admin/*, ?print=true).
  6. Click Start.

Within 60 seconds, crawl begins.

Depth

  • Depth 1. Homepage plus its direct links.
  • Depth 2. Default; two hops from the seed URL.
  • Depth 5. Maximum.

Deeper depth catches more content but takes longer. For typical marketing sites, the default depth of 2 usually needs bumping to 3-4 to reach content a couple of clicks deep; 5 is the ceiling.

Include / exclude patterns

URL globbing:

Include /docs/*. Only URLs starting with /docs/.

Exclude *?print=*. Skip print-friendly versions.

Exclude /admin/*. Skip admin areas.

Exclude /tag/* and /category/*. Skip taxonomy pages (just lists, not content).

Order: include first, then exclude. About 80% of crawls benefit from at least one exclude pattern.

By default, crawler follows:

  • Same-origin links. Within your domain.
  • Subdomain links if explicitly enabled.
  • Anchor links (#hash) deduplicated.

Crawler doesn't follow:

  • Cross-origin links. External sites.
  • rel="nofollow" links.
  • robots.txt-disallowed paths.

Robots.txt respect

By default, AskVault respects your robots.txt:

  • User-agent: AskVault/1.0 rules apply — this is the identifier the crawler evaluates rules under (not "AskVault-Bot").
  • User-agent: * rules apply as fallback.
  • Disallow rules skip those paths.

If you want AskVault to crawl despite robots.txt (your own site you own), set "Ignore robots.txt" in source settings.

Re-crawl behavior

After initial crawl:

  • Scheduled re-crawl daily, weekly, or monthly.
  • Incremental. Only changed pages re-index (detected via ETags, Last-Modified).
  • Full re-crawl any time on demand.

Incremental re-crawl typically completes in about 10% of the initial-crawl time.

Cookies for authenticated crawls

For sites with auth-gated content, see cookies for login crawls.

Performance considerations

For very large sites (1,000+ pages):

  • Crawl runs in background. Bot becomes queryable as pages complete.
  • Parallel up to 30 in flight; 8 per host.
  • About 5 to 15 minutes per 100 pages depending on site speed.
  • The crawl stops selecting new pages once your plan's content storage cap is reached — not at a fixed page count. Pages beyond that are deferred and shown as an upgrade opportunity, not silently dropped.

For sites whose content would exceed even the Business plan's 100 MB cap: contact us about Enterprise, where the content cap is custom-negotiated.

Audience tags during crawl

Apply audience tags to all crawled URLs:

  • Per-source default. All URLs public.
  • Per-path overrides. Use globs (/internal/* sets tag internal).

Limits

  • Pages per crawl. No fixed ceiling — bounded by your plan's content storage cap (5 to 100 MB; custom on Enterprise), not a page count.
  • Max depth. 5 (default 2).
  • Per-host throttle. 8 in flight.
  • Re-crawl frequency. As fast as daily.

Common pitfalls

Crawl gets stuck. Site rate-limiting or anti-bot. Try lower per-host throttle.

Indexes pages you don't want. Add exclude patterns. Re-crawl.

Pagination loops. Calendar pages or ?page=1...?page=999. Exclude with *?page=* if not needed.

JS-rendered content missed. Most sites work; SPA sites may need explicit JS rendering. AskVault auto-detects.

FAQ

Can I crawl a competitor's site?

Yes if their robots.txt allows. Don't violate Terms of Service.

Does the crawl affect my site's analytics?

Page requests rotate through standard browser User-Agent strings rather than a distinct bot signature, so they won't stand out from — or be filterable as — regular traffic in most analytics tools. For robots.txt purposes specifically, the crawler evaluates rules under User-agent: AskVault/1.0.

Will my server be hit hard?

Per-host throttle caps at 8 in flight. Light load typically.

Was this page helpful?