Skip to content
AskVault Docs
Try Free →

Cookies for authenticated crawls

Written by , Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 4 min read

When to use

Three cases:

  1. Customer portal docs that require login.
  2. Paid-content pages for subscriber sites.
  3. Internal wiki accessible only to authenticated users.

For OAuth-supporting sources, prefer OAuth where AskVault has it. Cookies are a fallback for sites without an API or OAuth path.

Setup

Under 5 minutes:

Step 1: log in to the target site

In your browser, log in normally.

Open DevTools (F12) > Application (or Storage) > Cookies, and copy the relevant cookie names and values for that site — or use a browser extension like "Cookie Editor" to export them as a name=value; name2=value2 string, or as JSON.

Step 3: paste into AskVault

  1. Knowledge Hub > URL Crawl source > Advanced options.
  2. Paste the cookie string or JSON into the Cookies field.
  3. Save, then start (or re-run) the crawl.

AskVault sends the pasted cookies as a request header for that crawl. Behind-login pages become reachable for the pages that crawl visits.

What happens after the crawl

This is the part that trips people up:

  • The cookie string isn't stored. It's used for the one crawl run you triggered, then discarded — there's no persistent cookie storage to inspect or manage later.
  • A scheduled re-crawl won't reuse it. If the source has a recurring sync schedule, later automatic runs run without the cookies you pasted, since nothing was saved to replay. Re-paste fresh cookies and manually re-run the crawl when you want an updated authenticated pass.
  • AskVault doesn't capture cookie rotation. Even within a crawl, if the target site issues a new session cookie mid-crawl (Set-Cookie on a response), AskVault does not adopt it for subsequent requests — this is a deliberate safety choice, since the underlying HTTP client is shared across other customers' crawl jobs and auto-adopting a Set-Cookie could leak one tenant's session into another's requests.

Cookies typically expire:

  • Session cookies. Expire on browser close.
  • Persistent cookies. Last days to months.

Either way, once the cookie you pasted expires, subsequent pages return unauthenticated (usually a login-wall page), so those pages won't have useful content indexed. Re-extract fresh cookies and re-run the crawl to pick up the real content again.

Security considerations

Cookies are sensitive:

  • Treat like passwords. Don't share or commit them anywhere.
  • They're not persisted by AskVault — there's nothing sitting in a database to leak later, but the cookie string does pass through AskVault's servers for that one crawl, same as any other request data.
  • Use a dedicated service-account login rather than your personal account, so a stale or over-shared cookie doesn't expose your own session.

Limits

  • Cookie field length. 4,000 characters total (the whole string or JSON blob you paste), not per individual cookie.
  • Persistence. None — re-paste for every crawl run that needs authenticated access.
  • Auto-rotation. Not supported — a site that rotates its session cookie mid-crawl will start failing partway through.

Alternatives

If cookies are too brittle:

  • OAuth, where AskVault supports it for the source type. Preferred when available.
  • API tokens, for sources with a token-based API (e.g. Confluence, Jira).
  • Direct file upload. Manually upload PDFs or exports of the behind-login content — see file uploads.
  • Public mirror. Many docs sites maintain a public mirror for SEO that doesn't need auth at all.

Common pitfalls

Cookies expire partway through a large crawl. Initial pages succeed; later ones fail once the session lapses. Re-extract and re-run for the pages that failed.

Cookie doesn't match the target domain. The cookie's domain must match the site you're crawling (or be a parent domain).

Expecting a scheduled re-crawl to keep working. It won't — cookies aren't stored, so a recurring sync on a cookie-gated source runs unauthenticated after the first pass. Re-paste and manually re-run when you need a fresh authenticated pass.

MFA-protected accounts. A session cookie captured after MFA works until it expires, same as any session cookie — but AskVault doesn't help you get past MFA itself. Use a service account without MFA if you need this to work repeatedly.

FAQ

Is this safe?

If you use a dedicated service account (not your personal admin login), yes — and since AskVault doesn't store the cookie afterward, there's no persisted copy to worry about beyond that one crawl request.

Can I use this for a site I don't own?

Only if you have explicit authorization. Respect the site's Terms of Service.

Will a scheduled re-crawl keep authenticating automatically?

No — cookies aren't retained between runs. Re-paste them and trigger the crawl manually whenever you want an updated authenticated pass.

What if the site rotates cookies aggressively during a single crawl?

AskVault doesn't adopt a rotated session mid-crawl, so a very aggressive rotation policy will make later pages in that same crawl fail. Consider an alternative (OAuth, API token, or file upload) for that source.

Was this page helpful?