CSS selector targeting for URL crawling
Why a CSS selector helps
Left on the default "smart" mode, AskVault already strips common page chrome — <nav>, <footer>, <header>, <aside>, and elements like .sidebar, .cookie-banner, .breadcrumbs, ARIA landmark roles, and similar — before indexing. For most sites that's enough. A custom CSS selector is for going further: pointing extraction at exactly the content element (<main>, .post-content, article.entry) when the automatic chrome-stripping still leaves in something you don't want indexed, like a "Related posts" module or an author bio card.
Setup
- Knowledge Hub > URL Crawl source > Advanced options.
- Enter a CSS selector in the "CSS Selector" field (e.g.,
article.post-content). - Re-crawl to apply it — it's used on every page the crawl visits.
There's one selector field, not separate include/exclude boxes. AskVault takes the first element that matches it on each page and indexes only that element's content.
Common selector patterns
main. Most semantic-HTML sites have a main element.article. Blog posts.#content. Common ID..post-content. WordPress..entry-content. Ghost.[role="main"]. ARIA-based.
Test the selector in your browser's DevTools (document.querySelector('your-selector')) against a real page before saving it — AskVault doesn't have a built-in selector preview today.
What "smart" mode strips automatically
When you leave the CSS Selector field blank, every crawl already removes:
<nav>,<footer>,<header>,<aside>.sidebar,.menu,.cookie-banner,.cookie-notice,.navbox,.breadcrumb/.breadcrumbs,.pagination,.masthead,.megamenu,.site-header,.site-footer,.skip-link,.toolbar- ARIA landmarks:
[role="banner"],[role="navigation"],[role="contentinfo"],[role="complementary"],[role="search"],[role="menu"],[role="menubar"]
This list isn't user-editable — it's the same on every plan and every source. A custom CSS selector replaces this behavior entirely (it scopes to one element instead of stripping a fixed list), it doesn't add to it.
Limits
- One selector per source, applied to every page in that crawl.
- Selects the first match only — a page with multiple candidates needs a more specific selector.
- Standard CSS3 syntax.
- Re-crawl to apply a changed selector — about 30 seconds per MB of resulting content.
Common pitfalls
Selector too specific. Misses content on pages with slightly different markup. Use a broader selector.
Selector matches the wrong element first. Since only the first match is used, a page with several elements matching your selector may extract the wrong one. Tighten the selector so it's unambiguous.
JS-rendered content. The selector is applied after the page renders, but only for hosts where the crawl has escalated to the browser tier. A selector that only matches JS-injected content may return nothing on the lighter-weight fetch tier.
Expecting separate exclude selectors. There isn't a second field for that — either your one selector already scopes past the unwanted content, or leave it blank and rely on smart mode's built-in stripping.
FAQ
Can I use different selectors per URL pattern?
Not today — one selector applies to the whole source. For very different page templates on the same domain, consider two separate sources with different starting paths, each with its own selector.
Does this work for JS-rendered SPAs?
Yes, once the crawl has escalated to browser rendering for that host — AskVault applies the selector to the fully rendered DOM in that case.
Is CSS selector targeting available on every plan?
Yes — it's not gated to a specific tier.