NLWeb protocol — agent-discoverable workspace endpoint

TL;DR: Every AskVault workspace can expose a public NLWeb endpoint at https://api.askvault.co/api/nlweb/{workspace_id}/ask. AI agents that follow the NLWeb spec (a growing list including ChatGPT, Claude with MCP, and Perplexity) discover and query your knowledge base natively — no scraping, no rate-limit risk, no broken extractions. Opt in per workspace; rate-limited per IP.

What is NLWeb

NLWeb is an open protocol proposed by Microsoft (R.V. Guha, who also designed Schema.org and RSS) that standardizes natural-language queries on websites. The pitch: every site becomes an AI app, and both human users and AI agents query it through the same standard interface.

Each NLWeb instance is also an MCP (Model Context Protocol) server, so agents can choose between the conversational HTTP shape and the structured MCP tools shape depending on what they support best.

Why turn it on

If you do nothing, AI agents (ChatGPT browsing, Perplexity, Claude with web access) crawl your public site. They scrape rendered HTML, run their own extraction, and answer questions about your business based on whatever fragments they pulled. This can go wrong in three ways:

Wrong information — extractors miss the page that actually has the answer.
Stale information — they cite a cached page from weeks ago.
No attribution — answers presented as the agent's own with no link back to you.

With NLWeb enabled, those same agents can query your workspace directly. They get:

Your canonical, indexed knowledge — the same pipeline that powers your chat widget.
Source citations in every response, with document name and snippet.
Real-time freshness — answers reflect your latest re-crawl, not a six-week-old cache.

You participate in the agentic web on your terms.

Enable NLWeb for a workspace

Dashboard → Workspace settings
Toggle NLWeb endpoint to on.
Note your workspace ID (shown at the top of settings).

That's it. Within seconds, agents can hit:

POST https://api.askvault.co/api/nlweb/{your_workspace_id}/ask

Wire format

Request

POST /api/nlweb/{workspace_id}/ask
Content-Type: application/json

{
  "query": "Does the Starter plan include WhatsApp?",
  "site": "https://yourcompany.com",
  "locale": "en",
  "previous_responses": [
    {"role": "user", "content": "What's your pricing?"},
    {"role": "assistant", "content": "Starter is $29/mo, Growth $99/mo..."}
  ]
}

query — the user's question. Required.
site — optional origin reference. Logged for analytics; doesn't gate the answer.
locale — ISO-639-1 code. AskVault auto-detects language from query if absent.
previous_responses — optional chat history so the agent can maintain a thread across calls.

Response

HTTP/1.1 200 OK
Content-Type: application/json

{
  "answer": "Yes, the Starter plan ($29/mo) includes the WhatsApp channel...",
  "answer_type": "text",
  "sources": [
    {
      "name": "Pricing page",
      "snippet": "Starter plan includes WhatsApp Business API integration, web widget, and email assistant...",
      "relevance": 0.91
    }
  ],
  "conversation_id": "conv_abc123"
}

answer_type is "text" today. Future iterations will return "structured" when the workspace's agent picks a GenUI component (carousel, link cards, etc.).

conversation_id can be echoed back as previous_responses items on the next call to maintain a continuous thread.

Discovery endpoint

Agents that crawl your domain looking for an NLWeb endpoint can also hit:

GET /api/nlweb/{workspace_id}

Returns a small JSON blob describing the workspace, the ask URL, and the MCP URL (if MCP is enabled). Public; safe for agents to crawl as part of discovery.

Embedding the discovery hint on your site

Add this to your site's <head> so spec-aware agents auto-discover:

<link rel="nlweb" href="https://api.askvault.co/api/nlweb/{workspace_id}/ask">

When NLWeb-aware agents (Bing Copilot, ChatGPT browsing) crawl your site, they pick up the link and start routing future questions about your business through your workspace endpoint instead of scraping.

Rate limiting

Per (workspace, IP) token bucket: 30 requests per 60 seconds. Enough for any legitimate single-agent session; tight enough to deny a runaway bot.

Hitting the cap returns HTTP 429 with a clear message. Owner-plan monthly quota also applies — NLWeb queries count against your workspace's chat quota the same way API queries do.

Security model

Public read — no auth on the request. NLWeb's whole point is to be agent-discoverable.
Owner opt-in — the nlweb_enabled flag is default-off. Nothing surfaces until you explicitly flip it.
Plan-quota gated — abusive query volume burns the owner's plan quota, same as the chat widget.
No mutating tools — the NLWeb surface is read-only. Skills that mutate state (collect_lead, escalate_to_human) require the in-widget confirmation flow and don't run on NLWeb calls.

What's NOT in v1

Streaming responses — v1 returns the complete answer in one HTTP response. Streaming arrives when the spec's streaming variant stabilizes.
Structured (GenUI) answers — answer_type is always "text" today; carousel/card responses ship next.
Per-end-user identity verification — every NLWeb caller is treated as anonymous. Agent-side identity (the actual end user behind the agent) is the agent's responsibility to surface.

Implementation reference

Source: backend/app/routers/nlweb.py. The Workspace model gained an nlweb_enabled Boolean (default off) to gate access.