POST /v1/query/stream, Server-Sent Events reference

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan

Last updated: June 18, 2026 · 5 min read

TL;DR

Stream a chat response token by token over Server-Sent Events. First token in under 300 ms, full response in 1.5 to 4 seconds. Same request body as POST /v1/query. Up to 15,000 streaming messages per month on Business across 13 channels.

Endpoint

POST https://api.askvault.co/v1/query/stream

Same request schema as POST /v1/query. The response is a Server-Sent Events stream instead of synchronous JSON.

Authentication: Authorization: Bearer ak_xxx. See authentication.

When to use streaming

Two cases where streaming wins over synchronous:

1. Live Chat UI. The customer sees the bot "typing" word by word. Perceived latency drops dramatically even though total latency is the same.
2. Long answers. Synchronous calls might time out at 30 seconds for very long responses. Streaming sends tokens as they're generated, so there's no single 30-second boundary to hit.

For backend automation (a nightly job processing emails), synchronous is simpler. For any user-facing real-time UI, streaming.

Minimal example

curl

curl -N -X POST https://api.askvault.co/v1/query/stream \
-H "Authorization: Bearer ak_xxx" \
-H "Content-Type: application/json" \
-d '{"query": "How does pricing work?"}'

Python

import os, requests

with requests.post(
  "https://api.askvault.co/v1/query/stream",
  headers={"Authorization": f"Bearer {os.environ['ASKVAULT_API_KEY']}"},
  json={"query": "How does pricing work?"},
  stream=True,
  timeout=60,
) as r:
  for line in r.iter_lines():
      if line and line.startswith(b"data: "):
          print(line[6:].decode())

Node.js

const response = await fetch("https://api.askvault.co/v1/query/stream", {
method: "POST",
headers: {
  "Authorization": `Bearer ${process.env.ASKVAULT_API_KEY}`,
  "Content-Type": "application/json",
},
body: JSON.stringify({ query: "How does pricing work?" }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop();
for (const line of lines) {
  if (line.startsWith("data: ")) {
    const event = JSON.parse(line.slice(6));
    if (!event.done) process.stdout.write(event.token || "");
    else console.log("\nSources:", event.sources.map(s => s.document_name));
  }
}
}

The curl -N flag disables output buffering so you see tokens as they arrive.

Event format

Every data: line is a JSON object. Read the done boolean to distinguish token events from the final event.

Token events (done: false)

Partial answer text. Many of these arrive as the LLM generates.

{"token": "To ", "done": false}

Concatenate token values in order to assemble the full answer.

Status events

Before tokens arrive, the server emits status events describing what the pipeline is doing. These are for UI display (animated progress indicator) and can be ignored in backend consumers.

{"event": "status", "stage": "understanding", "text": "Understanding your question…"}

Common stages: understanding, searching, reading, writing.

Final event (done: true)

Always the last event. Contains the full sources array and usage stats.

{
  "token": "",
  "done": true,
  "sources": [
    {
      "document_name": "Refund policy",
      "chunk_text": "Refunds are available within 30 days...",
      "url": "https://acme.co/policies/refunds",
      "relevance_score": 0.94
    }
  ],
  "tokens_used": 187,
  "conversation_id": "conv_xxx",
  "message_id": "msg_xxx"
}

After this event the server closes the connection.

Error events

Stream-level error. Always has done: true.

{"error": "Upstream LLM provider returned 503", "done": true}

Retry with backoff for infrastructure errors; don't retry for workspace-not-found or invalid-key errors.

Latency profile

Typical timing of events:

• 0 to 100 ms. Request validated, retrieval begins.
• 100 to 250 ms. source events arrive (3 to 5 of them in quick succession).
• 250 to 350 ms. First token event arrives.
• 350 ms to 2.5 seconds. Token stream continues.
• 2.5 to 4 seconds. done event arrives, connection closes.

First-token latency under 300 ms is the key UX number. The customer sees the bot start typing almost immediately, which feels fast even if the full answer takes 3 seconds.

Request parameters

Same as POST /v1/query: query (required), top_k, session_id, conversation_id. The workspace is identified by the API key, so there's no workspace_id in the body.

Parsing the stream

Server-Sent Events format:

data: {"token":"Hello","done":false}

data: {"token":" world","done":false}

data: {"token":"","done":true,"sources":[...],"tokens_used":187,...}

Each event has:

• A data: <json> line with the event payload.
• A blank line as the terminator.

Standard SSE parsers handle this, including the browser's built-in Event Source API. If you're writing a parser by hand, split on \n\n to find event boundaries, then on \n to find lines within an event. Check event.done rather than event.type to detect stream completion.

Error recovery

For 5xx errors mid-stream:

1. Close the stream connection.
2. Wait 2^attempt * 1 seconds (exponential backoff).
3. Retry the same request, optionally with conversation_id to resume context.
4. Max 5 attempts.

For client-side errors (network drop, browser tab backgrounded): don't auto-retry; show the user the partial response and a "regenerate" button.

Cancellation

To cancel a stream mid-flight, close the connection on the client. AskVault detects the close and stops generation within 200 ms, freeing the compute budget.

Cancelled streams count as 1 message against your quota (same as completed streams). We don't refund quota for cancellations.

Limits

• Plan availability. Same as synchronous: Starter through Enterprise. The REST API (streaming and non-streaming) requires a Starter plan or above; Free doesn't include API access.
• Rate limits. Per the rate limits page. Streaming queries count the same as synchronous.
• Concurrent streams per key. Not separately capped — governed by the same per-minute/per-day request limits as any other call. See rate limits to raise a key's ceiling.

Common pitfalls

Connection hangs after the first event. Output buffering in your HTTP Client. Disable it (curl -N, requests.iter_lines(decode_unicode=False), etc.).

Some tokens missing. SSE parsers occasionally drop events when chunks span TCP Packet boundaries. Use a battle-tested SSE library, not a hand-rolled parser.

Stream feels slow on first query, fast on subsequent. Cold-start latency. Pre-warm with a noop query at app start.

Done event never arrives. Connection dropped. Treat the partial response as final and show a "regenerate" option.

FAQ

Does streaming cost more than synchronous?

No. Same per-query cost.

Can I use streaming in a browser?

Yes, but your API key would be exposed. Proxy through your backend or use the widget channel which authenticates with a public workspace token.

How do I show "typing" indicators in my UI?

Show a typing animation as soon as you send the request. Hide it when the first token event arrives and start appending tokens to the bubble.

Can I get the source citations before tokens arrive?

No. Sources arrive in the final done event after all tokens. If you need to render citations progressively, parse the answer text for inline markers and resolve them against the sources in the done event.

Does cancellation save quota?

No. Once the request is accepted, it counts against your quota regardless of completion. Cancellation only saves the compute time on our side.

Related guides

• POST /v1/query reference
• API authentication
• Rate limits per plan
• Error codes
• How to embed an AI chatbot on a React website

Start building with the AskVault API

Generate an API key in the dashboard, then send your first request.

Get an API key → Read the API docs

Was this page helpful? Yes No