API reference

This is the realtime streaming interface: audio in over a WebSocket, final results back. When results are emitted is selected by mode at connect time; every mode shares this one protocol. Whole-recording transcription over HTTP (with PII redaction and other whole-audio features) is a separate surface and is not part of this endpoint.

The service is free during the beta. Sessions are metered in audio-seconds and the total is echoed at session close for visibility, but nothing is billed.

The service and its features are experimental and under active development. Expect occasional bugs and rough edges, and expect new capabilities to appear regularly. The feedback form goes straight to the team building it.

The public endpoint is:

wss://api.labs.bandwidth.com/audio/v1/listen

Authenticate the WebSocket upgrade with:

X-BW-LABS-API-KEY: bwa_key_...

The server validates the key before the stream starts. Browsers cannot set WebSocket headers; pass ?api_key=<key> as a query parameter instead. The header form is preferred anywhere headers are possible.

A minimal call

Any standard WebSocket client works, or use the official Python and TypeScript SDKs, which wrap this whole protocol in a few lines. Connect over WebSocket, send binary frames of raw audio, send CloseStream, and read JSON text messages until SessionClosed:

const ws = new WebSocket(
  "wss://api.labs.bandwidth.com/audio/v1/listen" +
    "?encoding=linear16&sample_rate=16000&api_key=bwa_key_...",
);

ws.onopen = () => {
  // 160 ms of 16 kHz mono linear16 = 5,120 bytes per frame
  for (const frame of audioFrames) ws.send(frame);
  ws.send(JSON.stringify({ type: "CloseStream" }));
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === "Segment" || msg.type === "Transcript") transcript += msg.text;
  if (msg.type === "SessionClosed") ws.close();
};

When streaming live audio rather than a file, pace frames at real time and send KeepAlive during long silences.

Connection parameters

Parameters are query parameters on the WebSocket URL. Any parameter outside this table is rejected with an in-band invalid_params error naming the parameter.

Modes

mode selects when finalized results are delivered. instant, the default, emits finalized Segment events as text is decoded. Finalize flushes buffered audio, and the resulting and subsequent finalized output is delivered as Segment events.

demand buffers finalized results server-side and does not send Segment events. Each Finalize is answered with exactly one Transcript event per channel covering audio finalized since the previous delivery. The event is always emitted, including when its text is empty. CloseStream delivers the remaining finalized audio, then sends SessionClosed.

A voice-agent turn loop is:

1. Stream one user turn of audio.
2. Send {"type":"Finalize"}.
3. Read one Transcript event for each channel and use its text and words.
4. Repeat for the next turn. Send {"type":"CloseStream"} after the final turn and read the remainder before SessionClosed.

Any other value is rejected with invalid_params.

PII redaction

PII redaction applies when redact_pii=true on a demand-mode listen session or on Transcribe. Detected spans are replaced in the returned text and in the corresponding returned words entries. The unredacted text is not returned in text or words; it is returned only in redacted_entities[].text when the caller explicitly sets redact_pii_return=true.

redact_pii_sub=entity_name replaces each detected span with a bracketed kind token, such as [CREDIT_CARD] or [SSN]. redact_pii_sub=hash replaces each span with a deterministic token in the form hash:v1:<16 hex chars>. The same entity value maps to the same token within an account, allowing counts and joins. Original spans are returned only with redact_pii_return=true.

redact_pii_return=true requires redact_pii=true and hash substitution. When redact_pii_sub is unset, it defaults to hash for this option. Combining redact_pii_return=true with redact_pii_sub=entity_name is rejected with invalid_params.

When enabled, redacted_entities is an array of detected spans. Each element has token, the exact substitution string appearing in the returned text; kind, the detected category such as credit_card; text, the original span; and start and end in seconds. start and end are null when the entity has no word overlap. The array is empty when redaction found nothing, and the field is absent when redact_pii_return is not enabled.

The redaction summary object reports applied and entities_redacted. Redaction currently looks for payment card numbers, Social Security numbers, phone numbers, email addresses, dates of birth, account and ticket numbers, and US ZIP codes. These appear in redacted_entities as the kind values credit_card, ssn, phone, email, dob, account_ticket, and zip. Detection coverage grows during the beta.

Keywords

keywords boosts recognition of supplied phrases. Repeat keywords for each phrase. A request accepts at most 100 phrases, every phrase must be non-empty, and the combined length must not exceed 4096 bytes. Keywords are available on both listen and Transcribe.

Audio frames

Client-to-server audio is a binary WebSocket message containing raw, headerless audio. Each frame must be between 20 ms and 1000 ms inclusive and must contain complete interleaved samples. A final 20–160 ms tail may be sent as-is; a tail shorter than 20 ms is rejected.

Recommended frame sizes are 160 ms multiples.

Bytes per frame for linear16 are:

sample_rate × duration_seconds × 2 bytes × channels

mulaw and alaw use one byte per sample: raw G.711 μ-law and A-law payloads, not WAV containers.

g722 carries 16 kHz audio at one byte per two output samples (64 kbps). It requires sample_rate=16000 and channels=1; the frame-duration rule applies to the decoded audio, so a 160 ms frame is 1,280 bytes.

opus is framed differently: each binary WebSocket message carries exactly ONE raw Opus packet (2.5–120 ms of audio; 20 ms packets are typical). The 20–1000 ms frame rule does not apply. Send the packets exactly as produced by the encoder, with no Ogg or WebM container. Requires sample_rate=16000 and channels=1.

Client control messages

Control messages are UTF-8 JSON text messages. They contain only type.

{"type":"KeepAlive"}

Keeps an otherwise quiet session alive. It contains no audio and adds no audio-seconds.

{"type":"Finalize"}

In instant mode, flushes buffered audio immediately while keeping the session open. The flush and subsequent finalized output arrive as Segment events. In demand mode, the server answers with exactly one Transcript event per channel covering audio finalized since the previous delivery, even when its text is empty. There is no separate finalize ack.

{"type":"CloseStream"}

Gracefully ends the session. The server flushes remaining audio, emits mode-specific messages, then sends SessionClosed as the terminal message. In instant mode the messages are Segment events. In demand mode the remaining audio is delivered as one Transcript event per channel.

Send KeepAlive during quiet periods; a 25 s cadence keeps a session safely inside the 60 s idle deadline.

Server messages

SessionOpened

First message after a successful upgrade:

{
  "type":"SessionOpened",
  "request_id":"6f58c1c6-7e0c-4bb8-9d72-3fb3d4c5c1aa",
  "model_info":{"name":"bw-listen-en","version":"current"},
  "channels":1,
  "sample_rate":16000,
  "encoding":"linear16"
}

Segment

Segment events are instant-mode only. Each event is final: output is append-only and never revised. Segments carry the raw decoded text delta the moment the model produces it (typically every 160 ms of audio, often subword pieces). text preserves its leading space when the delta starts a new word, so the full transcript is reconstructed by plain concatenation of text fields in arrival order. Do not insert separators. words contains timestamps in seconds:

{"type":"Segment","channel":0,"start":0.00,"end":0.20,"text":"i need","words":[
  {"word":"i","start":0.00,"end":0.12},{"word":"need","start":0.16,"end":0.20}]}
{"type":"Segment","channel":0,"start":0.24,"end":0.44,"text":" a dr","words":[
  {"word":"a","start":0.24,"end":0.32},{"word":"dr","start":0.36,"end":0.44}]}
{"type":"Segment","channel":0,"start":0.48,"end":0.72,"text":"y van","words":[
  {"word":"y","start":0.48,"end":0.56},{"word":"van","start":0.60,"end":0.72}]}

Concatenated: "i need a dry van".

For live word-by-word display, apply the same leading-space rule per word instead of per segment: a chunk of text that starts with a space begins a new display word; a chunk without one extends the previous display word in place. Rendering this way shows dr the instant it is decoded and grows it into dry when the next piece arrives, with no added latency and no separator bookkeeping. A merged display word's timestamps span the first piece's start to the last piece's end.

There is intentionally no interim_results, is_final, speech_final, confidence, or alternatives field in this schema. Future capabilities arrive as new event types behind opt-in connection parameters; unknown event types should be ignored.

Transcript

Transcript events are demand-mode only. Each Finalize produces exactly one event per channel, covering audio finalized since the previous delivery. The event is emitted even when text is empty. CloseStream delivers the remaining audio in the same form before SessionClosed.

{
  "type":"Transcript",
  "channel":0,
  "text":"Please charge hash:v1:9f2c41d08ab37e15 today.",
  "words":[
    {"word":"Please","start":0.00,"end":0.42},
    {"word":"charge","start":0.48,"end":0.82},
    {"word":"hash:v1:9f2c41d08ab37e15","start":0.88,"end":1.84},
    {"word":"today.","start":1.90,"end":2.26}
  ],
  "redacted_entities":[
    {
      "token":"hash:v1:9f2c41d08ab37e15",
      "kind":"credit_card",
      "text":"4111 1111 1111 1111",
      "start":0.88,
      "end":1.84
    }
  ],
  "redaction":{
    "applied":true,
    "entities_redacted":1
  }
}

text and words contain the finalized delivery window. When redaction is enabled, both contain the replacements described in PII redaction, never the unredacted spans. redacted_entities is included only when redact_pii_return=true and follows the field contract described above.

Error

{"type":"Error","code":"invalid_frame","message":"audio frame duration must be between 20ms and 1000ms"}

Codes sent over the WebSocket:

Failures before the WebSocket upgrade (bad or missing key, per-key rate or concurrency limits, capacity limits) are plain HTTP status responses (401, 403, 429, 503), not WebSocket Error frames. Honor Retry-After when supplied; do not replay audio automatically.

SessionClosed

Last protocol message on a graceful close:

{
  "type":"SessionClosed",
  "request_id":"6f58c1c6-7e0c-4bb8-9d72-3fb3d4c5c1aa",
  "audio_duration_seconds":184.32,
  "session_duration_seconds":190.11
}

delivery_failed appears with the value true when a final delivery could not be sent. The field is omitted when delivery succeeded, so treat absence as success.

audio_duration_seconds is measured by the server from decoded audio and is the usage echo. It is not wall-clock time and is not client-supplied. During the beta this figure is informational only; nothing is billed.

Channels and multichannel

PCM is interleaved and headerless. channels=2 describes the input layout; multichannel selects the transcription and metering policy:

multichannel=true with channels=1 is invalid. A split stereo session is one WebSocket session but meters both channels.

Reconnect semantics

There is no resume token, sequence number, or replay protocol in v1. After an unexpected close, connecting again creates a new request_id, a new server session, and a new usage boundary; callers must decide what audio, if any, to send again.

Transcribe (HTTP)

The SDKs wrap this endpoint as a single transcribe call.

Submit a complete recording with:

POST https://api.labs.bandwidth.com/audio/v1/transcribe

Authenticate with the same method as listen. Use the X-BW-LABS-API-KEY header or the api_key query parameter; the header is preferred.

X-BW-LABS-API-KEY: bwa_key_...

The request body must be one of these types:

Optional query parameters are model, the PII redaction family, including redact_pii_return, and repeated keywords. The server decodes up to five minutes of audio. A request with more than five minutes of decoded audio returns 413.

A successful response has this shape. words can be empty when word timing is unavailable.

{
  "request_id":"6f58c1c6-7e0c-4bb8-9d72-3fb3d4c5c1aa",
  "text":"Please send the invoice today.",
  "words":[
    {"word":"Please","start":0.00,"end":0.42},
    {"word":"send","start":0.48,"end":0.76},
    {"word":"the","start":0.82,"end":0.94},
    {"word":"invoice","start":1.00,"end":1.58},
    {"word":"today.","start":1.64,"end":2.18}
  ],
  "segments":[
    {"start":0.00,"end":2.18,"text":"Please send the invoice today."}
  ],
  "audio_duration_seconds":2.18,
  "model_info":{"name":"bw-listen-en","version":"current"}
}

With redact_pii=true, a successful Transcribe response returns replacements in text and the corresponding words entries and includes the redaction summary object. With redact_pii_return=true, it also includes redacted_entities as described in PII redaction. The field is an empty array when no entity was found and is absent when redact_pii_return is not enabled.

Errors from this endpoint are: