Skip to main content
This page builds a browser client for Simli’s raw streaming protocol, the one the JavaScript and Python SDKs speak for you. It takes the same steps as Simli’s own reference clients. Use it when you need your own WebRTC code, or Simli on a platform without an SDK.
If you don’t need your own client, the JavaScript SDK does all of this for you.

What you need

  • A Simli API key, from your Simli dashboard
  • A server route that creates session tokens, so your API key never reaches the browser
  • Your own WebRTC code: the browser’s RTCPeerConnection for a peer-to-peer session, or livekit-client (npm install livekit-client) for a LiveKit session
New to WebRTC or raw audio? The WebRTC and Audio formats guides cover the basics.

How a session runs

Each session has one WebSocket to Simli. It carries the connection setup, your audio and your controls, while the avatar’s video and voice come back over WebRTC. There are two transports, and they differ only in how that media connects:
  • Peer to peer (/compose/webrtc/p2p): you create an RTCPeerConnection and send Simli its offer, and Simli answers it.
  • LiveKit (/compose/webrtc/livekit): Simli puts the avatar in a LiveKit room and sends you a token to join it.
A session goes like this:
  1. Your server creates a session token with POST /compose/token. For peer to peer, it also gets ICE servers from GET /compose/ice.
  2. The browser opens the WebSocket with the token.
  3. Simli sends {"destination", "session_id"}.
  4. Peer to peer: you send your offer as the first message, and Simli sends back {"type": "answer", "sdp"}. LiveKit: Simli sends {"livekit_url", "livekit_token"}, and you join that room.
  5. Simli sends {"video_metadata": {"fps", "width", "height"}}, then START.
  6. You send audio and controls. Simli answers each message with ACK, and sends SPEAK and SILENT as the avatar starts and stops talking.
  7. When the session ends, Simli sends ENDFRAME:<frame>, CLOSING REASON: <reason> and STOP, and closes the WebSocket.
The browser code below all goes in one module, simli-raw.js.

1. Create the session token on your server

Creating a token takes your API key, so do it on your server and send only the token, and the ICE servers, to the browser. This route is for Next.js (App Router). On other backends, make the same two requests with your key in the x-simli-api-key header.
app/api/simli-session/route.ts
Send the plain face ID, with no model field. A token is valid for 5 minutes, so create a new one for each session, right before you connect. On an error, such as an invalid face ID or field, Simli answers with an error status and its reason in the body, and the route passes both on. warnings lists anything Simli accepted but changed, such as a background that isn’t yours.
Anyone who can reach your token route can start a session on your account. Put it behind your own sign-in or rate limiting before you go to production.

Trinity-2 options

With a Trinity-2 face, the token request can also set the video’s size, where the avatar sits, the starting camera and an uploaded background or foreground. All of them are optional:
POST /compose/token

2. Open the WebSocket

simli-raw.js

3. Connect the media

simli-raw.js
Send the offer within 20 seconds of opening the WebSocket. Otherwise Simli ends the session with STOP REASON: failed to receive RTC Offer.

4. Handle Simli’s messages

Everything Simli sends on the WebSocket is text. The JSON messages set up the connection, and the rest are plain strings.
simli-raw.js
The closing reasons you’ll see most often:

5. Send audio

Audio goes over the WebSocket as binary messages of raw PCM16: signed 16-bit little-endian samples, 16 kHz, mono, with no file header. Start sending it only after START.
simli-raw.js
  • Queueing: audio plays after whatever you sent before it. You can send it faster than real time, and Simli buffers it.
  • Chunks: chunks of about 6000 bytes work well. Split on whole samples, so every chunk is an even number of bytes.
  • Gaps: with handleSilence: true (the default), Simli fills the gaps between your audio with silence, so the avatar keeps moving. With false, the video waits for your audio. See Choosing handleSilence.
  • Sources: to turn a file into PCM16, use ffmpeg -i speech.mp3 -f s16le -acodec pcm_s16le -ar 16000 -ac 1 speech.pcm. For speech from a TTS API, ask it for 16 kHz PCM. See Audio formats.

6. Move the camera and swap the background

With a Trinity-2 face, you can move the camera and change the background or foreground while the session runs. Other faces answer these messages with a notice and ignore them.
simli-raw.js
BG_ and FG_ take only an id from POST /assets/backgrounds or /assets/foregrounds, never an image. For the camera fields and their ranges, see Camera. For how a new background loads, see Backgrounds.

7. End the session

simli-raw.js
After DONE, Simli plays the audio you’ve sent, ignores any you send after it, and then ends the session with CLOSING REASON: User Sent Final Chunk and STOP. To end the session at once instead, call close(session) from step 4: closing the WebSocket ends it, and so does closing the peer connection or leaving the room.

Putting it together

index.html
main.js
Serve these through your bundler or framework, since simli-raw.js imports livekit-client.