What you need
- A Simli API key, from your Simli dashboard
- A server route that creates session tokens, so your API key never reaches the browser
- Your own WebRTC code: the browser’s
RTCPeerConnectionfor a peer-to-peer session, or livekit-client (npm install livekit-client) for a LiveKit session
How a session runs
Each session has one WebSocket to Simli. It carries the connection setup, your audio and your controls, while the avatar’s video and voice come back over WebRTC. There are two transports, and they differ only in how that media connects:- Peer to peer (
/compose/webrtc/p2p): you create anRTCPeerConnectionand send Simli its offer, and Simli answers it. - LiveKit (
/compose/webrtc/livekit): Simli puts the avatar in a LiveKit room and sends you a token to join it.
- Your server creates a session token with
POST /compose/token. For peer to peer, it also gets ICE servers fromGET /compose/ice. - The browser opens the WebSocket with the token.
- Simli sends
{"destination", "session_id"}. - Peer to peer: you send your offer as the first message, and Simli sends back
{"type": "answer", "sdp"}. LiveKit: Simli sends{"livekit_url", "livekit_token"}, and you join that room. - Simli sends
{"video_metadata": {"fps", "width", "height"}}, thenSTART. - You send audio and controls. Simli answers each message with
ACK, and sendsSPEAKandSILENTas the avatar starts and stops talking. - When the session ends, Simli sends
ENDFRAME:<frame>,CLOSING REASON: <reason>andSTOP, and closes the WebSocket.
simli-raw.js.
1. Create the session token on your server
Creating a token takes your API key, so do it on your server and send only the token, and the ICE servers, to the browser. This route is for Next.js (App Router). On other backends, make the same two requests with your key in thex-simli-api-key header.
app/api/simli-session/route.ts
model field. A token is valid for 5 minutes, so create a new one for each session, right before you connect. On an error, such as an invalid face ID or field, Simli answers with an error status and its reason in the body, and the route passes both on. warnings lists anything Simli accepted but changed, such as a background that isn’t yours.
Trinity-2 options
With a Trinity-2 face, the token request can also set the video’s size, where the avatar sits, the starting camera and an uploaded background or foreground. All of them are optional:POST /compose/token
2. Open the WebSocket
simli-raw.js
3. Connect the media
- Peer to peer
- LiveKit
simli-raw.js
STOP REASON: failed to receive RTC Offer.4. Handle Simli’s messages
Everything Simli sends on the WebSocket is text. The JSON messages set up the connection, and the rest are plain strings.simli-raw.js
The closing reasons you’ll see most often:
5. Send audio
Audio goes over the WebSocket as binary messages of raw PCM16: signed 16-bit little-endian samples, 16 kHz, mono, with no file header. Start sending it only afterSTART.
simli-raw.js
- Queueing: audio plays after whatever you sent before it. You can send it faster than real time, and Simli buffers it.
- Chunks: chunks of about 6000 bytes work well. Split on whole samples, so every chunk is an even number of bytes.
- Gaps: with
handleSilence: true(the default), Simli fills the gaps between your audio with silence, so the avatar keeps moving. Withfalse, the video waits for your audio. See ChoosinghandleSilence. - Sources: to turn a file into PCM16, use
ffmpeg -i speech.mp3 -f s16le -acodec pcm_s16le -ar 16000 -ac 1 speech.pcm. For speech from a TTS API, ask it for 16 kHz PCM. See Audio formats.
6. Move the camera and swap the background
With a Trinity-2 face, you can move the camera and change the background or foreground while the session runs. Other faces answer these messages with a notice and ignore them.simli-raw.js
BG_ and FG_ take only an id from POST /assets/backgrounds or /assets/foregrounds, never an image. For the camera fields and their ranges, see Camera. For how a new background loads, see Backgrounds.
7. End the session
simli-raw.js
DONE, Simli plays the audio you’ve sent, ignores any you send after it, and then ends the session with CLOSING REASON: User Sent Final Chunk and STOP. To end the session at once instead, call close(session) from step 4: closing the WebSocket ends it, and so does closing the peer connection or leaving the room.
Putting it together
index.html
main.js
simli-raw.js imports livekit-client.