{
"livekit_url": "<string>",
"livekit_token": "<string>"
}{}{}{
"video_metadata": {
"fps": 123,
"width": 123,
"height": 123
}
}{}{}{}{}{}{}{}{}{}{}Livekit
Livekit Backed WebRTC
Compose Session Token is sent as query parameter string
No first message is required. However, You must successfully join the livekit room within 60 seconds, or the session ends with CLIENT NOT CONNECTED TO LIVEKIT ROOM, TERMINATED SESSION
{
"livekit_url": "<string>",
"livekit_token": "<string>"
}{}{}{
"video_metadata": {
"fps": 123,
"width": 123,
"height": 123
}
}{}{}{}{}{}{}{}{}{}{}Room info and join token that are used by the client to join a livekit room that hosts the avatar session
Server replies to a camera message, sent as text. None of them end the session.
Server replies to a background or foreground message, sent as text; a message that is applied gets none. None of them end the session, and the avatar keeps what it had.
Sent as JSON text once the session is set up, just before START. fps, width and height are integers: the frame rate and size of the video Simli streams. For a Trinity-2 face, width x height is the session token's streaming_width x streaming_height, an odd side rounded up by a pixel, at 25 fps. A Trinity-1 face streams 512 x 512 at 25 fps.
Server sent signals
Sent as text when the session ends, just before CLOSING REASON: <reason> and STOP: ENDFRAME: and the number of video frames the session streamed, an integer (the video's running time times its fps).
Termination messages that specify the reason for immediate termination, including a routing error (Routing error, no EU Sessions available. EU ONLY PROCESSING enabled) when a region-pinned session lands on a machine outside its region
Binary stream of the audio that the Avatar will lipsync to. Sent as a binary stream (in JS/TS UInt8Arr)
Interrupts the avatar and plays new audio at once. Sent as binary: the ASCII prefix PLAY_IMMEDIATE (14 bytes), then audio in the same format as audio_chunk (PCM16, 16 kHz, mono), in one message.
Everything you sent before it that the avatar hasn't spoken yet is dropped, as with SKIP, and this audio starts playing from where the avatar is now. Audio chunks you send after it play after it.
Use it when the user interrupts and the reply is already in hand: one message drops the old audio and starts the new one, instead of a SKIP and then audio chunks. It can carry the whole reply, or just its start (Simli's reference client sends the first 4 seconds this way) with the rest following as ordinary audio chunks. An odd trailing byte is dropped. After DONE, it only interrupts: the audio in it isn't played.
Client Signal to make the avatar stop speaking immediately and ignore all previously sent audio
Client Signal to tell the server that the final segment of audio has been already sent and that it should close after playing it back
Moves the camera, Trinity-2 faces only. Sent as binary: the ASCII prefix CAM_, then little-endian float32 camera offsets on top of the avatar's own framing, in this order (y comes before x):
| # | Field | Meaning | Supported range |
|---|---|---|---|
| 1 | pos_y_off | Moves the avatar down, in half canvas heights | -10 to 10 |
| 2 | pos_x_off | Moves the avatar right, in half canvas widths | -10 to 10 |
| 3 | pos_z_off | Metres the camera backs away from the avatar (negative moves in) | -5 to 5 |
| 4 | zoom | Multiplies the avatar's zoom; 1 is no change | above 0, up to 4 |
| 5 | rot_x_off | Degrees the camera tilts up (pitch) | -180 to 180 |
| 6 | rot_y_off | Degrees the camera pans right (yaw) | -180 to 180 |
| 7 | rot_z_off | Degrees the camera rolls clockwise | -180 to 180 |
| 8 | orbit_x_off | Degrees the camera orbits up over the avatar's head (elevation) | -80 to 80 |
| 9 | orbit_y_off | Degrees the camera orbits round to its right (azimuth) | -180 to 180 |
Send the first 4, the first 7 or all 9 values; the ones left off stay neutral (0).
Each message replaces the whole camera, including the one the session started with (the session token's camera): the values are offsets from the avatar's own framing, not from the previous message. 0, 0, 0, 1, 0, 0, 0, 0, 0 returns to the avatar's own framing.
The camera cuts to the new setting. To glide instead, append the ASCII tag LERP and one more float32: the seconds the move takes, from wherever the camera is now (part-way through an earlier glide, if one is running). 0 or less is a cut.
A message is 20, 32 or 40 bytes, or 28, 40 or 48 with LERP. A value outside its supported range is still applied, with a WARNING: reply. Any other length, NaN or infinity is rejected and the camera stays where it is. See camera_notice.
Puts one of your uploaded backgrounds behind the avatar, Trinity-2 faces only: the ASCII prefix BG_ and the background's id, as POST /assets/backgrounds returned it, 36 characters (hyphens included, either case). Text or binary; 39 characters in all, nothing else is accepted. An image can't be sent: upload it first. The background is cropped to fill the session's canvas.
Only your own backgrounds: an id that isn't one of your account's is ignored. It replaces the one the session started with (the session token's background_id) a moment later, once loaded; audio keeps playing meanwhile, and the last one sent wins. See asset_notice.
BG_00000000-0000-0000-0000-000000000000 (the nil UUID) goes back to the avatar's own background, at once.
Lays one of your uploaded foregrounds over the avatar through its transparency, Trinity-2 faces only: the ASCII prefix FG_ and the foreground's id, as POST /assets/foregrounds returned it, 36 characters. Text or binary; 39 characters in all, nothing else is accepted, an image least of all. Otherwise as background, in place of the session token's foreground_id. FG_00000000-0000-0000-0000-000000000000 (the nil UUID) takes the foreground away.