​
Key Concepts

  1. API Key: All endpoints require an API key which you can easily obtain by creating an account at Create an account. The API key makes it easier for you to track your usage and control who has access to Simli.

  2. Faces: All our available Avatar faces can be accessed through: Available Faces. We are continously adding new ones.

  3. WebSocket: Stream PCM16 audio bytes to our LipsyncStream WebSocket to recieve video and audio bytes to be decoded accoring to the WebSocket scheme.

  4. Simli React SDK: Get started by using our Simli React SDK to handle WebSocket communication, decode recieved bytes and sync audio & video playback.

​
Start Building

1

API Key

In order to use our API you need to get your apiKey first. Create an account, log in and in your profile you can view your API Key.

2

Install Simli-React-SDK

npm install simli-react-sdk

3

Start Session to retrive session_token

Function to start session

export const StartAudioToVideoSession = async (faceId: string, isJPG: Boolean, syncAudio: Boolean) => {
    const metadata = {
      faceId: faceId,
      isJPG: isJPG,
      apiKey: process.env.NEXT_PUBLIC_SIMLI_API_KEY,
      syncAudio: syncAudio,
    };
  
    const response = await fetch(
      'https://api.simli.ai/startAudioToVideoSession',
      {
        method: 'POST',
        body: JSON.stringify(metadata),
        headers: {
          'Content-Type': 'application/json',
        },
      }
    );
  
    return response.json();
  };

You can check available faceIDs here: Available Faces

4

Setup SimliFaceStream componenet and Stream Audio

  1. Intialize a React ref
const simliFaceStreamRef = useRef(null);
  1. Add ref to Simli componenet
<SimliFaceStream ref={simliFaceStreamRef} start={true} sessionToken={sessionToken} minimumChunkSize={8} />
  1. Stream audio using sendAudioDataToLipsync()

AudioData should be of type PCM16 and sample rate 16KHz

let audioData = new Uint8Array(10680);

if (simliFaceStreamRef.current) {
    simliFaceStreamRef.current.sendAudioDataToLipsync(audioData);
}

If done successfully, you will recieve video and audio bytes to be rendered on your frontend.

5

/textToVideoStreamSimli React SDK