​
Key Concepts

  1. API Key: Our endpoint require an API key which you can easily obtain by creating an account at Create an account. The API key makes it easier for you to track your usage and control who has access to Simli.

  2. Faces: All our available Avatar faces can be accessed through: Available Faces. We are continously adding new ones.

  3. simli-client: Stream PCM16 audio bytes to our SDK to start recieving media.

​
How it works?

Our API comprimises of two main components:

  • /startAudioToVideoSession API to initialize a session.
  • /StartWebRTCSession WebRTC API to start streaming PCM16 audio bytes to video.

If you’re a WebRTC enthusiast, we encourage you to learn more about Simli WebRTC

In this guide we will be using our simli-client SDK to avoid setting up all the WebRTC complexities.

​
Start Building

1

API Key

In order to use our API you need to get your apiKey first. Create an account, log in and in your profile you can view your API Key.

2

Import simli-client

Import the simli-client package to your project.

npm install simli-client

Check on GitHub: simli-client

3

Initialize and start streaming

  1. Create a React ref and pass it to the video and audio
import React, { useRef } from 'react';

function YourComponent() {
const videoRef = useRef(null);
const audioRef = useRef(null);

return (

<div>
  <video ref={videoRef} autoPlay playsInline></video>
  <audio ref={audioRef} autoPlay></audio>
</div>
); }
  1. Initialize SimliClient and pass in the video and audio ref
import { SimliClient } from 'simli-client';

const simliClient = new SimliClient();

const InitializeSimliClient = () => {

  const SimliConfig = {
    apiKey: SIMLI-API-KEY,
    faceID: FACE-ID,
    handleSilence: true,
    maxSessionLength: 3600, // in seconds
    maxIdleTime: 600, // in seconds
    videoRef: videoRef,
    audioRef: audioRef,
  };

  simliClient.Initialize(SimliConfig);
};
  1. Call start function to Setup WebRTC connection
simliClient.start();
  1. Stream audio using sendAudioData()

AudioData should be of type PCM16 and sample rate 16KHz

// Example: sending empty audio data
const emptyAudioData = new Uint8Array(6000).fill(0);
simliClient.sendAudioData(emptyAudioData);
  1. Finally

If done successfully, you will recieve media to be rendered on your frontend.

import React, { useRef } from "react";
import { SimliClient } from "simli-client";

const simliClient = new SimliClient();

function YourComponent() {
  const videoRef = useRef(null);
  const audioRef = useRef(null);

  // 1: Initialize SimliClient
  const InitializeSimliClient = () => {
    const SimliConfig = {
      apiKey: "SIMLI-API-KEY",
      faceID: "FACE-ID",
      handleSilence: true,
      maxSessionLength: 3600, // in seconds
      maxIdleTime: 600, // in seconds
      videoRef: videoRef,
      audioRef: audioRef,
    };
    simliClient.Initialize(SimliConfig);
  };

  // 2: Start WebRTC connection
  const startSimliClient = () => {
    simliClient.start();
  };

  // 3: Call InitializeSimliClient and startSimliClient
  useEffect(() => {
    InitializeSimliClient();
    startSimliClient();
  }, []);

  // 4: Send audio data
  const sendAudioData = (audioData) => {
    const emptyAudioData = new Uint8Array(6000).fill(0);
    simliClient.sendAudioData(emptyAudioData);
  };

  return (
    <div>
      <video ref={videoRef} autoPlay playsInline></video>
      <audio ref={audioRef} autoPlay></audio>
    </div>
  );
}
4

Working example

Clone our starter repo to get started with a working example:

create-simli-app

A simple demo demonstrating use of simli-client with OpenAI for LLM and ElevenLabs for speech

Suggest edits