> ## Documentation Index > Fetch the complete documentation index at: https://docs.simli.com/llms.txt > Use this file to discover all available pages before exploring further. # JavaScript SDK [SimliClient](https://github.com/simliai/simli-client) is a powerful tool for integrating real-time audio and video streaming capabilities into your web applications using WebRTC. This guide will walk you through the process of setting up and using SimliClient in your project. ## Getting Started ### Prerequisites Before you begin, make sure you have: 1. A Simli account with an API key 2. Node.js and npm installed in your development environment ### Installation Install the `simli-client` package in your project: ```bash theme={null} npm install simli-client ``` ## Usage ### Step 1: Get the Video and Audio element references If you're using React, create video and audio elements with refs: ```tsx theme={null} function YourComponent() { const videoRef = useRef(null); const audioRef = useRef(null); return (
); } ``` However, you can also just use direct dom refs we can also do that ```typescript theme={null} // assuming we have only one video element and one audio element const videoElement: HTMLVideoElement = document.getElementById("video"); const audioElement: HTMLAudioElement = document.getElementById("audio"); ``` ### Step 2: Create a session token You can create the session\_token from the client side, or put it behind a backend (or use server actions for Next.JS or equivalent in other frameworks). The recommended method for production deployments is to put it behind a backend as you don't want your api key to reside on the client If you're using listenToMediastreamTrack() set `handleSilence: false` in the SimliConfig to avoid audio artifacts ```typescript theme={null} import {generateSimliSessionToken, SimliSessionRequest} from "simli-client"; async function getSessionToken(): Promise { const SimliConfig: SimliSessionRequest = { faceId: "YOUR_FACE_ID", maxSessionLength: 600, // in seconds maxIdleTime: 180, // in seconds }; const session_token = (await generateSimliSessionToken({apiKey: "YOUR_SIMLI_API_KEY", config: SimliConfig})).session_token; return session_token; } ``` ### (Optional) Step 2.5: Generate Ice Server Config (For running Simli in peer to peer mode) This provides a slightly lower latency connection to Simli however it might have some issues in places with stronger firewall restrictions (in such case we recommend using livekit mode which doesn't need this step at all) ```typescript theme={null} async function getIceServers(): Promise { return await getIceServers("YOUR_SIMLI_API_KEY"); } ``` ### Step 3: Initialize SimliClient Create an instance of SimliClient and initialize it with your configuration: For P2P Mode: ```typescript theme={null} import {SimliClient, LogLevel} from "simli-client"; async function InitializeSimliClient(): SimliClient { simliClient = new SimliClient( await getSessionToken(), videoRef.current, audioRef.current, await getIceServers(), LogLevel.DEBUG, // Optional Paramter, in production it's recommended to set it to info "p2p", ); await simliClient.start(); return simliClient; } ``` For Livekit Mode: ```typescript theme={null} import {SimliClient, LogLevel} from "simli-client"; async function InitializeSimliClient(): SimliClient { simliClient = new SimliClient( await getSessionToken(), videoRef.current, audioRef.current, null, LogLevel.DEBUG, // Optional Paramter, in production it's recommended to set it to info "livekit", ); await simliClient.start(); return simliClient; } ``` ### Step 4: Send Audio Data Once the connection is established, you can start sending audio data: ```javascript theme={null} // Example: sending audio data (should be PCM16 format, 16KHz sample rate) const audioData = new Uint8Array(6000).fill(0); // Replace with your actual audio data simliClient.sendAudioData(audioData); ``` Ensure that your audio data is in PCM16 format with a 16KHz sample rate. ## SimliClient API reference #### Types ```typescript theme={null} interface SimliSessionRequest { faceId: string; handleSilence: boolean; maxSessionLength: number; maxIdleTime: number; model?: "fasttalk" | "artalk"; } interface TokenRequestData { config: SimliSessionRequest; apiKey: string; } interface SimliSessionToken { session_token: string; } type TransportMode = "livekit" | "p2p"; type SignalingMode = "websockets"; type session_token = string; ``` For SimliClient, check [the source code](https://github.com/simliai/simli-client) #### Methods * `async generateSimliSessionToken( request: TokenRequestData, SimliURL: string = "https://api.simli.ai",)`: Generate new session token given a `TokenRequestData` object. You can also redirect that request to your own backend. * `new SimliClient(session_token: string, videoElement: HTMLVideoElement, audioElement: HTMLAudioElement, iceServers: RTCIceServer[] | null, logLevel: LogLevel = LogLevel.DEBUG, transport_mode: TransportMode = "p2p", signaling: SignalingMode = "websockets", SimliWSURL: string = "wss://api.simli.ai", audioBufferSize: number = 3000,)`: Initializes the SimliClient with the provided configuration. * `async SimliClientInstance.start()`: Connects everything to Simli Servers with automatic retry on failure * `async SimliClientInstance.stop()`: Closes the WebRTC connection and cleans up resources. * `SimliClientInstance.sendAudioData(audioData: Uint8Array)`: Sends audio data to the server. * `SimliClientInstance.listenToMediastreamTrack(stream: MediaStreamTrack)`: Listens to a MediaStreamTrack and sends audio data to the server. *Can be used as an alternative to `sendAudioData`.* * `SimliClientInstance.ClearBuffer()`: Clears the audio buffer, best used when you want the avatar to stop talking. ## Advanced Usage ### Events * `start` when the webrtc connection is successful and avatar is shown on screen * `stop` when the simli server terminates a connection normally * `error` when the webRTC connection fails to connect or crashes in the middle * `speaking` when the avatar starts speaking * `silent` when the avatar stops speaking * `ack` when server acknowledges an audio segment has been received * `connection_info(serialized_info: string)` when rtc answer or livekit session info is received not useful in most applications and is meant for primarily internal use * `video_info(serialized_info: string)` shows video FPS, height, and width * `destination(serialized_info: string)` debugging information useful when reporting issues to simli * `unknown(message: string)` when the server sent a message that couldn't be parsed. Usually future messages that have been added to later versions of the client * `startup_error(message: string)` when there was a fundamental error causing simli to be unable to connect. This is usually triggered for invalid FaceIDs or depleted minutes. Message should be the cause of this startup failure. Automatic retries are disabled on startup\_error General Usage ```typescript theme={null} simliClient.on("EVENT", callbackFunction); ``` ### Error Handling SimliClient provides console logging for various events and errors. It's recommended to implement proper error handling in your application to manage potential issues, such as network disconnections or initialization failures. ### Fork and Contribute to simli-client Fork and contribute to the simli-client repository on GitHub or clone for your own usecase. ## Troubleshooting If you encounter issues: 1. Ensure your API key is correct and active. 2. Verify that your face ID is valid and associated with your account. 3. Check that your audio data is in the correct format (PCM16, 16KHz). 4. Verify that you have the necessary permissions for accessing the user's media devices. 5. Review the console logs for any error messages or warnings. Reach out to our team for further assistance on [Discord](https://discord.gg/yQx49zNF4d).