Custom LLM guide

Introduction

Using Simli Auto (our end-to-end API) is awesome! But it can be a bit difficult to tailor it out for your needs. You may have a custom knowledge base, a RAG powered system, using your fine-tuned LLM that’s the best doctor or tutor ever known to mankind that you self-host, or just want to use a model that we don’t readily support. If that’s you, you’re in the right place. There’s a fun trick you may already know: the DeepSeek API docs tell you to use the OpenAI SDK as if it were DeepSeek’s own. You give it DeepSeek’s base URL and API key, and it works as if nothing weird is going on. That works because DeepSeek, like a lot of LLM API providers, copied OpenAI’s API design. OpenAI’s API has a lot going on, but everyone copied the 3 essential parts: the request path (/chat/completions), the request body, and the streamed response. TL;DR: if your API works with the OpenAI SDK (try it with the Python or JS SDK), pass Simli its base URL, an API key and the model name in llmConfig, with provider set to "User", and you’re golden! The base URL must use HTTPS.

{
    "faceId": "bf13113a-355a-4d97-8b90-4edccd1da7b2",
    "llmConfig": {
        "provider": "User",
        "baseURL": "https://my-llm.example.com/v1",
        "apiKey": "YOUR_LLM_API_KEY",
        "model": "my-model"
    }
}

customLLMConfig (baseURL, llmAPIKey, model) still works but is deprecated: use llmConfig.

The basic OpenAI request

Simli calls your API the way the OpenAI SDK does: POST {baseURL}/chat/completions, with the header Authorization: Bearer <your apiKey> and a body like this:

{
    "model":"my-AGI",
    "messages": [
        {"role":"system", "content":"This is a sample system message (AKA base prompt)"},
        {"role":"assistant", "content":"This is a sample AI generated message"},
        {"role":"user","content":"This is a sample user Input"}
    ],
    "stream": true
}

and the response(s)

{
    "id": "chatcmpl-0",
    "choices": [{
        "delta": {"content": "Chunk Completion Text Example"}
    }],
    "created": 1730000000,
    "model": "the model name you had in request",
    "object": "chat.completion.chunk"
}

created is a Unix timestamp and object is always the literal chat.completion.chunk. Response has the mimetype text/event-stream and above is used for all chunks. You must ensure that the text body is correctly formatted. Additionally, you must indicate that your response is done by sending a DONE frame. All responses are followed by 2 newline delimiters. Using FastAPI for example, you would have an async generator looking like this:

import json
async def outputStream():
    ResultDict = [{
        "id":0,
        "choices" :[ {
            "delta": {"content":"Chunk Completion Text Example"}
        }],
        "created": "Unix Time Stamp",
        "model":"the model name you had in request",
        "object": "chat.completion.chunk" # Just a literal
    }]
    for chunk in ResultDict:
        yield f"data: {json.dumps(chunk)}\n\n"
    yield "data: [DONE]\n\n"

Here’s an example server sending out a mock response with FastAPI, which can be simply run using python app.py and passing the hosting URL to Simli. (more examples in other languages coming soon)

import time
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import json

app = FastAPI()


async def streamOutput():
    data = {
        "id": "",
        "choices": [
            {
                "delta": {"content": "This is a test. "},
                "index": 0,
            }
        ],
        "model": "gpt-4o-mini-2024-07-18",
        "created": 0,
        "object": "chat.completion.chunk",
    }
    for i in range(5):
        data["id"] = f"id-{i}"
        data["choices"][0]["index"] = i
        data["created"] = int(time.time())
        out = f"data: {json.dumps(data)}\n\n"
        yield out
    yield "data: [DONE]\n\n"


@app.post("/chat/completions")
async def chat_completions(_: Request):
    return StreamingResponse(streamOutput(), media_type="text/event-stream")


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=11000)

and another example, wrapping the OpenAI SDK on the server

import os
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI
from openai.types.chat import ChatCompletionChunk
from typing import AsyncGenerator
from dotenv import load_dotenv

app = FastAPI()

load_dotenv(override=True)
OPEN_AI_API_KEY = os.getenv("OPENAI_API_KEY")
async_client = AsyncOpenAI(api_key=OPEN_AI_API_KEY)


async def streamOutput(body: dict):
    output: AsyncGenerator[
        ChatCompletionChunk
    ] = await async_client.chat.completions.create(**body)

    async for item in output:
        out = f"data: {item.model_dump_json(exclude_unset=True)}\n\n"
        yield out
    yield "data: [DONE]\n\n"


@app.post("/chat/completions")
async def chat_completions(request: Request):
    # Read the body before streaming: reading it inside the stream can hang
    return StreamingResponse(
        streamOutput(await request.json()),
        media_type="text/event-stream",
    )


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=11000)

Endpoints

WebSockets

OpenAPI Specs (Agent Friendly Format)

Simli Auto Guides (deprecated)

Custom LLM guide

Introduction

The basic OpenAI request

​Introduction

​The basic OpenAI request

Introduction

The basic OpenAI request