LiveKit Integration

Overview

LiveKit Agents provides a realtime framework for voice, video, and multimodal applications. Our open-source integration lets you add a LemonSlice avatar to your existing agent stack.

Jump to the starter projects section for ready-to-use repos. For production UI patterns (avatar readiness, error handling, timeouts), see Production checklist.

Prerequisites

A LemonSlice account and API key associated with an active subscription. Create a key at lemonslice.com/developers.
A reference image for your avatar. See our guide for recommendations on how to design your avatar image to achieve the best quality possible.
LiveKit Agents app
- Your own existing application, or
- The ready-to-use LiveKit playground starter project, or
- Follow the LiveKit VoiceAI quickstart for Python or Node.js to create one.

How to use

Install the plugin

Within your LiveKit Agents app, install the plugin:

# When using PIP
pip install "livekit-agents[lemonslice]"

# When using UV
uv add "livekit-agents[lemonslice]"

pnpm add @livekit/agents-plugin-lemonslice

Authenticate

In your LiveKit Agents app, set LEMONSLICE_API_KEY in your .env file.

Add AvatarSession to AgentSession

In your LiveKit Agents app, create a lemonslice.AvatarSession alongside your AgentSession:

from livekit import agents
from livekit.agents import AgentSession, RoomOutputOptions
from livekit.plugins import lemonslice

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        # Add STT, LLM, TTS, and other components here
    )

    avatar = lemonslice.AvatarSession(
        # Publicly accessible image URL for the avatar
        agent_image_url="...",
        # Prompt to guide the avatar's demeanor while speaking
        agent_prompt="a person talking",
    )

    # Start the avatar and wait for it to join
    session_id = await avatar.start(session, room=ctx.room)

    # Start your agent session with the user
    await session.start(
        # ... room, agent, room_options, etc....
    )

import { voice } from '@livekit/agents';
import * as lemonslice from '@livekit/agents-plugin-lemonslice';

const session = new voice.AgentSession({
    // Add STT, LLM, TTS, and other components here
});

const avatar = new lemonslice.AvatarSession({
    // Publicly accessible image URL for the avatar
    agentImageUrl: "...",
    // Prompt to guide the avatar's demeanor while speaking
    agentPrompt: "a person talking",
});

// Start the avatar and wait for it to join
const sessionId = await avatar.start(session, room);

// Start your agent session with the user
await session.start(
    // ... room, agent, room_options, etc.
);

Python
Node.js

Parameter	Description
`agent_image_url`	(Optional) A publicly accessible URL to the avatar image. Exactly one of `agent_image_url` / `agent_id` / `agent_image` is required.
`agent_id`	(Optional) The ID of a LemonSlice avatar. Exactly one of `agent_image_url` / `agent_id` / `agent_image` is required. If provided, avatar-configured settings are used as defaults for `agent_prompt`, `agent_idle_prompt`, and `idle_timeout`.
`agent_image`	(Optional) A local PIL image to upload as the avatar. Exactly one of `agent_image_url` / `agent_id` / `agent_image` is required.
`agent_prompt`	(Optional) A high-level system prompt that subtly influences the avatar’s movements, expressions, and emotional demeanor while speaking. This prompt is best used to suggest general affect or demeanor (for example, “feel excited” or “look sad”) rather than precise or deterministic actions. Defaults to `"a person talking"`.
`agent_idle_prompt`	(Optional) A high-level system prompt that influences the avatar’s movements, expressions, and emotional demeanor during the idle state. Defaults to `"a serious person"`.
`idle_timeout`	(Optional) Idle timeout in seconds. If a negative number is provided, the session will have no idle timeout. Defaults to 60.
`response_done_timeout`	(Optional) Time in seconds to wait without receiving new audio bytes before marking the response as complete. Some TTS models do not send an end response event, or do not send it in a timely manner, after transmitting all audio bytes. This parameter enables detection of response completion when such events are missing or delayed.
`model`	(Optional) Model variant (`"lite"`, `"flash"`, or `"pro"`). Leave unset to use the normal flagship model.
`aspect_ratio`	(Optional) Output aspect ratio (`"2x3"`, `"9x16"`, or `"1x1"`).
`simulcast`	(Optional) Enable WebRTC simulcast, which publishes multiple resolutions of the avatar video so subscribers receive the best quality their bandwidth allows.

Parameter	Description
`agentImageUrl`	(Optional) A publicly accessible URL to the avatar image. Exactly one of `agentImageUrl` / `agentId` / `agentImage` is required.
`agentId`	(Optional) The ID of a LemonSlice avatar. Exactly one of `agentImageUrl` / `agentId` / `agentImage` is required. If provided, avatar-configured settings are used as defaults for `agentPrompt`, `agentIdlePrompt`, and `idleTimeout`.
`agentImage`	(Optional) A local image file path or `Buffer` to upload as the avatar. Exactly one of `agentImageUrl` / `agentId` / `agentImage` is required.
`agentImageMimeType`	(Optional) MIME type for `agentImage` when provided as a `Buffer` (for example, `'image/jpeg'`). Ignored for file paths. Defaults to `'image/png'`.
`agentPrompt`	(Optional) A high-level system prompt that subtly influences the avatar’s movements, expressions, and emotional demeanor while speaking. This prompt is best used to suggest general affect or demeanor (for example, “feel excited” or “look sad”) rather than precise or deterministic actions. Defaults to `"a person talking"`.
`agentIdlePrompt`	(Optional) A high-level system prompt that influences the avatar’s movements, expressions, and emotional demeanor during the idle state. Defaults to `"a serious person"`.
`idleTimeout`	(Optional) Idle timeout in seconds. If a negative number is provided, the session will have no idle timeout. Defaults to 60.
`extraPayload`	(Optional) Additional LemonSlice session fields forwarded on session create. See extraPayload below.

extraPayload (Node.js)

In the Node.js plugin, session fields that are not first-class AvatarSession options are passed through extraPayload.

Field	Description
`model`	Model variant (`"lite"`, `"flash"`, or `"pro"`). Leave unset to use the normal flagship model.
`aspectRatio`	Output aspect ratio (`"2x3"`, `"9x16"`, or `"1x1"`).
`responseDoneTimeout`	Time in seconds to wait without receiving new audio bytes before marking the response as complete.
`simulcast`	Enable WebRTC simulcast.

const avatar = new lemonslice.AvatarSession({
  agentImageUrl: "...",
  agentPrompt: "a person talking",
  agentIdlePrompt: "a serious person",
  extraPayload: {
    model: "flash",
    aspectRatio: "1x1",
    responseDoneTimeout: 0.8,
  },
});

When using the Gemini Live S2S model for realtime interactions, set response_done_timeout=0.8 to handle end of responses correctly.If you encounter stutters or glitches with any other TTS, please contact support@lemonslice.com.

Hook into Events

Listen to LemonSlice RPC events over the LiveKit data channel to better manage the avatar lifecycle:

LEMONSLICE_RPC_TOPIC = "lemonslice"
AVATAR_READY_MSG_TYPE = "bot_ready"

def on_data_received(packet: rtc.DataPacket) -> None:
    if packet.topic != LEMONSLICE_RPC_TOPIC:
        return

    try:
        payload = json.loads(packet.data.decode("utf-8"))
    except (UnicodeDecodeError, json.JSONDecodeError):
        logger.warning("Unable to decode LemonSlice RPC JSON", exc_info=True)
        return

    if (
        not isinstance(payload, dict)
        or payload.get("type") != AVATAR_READY_MSG_TYPE
    ):
        return

    session_id = payload.get("session_id")
    logger.info(f"Avatar has joined and is ready! session_id = {session_id}")

room.on("data_received", on_data_received)

const LEMONSLICE_RPC_TOPIC = "lemonslice";
const AVATAR_READY_MSG_TYPE = "bot_ready";

room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
    if (topic !== LEMONSLICE_RPC_TOPIC) return;

    try {
        const data = JSON.parse(new TextDecoder().decode(payload));

        if (typeof data !== "object" || data.type !== AVATAR_READY_MSG_TYPE) return;

        const sessionId = data.session_id;
        console.log(`LemonSlice is actively streaming A/V to the user! session_id = ${sessionId}`);
    } catch (e) {
        console.warn("Unable to decode LemonSlice RPC JSON", e);
    }
});

LemonSlice Events

Event Type	Description
`bot_ready`	Fired when LemonSlice is streaming audio and video to the user. There may be a small delay between this event and the user seeing actual video frames if there are network issues.
`idle_timeout`	Emitted when the avatar session stops due to inactivity. The idle timeout threshold is defined by the avatar’s configuration.
`error`	A pipeline-related error occurred. Includes `error` and `fatal` fields.
`video_generation_error`	A failure occurred while generating a video segment.
`metric`	Returned after every avatar response with performance metrics. Includes `time_to_first_push` (time from receiving the first audio byte to pushing the first generated video frame). If LiveKit/TTS failed to deliver audio in real time, `tts_audio_delay` will be set to `true`.

These events are available to both your backend agent and frontend client, allowing you to coordinate avatar state across your entire application.

Preview

Preview the avatar in the Agents Playground or refer to one of our starter projects for sample frontend code.

Shutdown the LiveKit room

Gracefully shut down the LiveKit room, LiveKit agent, and/or LemonSlice avatar session:

Call ctx.room.disconnect() to close the LiveKit room connection which will end the LemonSlice avatar session.
Call ctx.shutdown() to stop the Agent’s JobContext and the LemonSlice avatar session if you don’t want to shutdown the LiveKit room.
Call the session control endpoint with the terminate event to shutdown only the LemonSlice avatar without shutting down the LiveKit room or agent.

where ctx is LiveKit’s agents.JobContext defined by the function annotated with @server.rtc_session()

PYTHON

@server.rtc_session()
async def my_agent(ctx: agents.JobContext):
  ...

If the LemonSlice session is not shut down, it will remain active until the configured idle timeout is reached or your session reaches the maximum allowed duration.

Multiple avatars in one room

You can put several LemonSlice avatars in the same LiveKit room. Each avatar needs:

A unique LiveKit participant identity (and preferably a display name)
Its own AgentSession — one session drives one avatar

import asyncio

from livekit import agents
from livekit.agents import Agent, AgentSession
from livekit.plugins import lemonslice

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    characters = [
        {
            "name": "Alice",
            "identity": "lemonslice-avatar-alice",
            "image_url": "https://example.com/alice.jpg",
            "instructions": "You are Alice, a helpful travel guide.",
        },
        {
            "name": "Bob",
            "identity": "lemonslice-avatar-bob",
            "image_url": "https://example.com/bob.jpg",
            "instructions": "You are Bob, a helpful food guide.",
        },
    ]

    sessions = []
    for character in characters:
        session = AgentSession(
            # Add STT, LLM, TTS, and other components here
        )
        avatar = lemonslice.AvatarSession(
            agent_image_url=character["image_url"],
            avatar_participant_identity=character["identity"],
            avatar_participant_name=character["name"],
        )
        agent = Agent(instructions=character["instructions"])
        sessions.append((session, avatar, agent))

    # Start all avatars in parallel, then start the agents that drive them
    await asyncio.gather(
        *(avatar.start(session, room=ctx.room) for session, avatar, _ in sessions)
    )
    await asyncio.gather(
        *(
            session.start(room=ctx.room, agent=agent)
            for session, _, agent in sessions
        )
    )

import { defineAgent, voice } from '@livekit/agents';
import * as lemonslice from '@livekit/agents-plugin-lemonslice';

export default defineAgent({
  entry: async (ctx) => {
    await ctx.connect();
    const room = ctx.room;

    const characters = [
        {
            name: "Alice",
            identity: "lemonslice-avatar-alice",
            imageUrl: "https://example.com/alice.jpg",
            instructions: "You are Alice, a helpful travel guide.",
        },
        {
            name: "Bob",
            identity: "lemonslice-avatar-bob",
            imageUrl: "https://example.com/bob.jpg",
            instructions: "You are Bob, a helpful food guide.",
        },
    ];

    const sessions = characters.map((character) => ({
        session: new voice.AgentSession({
            // Add STT, LLM, TTS, and other components here
        }),
        avatar: new lemonslice.AvatarSession({
            agentImageUrl: character.imageUrl,
            avatarParticipantIdentity: character.identity,
            avatarParticipantName: character.name,
        }),
        agent: voice.Agent.create({ instructions: character.instructions }),
    }));

    // Start all avatars in parallel, then start the agents that drive them
    await Promise.all(
        sessions.map(({ session, avatar }) => avatar.start(session, room)),
    );
    await Promise.all(
        sessions.map(({ session, agent }) => session.start({ room, agent })),
    );
  },
});

For the fastest setup time, start the avatars in parallel, then start the agent sessions in parallel (asyncio.gather / Promise.all).

If a value is not provided for the avatar participant identity in the AvatarSession declaration, the default value "lemonslice-avatar-agent" is used. Since two avatars cannot share an identity, concurrent avatars must set this property explicitly. On the frontend, filter by participant identity rather than assuming a single avatar participant.

Deploy on LiveKit Cloud

While you can host your agents on an external Python or Node.js server, we recommend LiveKit Cloud. It reduces setup and provides infrastructure, hosting, and observability out of the box.

Starter projects

Repo	What’s included
`02-livekit-playground-demo`	Starter code to launch a LiveKit agent with the LemonSlice avatar plugin. Run your agent locally and connect in the LiveKit playground for rapid prototyping
`03-livekit-app-python`	Fullstack web app using the LiveKit Python SDK. Includes LiveKit Agent + LemonSlice code, front-end UI, and backend token server.
`04-livekit-app-nodejs`	Fullstack web app using the LiveKit Node.js SDK. Includes LiveKit Agent + LemonSlice code, front-end UI, and backend token server.

Integrations

Prompting guide

Advanced

Endpoints

Overview

Prerequisites

How to use

Multiple avatars in one room

Deploy on LiveKit Cloud

Starter projects

​Overview

​Prerequisites

​How to use

​Multiple avatars in one room

​Deploy on LiveKit Cloud

​Starter projects

Overview

Prerequisites

How to use

Multiple avatars in one room

Deploy on LiveKit Cloud

Starter projects