> ## Documentation Index
> Fetch the complete documentation index at: https://lemonslice.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# LiveKit Integration
> Connect to a LemonSlice Avatar via LiveKit's Agents realtime framework
## Overview
[LiveKit Agents](https://docs.livekit.io/agents/) provides a realtime framework for voice, video, and multimodal applications. Our open-source integration lets you add a LemonSlice avatar to your existing agent stack.
Jump to the [starter projects](#starter-projects) section for ready-to-use repos. For production UI patterns (avatar readiness, error handling, timeouts), see [Production checklist](/docs/reference/production-checklist).
## Prerequisites
1. LemonSlice avatar configuration (**either** a base image URL or a LemonSlice avatar ID)
* Agent base image — a publicly accessible image URL of your avatar, focused on the face. The image should be 368 × 560 pixels. LemonSlice will automatically center-crop your image to the target aspect ratio if the dimensions do not match the expected values. Best results are achieved with anthropomorphic images where the face and mouth are clearly identifiable.
* [LemonSlice Agent ID](https://lemonslice.com/agents)
Selected voices and personalities for a LemonSlice agent will be ignored when using the LiveKit plugin.
2. LiveKit Agents app
* Your own existing application, or
* The ready-to-use [LiveKit playground starter project](https://github.com/lemonsliceai/lemonslice-examples/tree/main/02-livekit-playground-demo), or
* Follow the [LiveKit VoiceAI quickstart](https://docs.livekit.io/agents/start/voice-ai-quickstart/) for Python or Node.js to create one.
## How to use
1. Within your LiveKit Agents app, install the plugin:
```shell PYTHON theme={null}
# When using PIP
pip install "livekit-agents[lemonslice]"
# When using UV
uv add "livekit-agents[lemonslice]"
```
```shell NODE.JS theme={null}
pnpm add @livekit/agents-plugin-lemonslice
```
1. Create a [LemonSlice API key](https://lemonslice.com/agents/api)
2. In your LiveKit Agents app, set `LEMONSLICE_API_KEY` in your .env file
In your LiveKit Agents app, create a `lemonslice.AvatarSession` alongside your `AgentSession`:
```python PYTHON theme={null}
from livekit import agents
from livekit.agents import AgentSession, RoomOutputOptions
from livekit.plugins import lemonslice
async def entrypoint(ctx: agents.JobContext):
await ctx.connect()
session = AgentSession(
# Add STT, LLM, TTS, and other components here
)
avatar = lemonslice.AvatarSession(
# Publicly accessible image URL for the avatar
agent_image_url="...",
# Prompt to guide the avatar's demeanor while speaking
agent_prompt="a person talking",
)
# Start the avatar and wait for it to join
session_id = await avatar.start(session, room=ctx.room)
# Start your agent session with the user
await session.start(
# ... room, agent, room_options, etc....
)
```
```javascript NODE.JS theme={null}
import { voice } from '@livekit/agents';
import * as lemonslice from '@livekit/agents-plugin-lemonslice';
const session = new voice.AgentSession({
// Add STT, LLM, TTS, and other components here
});
const avatar = new lemonslice.AvatarSession({
// Publicly accessible image URL for the avatar
agentImageUrl: "...",
// Prompt to guide the avatar's demeanor while speaking
agentPrompt: "a person talking",
});
// Start the avatar and wait for it to join
const sessionId = await avatar.start(session, room);
// Start your agent session with the user
await session.start(
// ... room, agent, room_options, etc.
);
```
| Parameter | Description |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent_image_url` | *(Optional)* A publicly accessible URL to the avatar image. Exactly one of `agent_image_url` / `agent_id` / `agent_image` is required. |
| `agent_id` | *(Optional)* The ID of the LemonSlice agent. Exactly one of `agent_image_url` / `agent_id` / `agent_image` is required. If provided, we will also use agent-configured settings as defaults for `agent_prompt`, `agent_idle_prompt`, and `idle_timeout`. |
| `agent_image` | *(Optional)* A local PIL image to upload as the avatar. Exactly one of `agent_image_url` / `agent_id` / `agent_image` is required. |
| `agent_prompt` | *(Optional)* A high-level system prompt that subtly influences the avatar's movements, expressions, and emotional demeanor while speaking. This prompt is best used to suggest general affect or demeanor (for example, "feel excited" or "look sad") rather than precise or deterministic actions. Defaults to `"a person talking"`. |
| `agent_idle_prompt` | *(Optional)* A high-level system prompt that influences the avatar's movements, expressions, and emotional demeanor during the idle state. Defaults to `"a serious person"`. |
| `idle_timeout` | *(Optional)* Idle timeout in seconds. If a negative number is provided, the session will have no idle timeout. Defaults to 60. |
| `response_done_timeout` | *(Optional)* Time in seconds to wait without receiving new audio bytes before marking the response as complete. Some TTS models do not send an end response event, or do not send it in a timely manner, after transmitting all audio bytes. This parameter enables detection of response completion when such events are missing or delayed. |
| `model` | *(Optional)* Model variant (`"lite"`, `"flash"`, or `"pro"`). Leave unset to use the normal flagship model. |
| `aspect_ratio` | *(Optional)* Output aspect ratio (`"2x3"`, `"9x16"`, or `"1x1"`). |
| `simulcast` | *(Optional)* Enable [WebRTC simulcast](https://livekit.com/blog/an-introduction-to-webrtc-simulcast), which publishes multiple resolutions of the avatar video so subscribers receive the best quality their bandwidth allows. **LiveKit rooms only.** |
| Parameter | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agentImageUrl` | *(Optional)* A publicly accessible URL to the avatar image. Exactly one of `agentImageUrl` / `agentId` / `agentImage` is required. |
| `agentId` | *(Optional)* The ID of the LemonSlice agent. Exactly one of `agentImageUrl` / `agentId` / `agentImage` is required. If provided, we will also use agent-configured settings as defaults for `agentPrompt`, `agentIdlePrompt`, and `idleTimeout`. |
| `agentImage` | *(Optional)* A local image file path or `Buffer` to upload as the avatar. Exactly one of `agentImageUrl` / `agentId` / `agentImage` is required. |
| `agentImageMimeType` | *(Optional)* MIME type for `agentImage` when provided as a `Buffer` (for example, `'image/jpeg'`). Ignored for file paths. Defaults to `'image/png'`. |
| `agentPrompt` | *(Optional)* A high-level system prompt that subtly influences the avatar's movements, expressions, and emotional demeanor while speaking. This prompt is best used to suggest general affect or demeanor (for example, "feel excited" or "look sad") rather than precise or deterministic actions. Defaults to `"a person talking"`. |
| `agentIdlePrompt` | *(Optional)* A high-level system prompt that influences the avatar's movements, expressions, and emotional demeanor during the idle state. Defaults to `"a serious person"`. |
| `idleTimeout` | *(Optional)* Idle timeout in seconds. If a negative number is provided, the session will have no idle timeout. Defaults to 60. |
| `extraPayload` | *(Optional)* Additional LemonSlice session fields forwarded on session create. See [extraPayload](#extrapayload-nodejs) below. |
In the Node.js plugin, session fields that are not first-class `AvatarSession` options are passed through `extraPayload`.
| Field | Description |
| --------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `model` | Model variant (`"lite"`, `"flash"`, or `"pro"`). Leave unset to use the normal flagship model. |
| `aspectRatio` | Output aspect ratio (`"2x3"`, `"9x16"`, or `"1x1"`). |
| `responseDoneTimeout` | Time in seconds to wait without receiving new audio bytes before marking the response as complete. |
| `simulcast` | Enable [WebRTC simulcast](https://livekit.com/blog/an-introduction-to-webrtc-simulcast). **LiveKit rooms only.** |
```javascript theme={null}
const avatar = new lemonslice.AvatarSession({
agentImageUrl: "...",
agentPrompt: "a person talking",
agentIdlePrompt: "a serious person",
extraPayload: {
model: "flash",
aspectRatio: "1x1",
responseDoneTimeout: 0.8,
},
});
```
When using the [**Gemini Live S2S**](https://docs.livekit.io/agents/models/realtime/plugins/gemini/) model for realtime interactions, set `response_done_timeout=0.8` to handle end of responses correctly.
If you encounter stutters or glitches with any other TTS, please contact [support@lemonslice.com](mailto:support@lemonslice.com).
Listen to LemonSlice RPC events over the LiveKit data channel to better manage the avatar lifecycle:
```python Python theme={null}
LEMONSLICE_RPC_TOPIC = "lemonslice"
AVATAR_READY_MSG_TYPE = "bot_ready"
def on_data_received(packet: rtc.DataPacket) -> None:
if packet.topic != LEMONSLICE_RPC_TOPIC:
return
try:
payload = json.loads(packet.data.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError):
logger.warning("Unable to decode LemonSlice RPC JSON", exc_info=True)
return
if (
not isinstance(payload, dict)
or payload.get("type") != AVATAR_READY_MSG_TYPE
):
return
session_id = payload.get("session_id")
logger.info(f"Avatar has joined and is ready! session_id = {session_id}")
room.on("data_received", on_data_received)
```
```javascript Node.js theme={null}
const LEMONSLICE_RPC_TOPIC = "lemonslice";
const AVATAR_READY_MSG_TYPE = "bot_ready";
room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
if (topic !== LEMONSLICE_RPC_TOPIC) return;
try {
const data = JSON.parse(new TextDecoder().decode(payload));
if (typeof data !== "object" || data.type !== AVATAR_READY_MSG_TYPE) return;
const sessionId = data.session_id;
console.log(`LemonSlice is actively streaming A/V to the user! session_id = ${sessionId}`);
} catch (e) {
console.warn("Unable to decode LemonSlice RPC JSON", e);
}
});
```
**LemonSlice Events**
| Event Type | Description |
| :----------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bot_ready` | Fired when LemonSlice is streaming audio and video to the user. There may be a small delay between this event and the user seeing actual video frames if there are network issues. |
| `idle_timeout` | Emitted when the agent stops due to inactivity. The idle timeout threshold is defined by the agent's configuration. |
| `error` | A pipeline-related error occurred. Includes `error` and `fatal` fields. |
| `video_generation_error` | A failure occurred while generating a video segment. |
| `metric` | Returned after every avatar response with performance metrics. Includes `time_to_first_push` (time from receiving the first audio byte to pushing the first generated video frame). If LiveKit/TTS failed to deliver audio in real time, `tts_audio_delay` will be set to `true`. |
These events are available to both your backend agent and frontend client, allowing you to coordinate avatar state across your entire application.
Preview the avatar in the [Agents Playground](https://docs.livekit.io/agents/start/playground) or refer to one of our [starter projects](#starter-projects) for sample frontend code.
Gracefully shut down the LiveKit room, LiveKit agent, and/or LemonSlice avatar session:
1. Call `ctx.room.disconnect()` to close the LiveKit room connection which will end the LemonSlice avatar session.
2. Call `ctx.shutdown()` to stop the Agent's JobContext and the LemonSlice avatar session if you don't want to shutdown the LiveKit room.
3. Call the [session control endpoint](/docs/api-reference/control-session) with the `terminate` event to shutdown only the LemonSlice avatar without shutting down the LiveKit room or agent.
where `ctx` is LiveKit's `agents.JobContext` defined by the function annotated with `@server.rtc_session()`
```python PYTHON theme={null}
@server.rtc_session()
async def my_agent(ctx: agents.JobContext):
...
```
If the LemonSlice session is not shut down, the LemonSlice session will remain active until the configured idle timeout is reached or the 1-hour maximum session duration expires.
## Deploy on LiveKit Cloud
While you can host your agents on an external Python or Node.js server, we recommend [LiveKit Cloud](https://livekit.io/cloud). It reduces setup and provides infrastructure, hosting, and observability out of the box.
## Starter projects
| Repo | What's included |
| :----------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`02-livekit-playground-demo`](https://github.com/lemonsliceai/lemonslice-examples/tree/main/02-livekit-playground-demo) | Starter code to launch a LiveKit agent with the LemonSlice avatar plugin. Run your agent locally and connect in the [LiveKit playground](https://docs.livekit.io/agents/start/playground) for rapid prototyping |
| [`03-livekit-app-python`](https://github.com/lemonsliceai/lemonslice-examples/tree/main/03-livekit-app-python) | Fullstack web app using the [LiveKit Python SDK](https://github.com/livekit/agents). Includes LiveKit Agent + LemonSlice code, front-end UI, and backend token server. |
| [`04-livekit-app-nodejs`](https://github.com/lemonsliceai/lemonslice-examples/tree/main/04-livekit-app-nodejs) | Fullstack web app using the [LiveKit Node.js SDK](https://github.com/livekit/agents-js). Includes LiveKit Agent + LemonSlice code, front-end UI, and backend token server. |