> ## Documentation Index
> Fetch the complete documentation index at: https://lemonslice.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocket Integration

> Drive a LemonSlice Avatar from your own audio stack over WebSocket

## Overview

The WebSocket integration lets you drive a LemonSlice avatar from any audio stack. Send PCM audio and control events to LemonSlice over a WebSocket. Synchronized audio and video is streamed to a [LiveKit](https://livekit.io) or [Daily](https://www.daily.co) room that you control.

## Prerequisites

1. A LemonSlice account and API key associated with an active subscription. Create a key at [lemonslice.com/developers](https://lemonslice.com/developers).
2. A reference image for your avatar. See [our guide](/docs/prompting-guide/avatar-image-tips) for recommendations, or set `edit_image` to `true` to automatically reframe your image for best results.
3. A LiveKit or Daily room with separate credentials for:
   * LemonSlice to join and publish audio and video.
   * Your client to join and subscribe to the avatar stream.

For LiveKit, [generate access tokens](https://docs.livekit.io/frontends/reference/tokens-grants/); the room is created when the first participant joins. For Daily, [create a room](https://docs.daily.co/reference/rest-api/rooms/create-room) and [create meeting tokens](https://docs.daily.co/reference/rest-api/meeting-tokens/create-meeting-token).

## How to use

<Steps>
  <Step title="Create a session">
    Send a request to the LemonSlice sessions endpoint with your avatar and the credentials LemonSlice should use to publish into your room.

    <CodeGroup>
      ```bash LIVEKIT theme={null}
      curl --request POST \
        --url "https://lemonslice.com/api/liveai/sessions" \
        --header "X-API-Key: $LEMONSLICE_API_KEY" \
        --header "Content-Type: application/json" \
        --data '{
          "transport_type": "websocket-livekit",
          "agent_image_url": "https://example.com/avatar.jpg",
          "livekit_properties": {
            "livekit_url": "wss://your-project.livekit.cloud",
            "livekit_token": "<agent_access_token>"
          }
        }'
      ```

      ```bash DAILY theme={null}
      curl --request POST \
        --url "https://lemonslice.com/api/liveai/sessions" \
        --header "X-API-Key: $LEMONSLICE_API_KEY" \
        --header "Content-Type: application/json" \
        --data '{
          "transport_type": "websocket-daily",
          "agent_image_url": "https://example.com/avatar.jpg",
          "daily_properties": {
            "daily_url": "https://your-domain.daily.co/room-name",
            "daily_token": "<agent_meeting_token>"
          }
        }'
      ```
    </CodeGroup>

    You can also provide `agent_image_base64` for a local image, or `agent_id` to use the image for an avatar created on the LemonSlice website.

    | Parameter | Description |
    | - | - |
    | `transport_type` | `"websocket-livekit"` or `"websocket-daily"`. |
    | `agent_image_url` | A publicly accessible URL to the avatar image. Exactly one of `agent_image_url` / `agent_id` / `agent_image_base64` is required. |
    | `agent_id` | The ID of a LemonSlice avatar. Exactly one of `agent_image_url` / `agent_id` / `agent_image_base64` is required. If provided, avatar-configured settings are used as defaults for `agent_prompt`, `agent_idle_prompt`, and `idle_timeout`. |
    | `agent_image_base64` | Base64-encoded avatar image bytes. Exactly one of `agent_image_url` / `agent_id` / `agent_image_base64` is required. |
    | `edit_image` | Boolean. If True, the source image is reframed before the session starts. Defaults to True for accounts created after September 22, 2026. Currently only works with a `2x3` aspect ratio. See more in [Avatar image tips](/docs/prompting-guide/avatar-image-tips#how-to-get-the-best-quality-avatars). |
    | `edit_image_variant` | Framing to generate when `edit_image` runs: `"standing"` (default), `"bust"` (default when actions are enabled), `"sitting"`, or `"portrait"`. |
    | `agent_prompt` | A high-level system prompt that subtly influences the avatar's movements, expressions, and emotional demeanor while speaking. Defaults to `"a person talking"`. |
    | `agent_idle_prompt` | A high-level system prompt that influences the avatar's movements, expressions, and emotional demeanor during the idle state. Defaults to `"a serious person"`. |
    | `idle_timeout` | Idle timeout in seconds. If a negative number is provided, the session will have no idle timeout. Defaults to 60. We **strongly** discourage disabling the idle timeout during development to avoid excessive credit usage. |
    | `response_done_timeout` | Time in seconds to wait without receiving new audio bytes before marking the response as complete. Some TTS models do not send an end response event, or do not send it in a timely manner. |
    | `model` | Model variant (`"cwm-1"`, `"cwm-1-lite"`, `"cartoon"`, `"flash"`, or `"pro"`). Leave unset to use the normal flagship model. |
    | `aspect_ratio` | Output aspect ratio (`"2x3"`, `"9x16"`, or `"1x1"`). Defaults to `"2x3"`. |
    | `background_image_url` | Public URL of an image composited behind the avatar. Incompatible with `video_out_scale` other than `1`. |
    | `video_out_scale` | Output scale from `1` to `2.5`. Incompatible with `background_image_url` when set above `1`. |
    | `enable_actions` | Boolean. Defaults to `false`. Set to `true` to use `action_engine` and `intro_action`. See [CWM-1](/docs/reference/cwm-1). |
    | `action_engine` | Action engine to use when `enable_actions` is `true`. Defaults to `"none"`. |
    | `intro_action` | Action to play when the session starts. Requires `action_engine` set to an engine other than `"none"`. |
    | `livekit_properties` | Required when `transport_type` is `"websocket-livekit"`. See below. |
    | `daily_properties` | Required when `transport_type` is `"websocket-daily"`. See below. |

    `livekit_properties`:

    | Field | Description |
    | - | - |
    | `livekit_url` | LiveKit server URL of the room LemonSlice should join and publish avatar audio and video into. |
    | `livekit_token` | LiveKit access token with publish permission. |
    | `simulcast` | Boolean. Defaults to `false`. Set to `true` to enable [WebRTC simulcast](https://livekit.com/blog/an-introduction-to-webrtc-simulcast). |
    | `video_codec` | Video codec for the published avatar track (`"vp8"`, `"h264"`, `"vp9"`, `"av1"`, or `"h265"`). Defaults to `"vp8"`. |
    | `scalability_mode` | SVC mode that lets LiveKit send a lighter stream when a viewer's connection can't keep up. Applied only when `video_codec` is `"vp9"` or `"av1"`; ignored otherwise. |

    `daily_properties`:

    | Field | Description |
    | - | - |
    | `daily_url` | Daily room URL LemonSlice should join and publish avatar audio and video into. |
    | `daily_token` | Daily meeting token with publish permission. |

    When an edit is applied, the response includes `edited_image_url`.

    A successful response includes the WebSocket address:

    ```json theme={null}
    {
      "session_id": "...",
      "websocket_address": "wss://...",
      "control_url": "..."
    }
    ```
  </Step>

  <Step title="Open the WebSocket">
    Connect a standard WebSocket client to `websocket_address`:

    ```javascript theme={null}
    const socket = new WebSocket(websocketAddress);
    ```

    No additional authentication header is required. All messages are JSON text frames, and only one WebSocket client may be connected to a session at a time.
  </Step>

  <Step title="Stream an audio turn">
    Send mono PCM16 audio as base64-encoded chunks. We recommend chunks containing approximately 80–100 ms of audio. You can send them as quickly as they become available. They do not need to be paced in real time.

    <Tip>
      LemonSlice processes audio at 16 kHz. Send audio at that rate to avoid overhead associated with resampling.
    </Tip>

    ```json theme={null}
    {
      "command": "audio",
      "audio": "<base64_pcm16_mono>",
      "sampleRate": 16000,
      "encoding": "PCM16"
    }
    ```

    | Field | Description |
    | - | - |
    | `audio` | Base64-encoded mono PCM16 audio. |
    | `sampleRate` | Sample rate of the audio in Hz. Required. Other rates are accepted and resampled to 16 kHz. |
    | `encoding` | Must be `"PCM16"` when provided. Defaults to `"PCM16"` if omitted. |

    End every response with `audio_end`:

    ```json theme={null}
    { "command": "audio_end" }
    ```

    A complete turn is `audio` → `audio` → … → `audio_end`. The final command commits the turn so LemonSlice can finish rendering it. Without `audio_end`, trailing audio may be dropped and your avatar will freeze between turns.
  </Step>

  <Step title="Subscribe to the avatar">
    Join the LiveKit or Daily room from your client using its viewer credentials. LemonSlice publishes the generated avatar audio and video into this room.
  </Step>
</Steps>

## Handle playback completion

LemonSlice sends a `playback_finished` event over the WebSocket when avatar A/V playback completes:

```json theme={null}
{
  "command": "playback_finished",
  "playback_position": 4.2,
  "interrupted": false
}
```

Use this event as the source of truth for whether the avatar is still playing:

* After sending audio for a response, treat playback as active until you receive its `playback_finished` event.
* If the user interrupts before that event arrives, send an `interrupt` command even if your TTS provider has finished sending audio or the estimated playback duration has elapsed.
* Once `playback_finished` arrives, the response is complete and an `interrupt` for that response is unnecessary.

This avoids a desynchronization window between your TTS pipeline and the avatar's actual A/V playback.

## Interrupt playback

To stop the current avatar response, send an `interrupt` event over the WebSocket and drop any queued audio chunks that had not yet been sent:

```json theme={null}
{ "command": "interrupt" }
```

## Heartbeats

Use heartbeats to confirm that the WebSocket is responsive:

```json theme={null}
{
  "command": "heartbeat",
  "event_id": "optional-id",
  "timestamp": 1710000000000
}
```

LemonSlice replies with `heartbeat_ack`, echoing `event_id` and `timestamp` when present.

## End the session

When the call is complete, shut down the LemonSlice session by sending a `terminate` event, then close the WebSocket:

```json theme={null}
{ "command": "terminate" }
```

<Warning>
  Always terminate sessions explicitly. Otherwise, the session and billing remain active until its idle timeout or maximum duration is reached.
</Warning>

## WebSocket event summary

| Client → LemonSlice | Description |
| :- | :- |
| `audio` | Base64-encoded mono PCM16 audio chunk. Include `sampleRate` and optional `encoding` (`"PCM16"`). |
| `audio_end` | Commits the current turn after the last `audio` chunk. |
| `interrupt` | Stops the current avatar response mid-playback. |
| `heartbeat` | Liveness check. Optional `event_id` and `timestamp` are echoed in the ack. |
| `terminate` | Ends the LemonSlice session. Close the WebSocket afterward. |

| LemonSlice → client | Description |
| :- | :- |
| `playback_finished` | Avatar A/V playback for a response has completed. Includes `playback_position` (float) and `interrupted` (bool). |
| `heartbeat_ack` | Reply to `heartbeat`, echoing `event_id` and `timestamp` when present. |

## Starter projects

| Repo | What's included |
| :- | :- |
| [`10-websocket-app`](https://github.com/lemonsliceai/lemonslice-examples/tree/main/10-websocket-app) | Fullstack web app that connects an ElevenLabs voice agent to LemonSlice via WebSocket and receives avatar A/V over LiveKit. |

## Additional resources

Use the [session control endpoint](/docs/api-reference/control-session) for mid-call triggers such as real-time avatar image updates.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.