> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akool.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming Avatar API Overview

> Comprehensive guide to the Streaming Avatar API

> [!NOTE]
>
> Both the avatar\_id and voice\_id can be easily obtained by copying them directly from the web interface. You can also create and manage your streaming avatars using our intuitive web platform.
>
> Create and manage your avatars at: [https://akool.com/apps/upload/avatar?from=%2Fapps%2Fstreaming-avatar%2Fedit](https://akool.com/apps/upload/avatar?from=%2Fapps%2Fstreaming-avatar%2Fedit)

> [!WARNING]
>
> The resources (image, video, voice) generated by our API are valid for 7 days.
> Please save the relevant resources as soon as possible to prevent expiration.

> [!NOTE]
>
> To experience our live avatar streaming feature in action, explore our demo built on the Agora streaming service: [AKool Streaming Avatar React Demo](https://github.com/AKOOL-Official/akool-streaming-avatar-react-demo).

> [!NOTE]
>
> **Knowledge Base Integration:** You can enhance your streaming avatar with contextual AI responses by integrating a [Knowledge Base](https://docs.akool.com/ai-tools-suite/knowledge-base.md). When creating a session, provide a `knowledge_id` parameter to enable the AI to use documents and URLs from your knowledge base for more accurate and relevant responses.

### API Endpoints

#### Avatar Management

* [Upload Streaming Avatar](https://docs.akool.com/ai-tools-suite/live-avatar/upload.md) - Create a new streaming avatar from a video URL
* [Get Avatar List](https://docs.akool.com/ai-tools-suite/live-avatar/list.md) - Retrieve a list of all streaming avatars
* [Get Avatar Detail](https://docs.akool.com/ai-tools-suite/live-avatar/detail.md) - Get detailed information about a specific avatar

#### Session Management

* [Create Session](https://docs.akool.com/ai-tools-suite/live-avatar/create-session.md) - Create a new streaming avatar session
* [Get Session Detail](https://docs.akool.com/ai-tools-suite/live-avatar/session-detail.md) - Retrieve detailed information about a specific session
* [Close Session](https://docs.akool.com/ai-tools-suite/live-avatar/close-session.md) - Close an active streaming avatar session
* [Get Session List](https://docs.akool.com/ai-tools-suite/live-avatar/list-sessions.md) - Retrieve a list of all streaming avatar sessions

### Live Avatar Stream Message

```ts theme={null}
IAgoraRTCClient.on(event: "stream-message", listener: (uid: UID, pld: Uint8Array) => void)

IAgoraRTCClient.sendStreamMessage(msg: Uint8Array | string, flag: boolean): Promise<void>;
```

**Send Data**

**Chat Type Parameters**

| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| - | - | - | - | - |
| v | Number | Yes | 2 | Version of the message |
| type | String | Yes | chat | Message type for chat interactions |
| mid | String | Yes | | Unique message identifier for conversation tracking |
| idx | Number | Yes | | Sequential index of the message, start from 0 |
| fin | Boolean | Yes | | Indicates if this is the final part of the message |
| pld | Object | Yes | | Container for message payload |
| pld.text | String | Yes | | Text content to send to avatar (e.g. "Hello") |

**Command Type Parameters**

| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| - | - | - | - | - |
| v | Number | Yes | 2 | Protocol version number |
| type | String | Yes | command | Specifies this is a system command message |
| mid | String | Yes | | Unique ID to track and correlate command messages |
| pld | Object | Yes | | Contains the command details and parameters |
| pld.cmd | String | Yes | | Command action to execute. Valid values: **"set-params"** (update avatar settings), **"interrupt"** (stop current avatar response) |
| pld.data | Object | No | | Parameters for the command (required for **"set-params"**) |
| pld.data.vid | String | No | | **Deprecated. Use pld.data.vparams.vid instead.** Voice ID to change avatar's voice. Only used with **"set-params"**. Get valid IDs from [Voice List API](https://docs.akool.com/ai-tools-suite/voiceLab.md#get-voice-list) |
| pld.data.vurl | String | No | | **Deprecated. Use pld.data.vparams.vurl instead.** Custom voice model URL. Only used with **"set-params"**. Get valid URLs from [Voice List API](https://docs.akool.com/ai-tools-suite/voiceLab.md#get-voice-list) |
| pld.data.lang | String | No | | Language code for avatar responses (e.g. "en", "es"). Only used with **"set-params"**. Get valid codes from [Language List API](https://docs.akool.com/ai-tools-suite/video-translation.md#get-language-list-result) |
| pld.data.mode | Number | No | | Avatar interaction style. Only used with **"set-params"**. "1" = Retelling (avatar repeats content), "2" = Dialogue (avatar engages in conversation) |
| pld.data.bgurl | String | No | | URL of background image/video for avatar scene. Only used with **"set-params"** |
| pld.data.vparams | Object | No | | Voice parameters to use for the session. |

**Voice Parameters**

| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| - | - | - | - | - |
| vid | String | No | | Voice ID to change avatar's voice. Only used with **"set-params"**. Get valid IDs from [Voice List API](https://docs.akool.com/ai-tools-suite/voiceLab.md#get-voice-list) |
| vurl | String | No | | Custom voice model URL. Only used with **"set-params"**. Get valid URLs from [Voice List API](https://docs.akool.com/ai-tools-suite/voiceLab.md#get-voice-list) |
| speed | double | No | 1 | Controls the speed of the generated speech. Values range from 0.8 to 1.2, with 1.0 being the default speed. |
| stt\_language | String | No | | Language code for speech-to-text recognition to improve accuracy by using language-specific models |
| pron\_map | Object | No | | Pronunciation mapping for custom words. Example: "pron\_map": \{ "akool" : "ai ku er" } |
| stt\_type | String | No | | Speech-to-text type. `"openai_realtime"` = OpenAI Realtime |
| turn\_detection | Object | No | | Turn detection configuration. |

**Turn Detection Configuration**

| **Parameter** | **Type** | **Required** | **Value** | **Description** |
| - | - | - | - | - |
| type | String | No | "server\_vad" | Turn detection type. `"server_vad"` = Server VAD, `"semantic_vad"` = Semantic VAD |
| threshold | Number | No | 0.5 | Activation threshold (0 to 1). A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. Available when type is `"server_vad"`. |
| prefix\_padding\_ms | Number | No | 300 | Amount of audio (in milliseconds) to include before the VAD detected speech. Available when type is `"server_vad"`. |
| silence\_duration\_ms | Number | No | 500 | Duration of silence (in milliseconds) to detect speech stop. With shorter values turns will be detected more quickly. Available when type is `"server_vad"`. |

**JSON Example**

```json Chat Request theme={null}
{
    "v": 2,
    "type": "chat",
    "mid": "msg-1723629433573",
    "idx": 0,
    "fin": true,
    "pld": {
      "text": "Hello"
    },
}
```

```json Set Avatar Params theme={null}
{
    "v": 2,
    "type": "command",
    "mid": "msg-1723629433573",
    "pld": {
      "cmd": "set-params",
      "data": {
        "vid": "1",
        "lang": "en",
        "mode": 1,
        "bgurl": "https://example.com/background.jpg"
      }
    },
}
```

```json Interrupt Response theme={null}
{
    "v": 2,
    "type": "command",
    "mid": "msg-1723629433573",
    "pld": {
      "cmd": "interrupt"
    },
}
```

```json Set Avatar Actions theme={null}
{
    "v": 2,
    "type": "command",
    "pld": {
        "cmd": "set-action",
        "data": {
            "action": "hand_wave", // "hand_wave"|"cheer"|"thumbs_up"|"pump_fists"|"nod"|"shake"|"waft_left"|"waft_right"
            "expression": "happiness" // "angry"|"disgust"|"fear"|"happiness"|"sadness"|"surprise"|"contempt"
        }
    }
}
```

**Receive Data**

**Chat Type Parameters**

| **Parameter** | **Type** | **Value** | **Description** |
| - | - | - | - |
| v | Number | 2 | Version of the message |
| type | String | chat | Message type for chat interactions |
| mid | String | | Unique message identifier for tracking conversation flow |
| idx | Number | | Sequential index of the message part |
| fin | Boolean | | Indicates if this is the final part of the response |
| pld | Object | | Container for message payload |
| pld.from | String | "bot" or "user" | Source of the message - "bot" for avatar responses, "user" for speech recognition input |
| pld.text | String | | Text content of the message |

**Command Type Parameters**

| **Parameter** | **Type** | **Value** | **Description** |
| - | - | - | - |
| v | Number | 2 | Version of the message |
| type | String | command | Message type for system commands |
| mid | String | | Unique identifier for tracking related messages in a conversation |
| pld | Object | | Container for command payload |
| pld.cmd | String | "set-params", "interrupt" | Command to execute: **"set-params"** to update avatar settings, **"interrupt"** to stop current response |
| pld.code | Number | 1000 | Response code from the server, 1000 indicates success |
| pld.msg | String | | Response message from the server |

**JSON Example**

```json Chat Response theme={null}
{
    "v": 2,
    "type": "chat",
    "mid": "msg-1723629433573",
    "idx": 0,
    "fin": true,
    "pld": {
      "from": "bot",
      "text": "Hello!  How can I assist you today? "
    }
}
```

```json Command Response theme={null}
{
    "v": 2,
    "type": "command",
    "mid": "msg-1723629433573",
    "pld": {
      "cmd": "set-params",
      "code": 1000,
      "msg": "OK"
    }
}
```

**Typescript Example**

```ts Create Client theme={null}
const client: IAgoraRTCClient = AgoraRTC.createClient({
  mode: 'rtc',
  codec: 'vp8',
});

client.join(agora_app_id, agora_channel, agora_token, agora_uid);

client.on('stream-message', (message: Uint8Array | string) => {
  console.log('received: %s', message);
});
```

```ts Send Message theme={null}
const msg = JSON.stringify({
  v: 2,
  type: "chat",
  mid: "msg-1723629433573",
  idx: 0,
  fin: true,
  pld: {
      text: "hello"
  },
});

await client.sendStreamMessage(msg, false);
```

```ts Set Avatar Params theme={null}
const setAvatarParams = async () => {
  const msg = JSON.stringify({
    v: 2,
    type: 'command',
    pld: {
      cmd: 'set-params',
      params: {
        vid: voiceId,
        lang: language,
        mode: modeType,
      },
    },
  });
  return client.sendStreamMessage(msg, false);
};
```

```ts Interrupt Response theme={null}
const interruptResponse = async () => {
  const msg = JSON.stringify({
    v: 2,
    type: 'command',
    pld: {
      cmd: 'interrupt',
    },
  });
  return client.sendStreamMessage(msg, false);
};
```

### Integrating Your Own LLM Service

Before dispatching a message to the WebSocket, consider executing an HTTP request to your LLM service.

```ts TypeScript theme={null}
const client: IAgoraRTCClient = AgoraRTC.createClient({
  mode: 'rtc',
  codec: 'vp8',
});

client.join(agora_app_id, agora_channel, agora_token, agora_uid);

client.on('stream-message', (message: Uint8Array | string) => {
  console.log('received: %s', message);
});

let inputMessage = 'hello';

try {
  const response = await fetch('https://your-backend-host/api/llm/answer', {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      question: inputMessage,
    }),
  });

  if (response.ok) {
    const result = await response.json();
    inputMessage = result.answer;
  } else {
    console.error("Failed to fetch from backend", response.statusText);
  }
} catch (error) {
  console.error("Error during fetch operation", error);
}

const message = {
  v: 2,
  type: "chat",
  mid: "msg-1723629433573",
  idx: 0,
  fin: true,
  pld: {
      text: inputMessage,
  },
};
client.sendStreamMessage(JSON.stringify(message), false);
```

### Response Code Description

> [!NOTE]
>
> &#x20;
>
> Please note that if the value of the response code is not equal to 1000, the request
> is failed or wrong

| **Parameter** | **Value** | **Description** |
| - | - | - |
| code | 1000 | Success |
| code | 1003 | Parameter error or Parameter can not beempty |
| code | 1008 | The content you get does not exist |
| code | 1009 | Youdo not have permission to operate |
| code | 1101 | Invalid authorization or Therequest token has expired |
| code | 1102 | Authorization cannot be empty |
| code | 1200 | The account has been banned |
| code | 1201 | create audio error, pleasetry again later |
| code | 1202 | The same video cannot be translated lipSync inthe same language more than 1 times |
| code | 1203 | video should be with audio |
| code | 1204 | Your video duration is exceed 60s! |
| code | 1205 | Create videoerror, please try again later |
| code | 1207 | The video you are using exceeds thesize limit allowed by the system by 300M |
| code | 1209 | Please upload a videoin another encoding format |
| code | 1210 | The video you are using exceeds thevalue allowed by the system by 60fp |
| code | 1211 | Create lipsync error, pleasetry again later |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.