anam plugin #692

i25959341 · 2025-09-12T12:54:19Z

Description

This PR add a plugin that works with Anam to provider AI avatar experience

Changes Made

add api.py for the api route with anam needed
add avatar.py for managing the session, and api calls with anam
add an example script that run the anam avatar

Pre-Review Checklist

Build passes: All builds (lint, typecheck, tests) pass locally
AI-generated code reviewed: Removed unnecessary comments and ensured code quality
Changes explained: All changes are properly documented and justified above
Scope appropriate: All changes relate to the PR title, or explanations provided for why they're included

Testing

Automated tests added/updated (if applicable) N/A
All tests pass
Make sure both restaurant_agent.ts and realtime_agent.ts work properly (for major changes) N/A

Additional Notes

Note to reviewers: Please ensure the pre-review checklist is completed before starting your review.

changeset-bot · 2025-09-12T12:54:22Z

🦋 Changeset detected

Latest commit: 3bbac7c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@livekit/agents-plugin-anam	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CLAassistant · 2025-09-12T12:54:30Z

All committers have signed the CLA.

…plugin

i25959341 · 2025-09-12T13:07:22Z

i25959341 · 2025-09-12T13:37:21Z

Transcription is not working for the agent, other than that I can have a converation with the avatar. Not sure if it s avatar plugin related

Shubhrakanti · 2025-10-06T15:30:58Z

https://www.loom.com/share/e9179bd171f142498124ad4583f2f5c5

the only remaining issue is the agent transcription, @Shubhrakanti @toubatbrian

Ahh is this because you need timed string response support?

i25959341 · 2025-10-06T18:37:03Z

timed string response support

Sorry @Shubhrakanti, can you expand on this a little bit? Is it related to something like elevenlab where they send words with timestamp for transcription sync on the agents side? So i need to swap to use a tts, sst rather than a real time model?

i25959341 · 2025-10-06T19:18:50Z

I remember this is an existing issue on python side as well

i have turn this syncTranscription off for now and we have the agent transcription now - this seems to suggest that we can not sync transciption with the audio

await session.start({ agent, room: ctx.room, outputOptions: { syncTranscription: false, }, });

Shubhrakanti · 2025-10-06T21:14:39Z

I'm super sorry this actually shouldn't affect the avatar side. What's the issue you're having with transcriptions?

Shubhrakanti · 2025-10-06T21:23:42Z

Oh I see from the look your not seeing the transcripts at all. @toubatbrian can you take a look? Might be an issue with DataStreamAOutput

…plugin

i25959341 · 2025-10-07T02:12:51Z

I remember that Anam only send Audio and Video frame back into the room but no transcription that goes with the Audio frame

https://github.com/livekit/agents/blob/main/examples/avatar_agents/audio_wave/avatar_runner.py#L77 - example implementation

Shubhrakanti · 2025-10-07T09:44:13Z

Interesting.... @toubatbrian can you take a look?

toubatbrian · 2025-10-07T18:04:40Z

Hey @i25959341 I'm trying to test your avatar plugin with my own API key but keep getting error

[02:03:36.210] ERROR (22281): Anam /v1/auth/session-token failed: 401 {"error":"Invalid API key"}
    jobID: "AJ_n3NavqiRdM36"
    err: {
      "type": "AnamException",
      "message": "Anam /v1/auth/session-token failed: 401 {\"error\":\"Invalid API key\"}",
      "stack":
          Error: Anam /v1/auth/session-token failed: 401 {"error":"Invalid API key"}
              at AnamAPI.postWithHeaders (file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/plugins/anam/dist/api.js:73:17)
              at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
              at async AvatarSession.start (file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/plugins/anam/dist/avatar.js:63:30)
              at async entry (file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/examples/src/anam_realtime_agent.ts:69:5)
              at async file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/agents/dist/ipc/job_proc_lazy_main.js:72:5

I tried creating multiple keys but getting the same result. Anything that I might be missing for my account?

toubatbrian · 2025-10-07T18:06:48Z

For your reference, here's the email I used to create anam account [email protected]

i25959341 · 2025-10-07T18:18:29Z

i think I have set

export ANAM_AVATAR_ID=
export ANAM_API_KEY=

ANAM_API_KEY should be the one that you are copying from the site

ANAM_AVATAR_ID, you can get one from here - https://docs.anam.ai/resources/avatar-gallery

This is gabriel for example, 6cc28442-cccd-42a8-b6e4-24b7210a09c5

i25959341 · 2025-10-07T18:22:53Z

@toubatbrian it is okay?

toubatbrian · 2025-10-07T18:23:08Z

I have this exact setup:

Brians-MacBook-Pro agents-js-anam % echo $ANAM_API_KEY && echo $ANAM_AVATAR_ID
MmI0Y2*****09sRzQ9
6cc28442-cccd-42a8-b6e4-24b7210a09c5

but still producing above error, not sure why. I also tested the same env vars with python agent and getting the same error

toubatbrian · 2025-10-07T18:26:59Z

Yes, Here's a more complete error log:

[02:22:49.411] ERROR (40089): Anam API request failed                                                                                              
    module: "AnamAPI"
    url: "https://api.anam.ai/v1/auth/session-token"
    method: "POST"
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer ****"
    }
    body: {
      "personaConfig": {
        "type": "ephemeral",
        "name": "Agent",
        "avatarId": "6cc28442-cccd-42a8-b6e4-24b7210a09c5",
        "llmId": "CUSTOMER_CLIENT_V1"
      },
      "environment": {
        "livekitUrl": "wss://dev-****.livekit.cloud",
        "livekitToken": "****"
      }
    }
    status: 401
    response: "{\"error\":\"Invalid API key\"}"
[02:22:49.411] ERROR (40089): Anam /v1/auth/session-token failed: 401 {"error":"Invalid API key"}
    jobID: "AJ_6XPC9qh7vHHo"
    err: {
      "type": "AnamException",
      "message": "Anam /v1/auth/session-token failed: 401 {\"error\":\"Invalid API key\"}",
      "stack":
          Error: Anam /v1/auth/session-token failed: 401 {"error":"Invalid API key"}
              at AnamAPI.postWithHeaders (file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/plugins/anam/dist/api.js:73:17)
              at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
              at async AvatarSession.start (file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/plugins/anam/dist/avatar.js:63:30)
              at async entry (file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/examples/src/anam_realtime_agent.ts:69:5)
              at async file:///Users/toubatbrian/Documents/GitHub/agents-js-anam/agents/dist/ipc/job_proc_lazy_main.js:72:5
    }

i25959341 · 2025-10-07T18:29:28Z

let me get back to you in 30 minutes

toubatbrian · 2025-10-07T18:33:17Z

Thanks, @i25959341 ! I'm heading to sleep now as I'm in Asia timezone, and will be back in about 7 hours. I'll be focusing on a fix for it first thing tomorrow.

toubatbrian · 2025-10-08T07:04:54Z

Hi @i25959341, from my end I can see exact transcripts coming as agent and avatar are speaking: . Is this not the case on your end? (I'm running the anam_realtime_agent.ts from this branch.

i25959341 · 2025-10-08T07:15:36Z

yes, it is working now as i have just made add the below option in the agent script

outputOptions: { syncTranscription: false, }

Sorry that I did not make this clear earlier @toubatbrian

if you are happy with this PR, maybe we can proceed to fix the nits and merge this PR?

toubatbrian · 2025-10-08T09:07:41Z

@Shubhrakanti We might need to look into potential bugs related to syncTranscription and TranscriptionSynchronizer. However, since that's a separate issue, I think it's fine to go ahead and merge the avatar PR for now. I'll create a ticket to track the fix.

toubatbrian

LGTM! For now, let's add a comment in anam_realtime_agent.ts to remind users to set syncTranscription to false. We'll work on a proper fix for it later.

i25959341 · 2025-10-08T13:58:26Z

@toubatbrian added the comment

Shubhrakanti · 2025-10-08T17:57:13Z

LGTM! For now, let's add a comment in anam_realtime_agent.ts to remind users to set syncTranscription to false. We'll work on a proper fix for it later.

@toubatbrian can you link this PR in the ticket?

toubatbrian · 2025-10-08T18:02:04Z

done

toubatbrian and others added 11 commits September 5, 2025 12:09

add

e08d21d

Update datastream_io.ts

85b05ff

Add DataStreamAudioOutput Utils (livekit#686)

1f98d41

Update datastream_io.ts

65ba88d

Update datastream_io.ts

a134c8c

fix lint

95cbf1a

import

a03f887

add license

41d237c

Create cold-cooks-smash.md

2a2101b

resolve import path

2240e3f

add anam plugni

efba233

i25959341 mentioned this pull request Sep 12, 2025

Support for avatar plugins #553

Open

i25959341 added 7 commits September 12, 2025 13:56

Merge branch 'main' of github.com:i25959341/agents-js into feat/anam-…

2d1aa26

…plugin

fix

d857367

fix

02e2e82

fix

84b1f77

fix

6cce422

fix

2e14c2a

fix

806afbc

i25959341 added 6 commits September 12, 2025 14:08

fix

4234c23

update

9ab5546

fix

587006d

fix

d738923

fix

0c5eb6f

fix

53b1dca

fix

91b9fcb

i25959341 requested review from Shubhrakanti and toubatbrian October 6, 2025 04:53

update

ca51c77

i25959341 added 2 commits October 7, 2025 03:08

Merge branch 'main' of github.com:i25959341/agents-js into feat/anam-…

ff88048

…plugin

fix

96c0213

toubatbrian approved these changes Oct 8, 2025

View reviewed changes

fix

3bbac7c

toubatbrian merged commit dcfa03f into livekit:main Oct 8, 2025
5 checks passed

github-actions bot mentioned this pull request Oct 8, 2025

Version Packages #760

Merged

anam plugin #692

anam plugin #692

Conversation

i25959341 commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes Made

Pre-Review Checklist

Testing

Additional Notes

Uh oh!

changeset-bot bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

CLAassistant commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i25959341 commented Sep 12, 2025

Uh oh!

i25959341 commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shubhrakanti commented Oct 6, 2025

Uh oh!

i25959341 commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i25959341 commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shubhrakanti commented Oct 6, 2025

Uh oh!

Shubhrakanti commented Oct 6, 2025

Uh oh!

i25959341 commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shubhrakanti commented Oct 7, 2025

Uh oh!

toubatbrian commented Oct 7, 2025

Uh oh!

toubatbrian commented Oct 7, 2025

Uh oh!

i25959341 commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i25959341 commented Oct 7, 2025

Uh oh!

toubatbrian commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toubatbrian commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i25959341 commented Oct 7, 2025

Uh oh!

toubatbrian commented Oct 7, 2025

Uh oh!

toubatbrian commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i25959341 commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toubatbrian commented Oct 8, 2025

Uh oh!

toubatbrian left a comment

Choose a reason for hiding this comment

Uh oh!

i25959341 commented Oct 8, 2025

Uh oh!

Uh oh!

Shubhrakanti commented Oct 8, 2025

Uh oh!

toubatbrian commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

i25959341 commented Sep 12, 2025 •

edited

Loading

changeset-bot bot commented Sep 12, 2025 •

edited

Loading

CLAassistant commented Sep 12, 2025 •

edited

Loading

i25959341 commented Sep 12, 2025 •

edited

Loading

i25959341 commented Oct 6, 2025 •

edited

Loading

i25959341 commented Oct 6, 2025 •

edited

Loading

i25959341 commented Oct 7, 2025 •

edited

Loading

i25959341 commented Oct 7, 2025 •

edited

Loading

toubatbrian commented Oct 7, 2025 •

edited

Loading

toubatbrian commented Oct 7, 2025 •

edited

Loading

toubatbrian commented Oct 8, 2025 •

edited

Loading

i25959341 commented Oct 8, 2025 •

edited

Loading