Skip to content

feat(js/plugins/vertexai): add support to anthropic prompt caching #2886

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

amondnet
Copy link

@amondnet amondnet commented May 8, 2025

Description here... Help the reviewer by:

When using Anthropic models, you can set up cache control for system messages. You can enable this feature by adding a cacheControl property to the custom field of a message. The only currently supported cache type is ephemeral.

For example, to use cache control for a specific system message, you would set it up like this:

const llmResponse = await ai.generate({
  model: claude3Sonnet, // or another Anthropic model
  messages: [
    {
      role: 'system',
      content: [
        {
          text: 'This is an important instruction that can be cached.',
          custom: {
            cacheControl: {
              type: 'ephemeral',
            },
          },
        },
      ],
    },
    {
      role: 'user',
      content: [{ text: 'What should I do when I visit Melbourne?' }],
    },
  ],
});

https://github.com/anthropics/anthropic-sdk-typescript/releases/tag/sdk-v0.33.0
anthropics/anthropic-sdk-typescript#631

Checklist (if applicable):

@github-actions github-actions bot added docs Improvements or additions to documentation js config labels May 8, 2025
@amondnet amondnet force-pushed the vertexai-anthropic-context-caching branch from 72b29ea to 20881ee Compare May 8, 2025 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
config docs Improvements or additions to documentation js
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

[JS] Add support to anthropic prompt caching
1 participant