Skip to content

Read REPLICATE_API_TOKEN from cog context #434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 27, 2025
Merged

Conversation

aron
Copy link
Contributor

@aron aron commented May 23, 2025

This PR introduces support for the cog context introduced in 0.14.11 into the Replicate SDK. The current_scope helper now makes per-prediction context available via the current_scope().context dict.

A cog model can then provide a REPLICATE_API_TOKEN on a per-prediction basis to be used by the model.

curl -X POST http://localhost/predictions -H 'Content-Type: application/json' -d '{
    "inputs": { "prompt": "make me a sandwich" },
    "context": { "REPLICATE_API_TOKEN": "r8_xyz..." }
}'
def predict(prompt: str) -> str:
    # create a Replicate client using the token provided in the request
    replicate = Replicate()
    output = replicate.run("anthropic/claude-3.5-haiku", {input:
{"prompt": "prompt"}})
    return output

@philandstuff philandstuff force-pushed the support-cog-context branch from 5f07b02 to 724ff02 Compare May 27, 2025 11:02
aron and others added 6 commits May 27, 2025 12:14
This commit introduces support for the cog context into the Replicate
SDK. The `current_scope` helper now makes per-prediction context
available via the `current_scope().context` dict.

A cog model can then provide a REPLICATE_API_TOKEN on a per-prediction
basis to be used by the model.

    def predict(prompt: str) -> str:
        replicate = Replicate()
        output = replicate.run("anthropic/claude-3.5-haiku", {input:
{"prompt": "prompt"}})
        return output
Co-authored-by: Philip Potter <[email protected]>
@philandstuff philandstuff force-pushed the support-cog-context branch from 724ff02 to 855a596 Compare May 27, 2025 11:14
@philandstuff philandstuff merged commit 36c95a8 into main May 27, 2025
7 checks passed
@philandstuff philandstuff deleted the support-cog-context branch May 27, 2025 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants