Skip to content

Support runai model streamer for fast model loading #352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
1 of 3 tasks
kerthcet opened this issue Apr 18, 2025 · 5 comments · May be fixed by #423
Open
1 of 3 tasks

Support runai model streamer for fast model loading #352

kerthcet opened this issue Apr 18, 2025 · 5 comments · May be fixed by #423
Assignees
@cr7258
Labels
feature Categorizes issue or PR as related to a new feature. needs-kind Indicates a PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.
Milestone
v0.2.0

Comments

@kerthcet
Copy link
Member

kerthcet commented Apr 18, 2025 •
edited
Loading

What would you like to be added:

https://github.com/run-ai/runai-model-streamer is a library helps to load tensors from source to the GPU memory directly, we may integrate with this project to accelerate model loading from s3 like object store.

part of #119

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

  • Design doc
  • API change
  • Docs update

The artifacts should be linked in subsequent comments.
@kerthcet kerthcet added the feature Categorizes issue or PR as related to a new feature. label Apr 18, 2025
@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-kind Indicates a PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. labels Apr 18, 2025
@kerthcet kerthcet mentioned this issue Apr 18, 2025
Open
3 tasks
@kerthcet
Copy link
Member Author

kerthcet commented Apr 18, 2025 •
edited
Loading

However, we need to find out:

@kerthcet
Copy link
Member Author

/milestone v0.2.0

@InftyAI-Agent InftyAI-Agent added this to the v0.2.0 milestone Apr 23, 2025
@cr7258
Copy link
Contributor

cr7258 commented May 14, 2025

/assign

@kerthcet
Copy link
Member Author

Any progress on this? Hope to have a conclusion on this this week since the deadline is this month. :)

@cr7258 cr7258 linked a pull request May 19, 2025 that will close this issue
Open
@cr7258
Copy link
Contributor

cr7258 commented May 19, 2025

The PR has been created: 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cr7258 cr7258

Labels
feature Categorizes issue or PR as related to a new feature. needs-kind Indicates a PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.
Projects
None yet
Milestone
v0.2.0
Development

Successfully merging a pull request may close this issue.

feat: support runai streamer for vllm cr7258/llmaz
3 participants
@kerthcet @cr7258 @InftyAI-Agent