Skip to content

fix: limit Support for HTTP Body limit in axum server #2581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 21, 2025
Merged

fix: limit Support for HTTP Body limit in axum server #2581

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
61a7d94
body limit
michaelfeil Aug 20, 2025
d0fa8e6
body limit
michaelfeil Aug 20, 2025
8234cbd
Update lib/llm/src/http/service/openai.rs
michaelfeil Aug 21, 2025
08854b4
as env var at runtime
michaelfeil Aug 21, 2025
193e808
Merge branch 'main' into mf/max-limit
michaelfeil Aug 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions lib/llm/src/http/service/openai.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,17 @@ pub const DYNAMO_REQUEST_ID_HEADER: &str = "x-dynamo-request-id";
/// Dynamo Annotation for the request ID
pub const ANNOTATION_REQUEST_ID: &str = "request_id";

// Default axum max body limit without configuring is 2MB: https://docs.rs/axum/latest/axum/extract/struct.DefaultBodyLimit.html
/// Default body limit in bytes (45MB) to support 500k+ token payloads.
/// Can be configured at compile time using the DYN_FRONTEND_BODY_LIMIT_MB environment variable
fn get_body_limit() -> usize {
std::env::var("DYN_HTTP_BODY_LIMIT_MB")
.ok()
.and_then(|s| s.parse::<usize>().ok())
.map(|mb| mb * 1024 * 1024)
.unwrap_or(45 * 1024 * 1024)
}

pub type ErrorResponse = (StatusCode, Json<ErrorMessage>);

#[derive(Serialize, Deserialize)]
Expand Down Expand Up @@ -1002,6 +1013,7 @@ pub fn completions_router(
let doc = RouteDoc::new(axum::http::Method::POST, &path);
let router = Router::new()
.route(&path, post(handler_completions))
.layer(axum::extract::DefaultBodyLimit::max(get_body_limit()))
.with_state(state);
(vec![doc], router)
}
Expand All @@ -1017,6 +1029,7 @@ pub fn chat_completions_router(
let doc = RouteDoc::new(axum::http::Method::POST, &path);
let router = Router::new()
.route(&path, post(handler_chat_completions))
.layer(axum::extract::DefaultBodyLimit::max(get_body_limit()))
.with_state((state, template));
(vec![doc], router)
}
Expand All @@ -1031,6 +1044,7 @@ pub fn embeddings_router(
let doc = RouteDoc::new(axum::http::Method::POST, &path);
let router = Router::new()
.route(&path, post(embeddings))
.layer(axum::extract::DefaultBodyLimit::max(get_body_limit()))
.with_state(state);
(vec![doc], router)
}
Expand Down
Loading