Skip to content

[API server] uvicorn worker crash #5401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
aylei opened this issue Apr 27, 2025 · 0 comments
Open

[API server] uvicorn worker crash #5401

aylei opened this issue Apr 27, 2025 · 0 comments
Assignees
@aylei

Comments

@aylei
Copy link
Collaborator

aylei commented Apr 27, 2025 

E 04-27 13:23:32 httptools_impl.py:414] Exception in ASGI application
E 04-27 13:23:32 httptools_impl.py:414]   + Exception Group Traceback (most recent call last):
E 04-27 13:23:32 httptools_impl.py:414]   |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
E 04-27 13:23:32 httptools_impl.py:414]   |     yield
E 04-27 13:23:32 httptools_impl.py:414]   |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 263, in __call__
E 04-27 13:23:32 httptools_impl.py:414]   |     async with anyio.create_task_group() as task_group:
E 04-27 13:23:32 httptools_impl.py:414]   |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
E 04-27 13:23:32 httptools_impl.py:414]   |     raise BaseExceptionGroup(
E 04-27 13:23:32 httptools_impl.py:414]   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
E 04-27 13:23:32 httptools_impl.py:414]   +-+---------------- 1 ----------------
E 04-27 13:23:32 httptools_impl.py:414]     | Traceback (most recent call last):
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
E 04-27 13:23:32 httptools_impl.py:414]     |     result = await app(  # type: ignore[func-returns-value]
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     return await self.app(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     await super().__call__(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.middleware_stack(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     raise exc
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.app(scope, receive, _send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 183, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     raise app_exc
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.app(scope, receive_or_disconnect, send_no_error)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.app(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 183, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     raise app_exc
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.app(scope, receive_or_disconnect, send_no_error)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     |     raise exc
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     |     await app(scope, receive, sender)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 714, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.middleware_stack(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 734, in app
E 04-27 13:23:32 httptools_impl.py:414]     |     await route.handle(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
E 04-27 13:23:32 httptools_impl.py:414]     |     await self.app(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
E 04-27 13:23:32 httptools_impl.py:414]     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     |     raise exc
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     |     await app(scope, receive, sender)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
E 04-27 13:23:32 httptools_impl.py:414]     |     await response(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 262, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     |     with collapse_excgroups():
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/contextlib.py", line 153, in __exit__
E 04-27 13:23:32 httptools_impl.py:414]     |     self.gen.throw(typ, value, traceback)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
E 04-27 13:23:32 httptools_impl.py:414]     |     raise exc
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 266, in wrap
E 04-27 13:23:32 httptools_impl.py:414]     |     await func()
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 246, in stream_response
E 04-27 13:23:32 httptools_impl.py:414]     |     async for chunk in self.body_iterator:
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/skypilot/sky/server/stream_utils.py", line 82, in log_streamer
E 04-27 13:23:32 httptools_impl.py:414]     |     async with aiofiles.open(log_path, 'rb') as f:
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/aiofiles/base.py", line 63, in __aenter__
E 04-27 13:23:32 httptools_impl.py:414]     |     return await self
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/aiofiles/base.py", line 59, in __await__
E 04-27 13:23:32 httptools_impl.py:414]     |     self._obj = yield from self._coro.__await__()
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/aiofiles/threadpool/__init__.py", line 92, in _open
E 04-27 13:23:32 httptools_impl.py:414]     |     f = await loop.run_in_executor(executor, cb)
E 04-27 13:23:32 httptools_impl.py:414]     |   File "/home/ubuntu/miniconda3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
E 04-27 13:23:32 httptools_impl.py:414]     |     result = self.fn(*self.args, **self.kwargs)
E 04-27 13:23:32 httptools_impl.py:414]     | FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/sky_logs/api_server/requests/cffae6c3-7840-4eba-95ea-c0b967673943.log'
E 04-27 13:23:32 httptools_impl.py:414]     +------------------------------------
E 04-27 13:23:32 httptools_impl.py:414]
E 04-27 13:23:32 httptools_impl.py:414] During handling of the above exception, another exception occurred:
E 04-27 13:23:32 httptools_impl.py:414]
E 04-27 13:23:32 httptools_impl.py:414] Traceback (most recent call last):
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
E 04-27 13:23:32 httptools_impl.py:414]     result = await app(  # type: ignore[func-returns-value]
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     return await self.app(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     await super().__call__(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     await self.middleware_stack(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     raise exc
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     await self.app(scope, receive, _send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 183, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     raise app_exc
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
E 04-27 13:23:32 httptools_impl.py:414]     await self.app(scope, receive_or_disconnect, send_no_error)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     await self.app(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 183, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     raise app_exc
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
E 04-27 13:23:32 httptools_impl.py:414]     await self.app(scope, receive_or_disconnect, send_no_error)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     raise exc
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     await app(scope, receive, sender)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 714, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     await self.middleware_stack(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 734, in app
E 04-27 13:23:32 httptools_impl.py:414]     await route.handle(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
E 04-27 13:23:32 httptools_impl.py:414]     await self.app(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
E 04-27 13:23:32 httptools_impl.py:414]     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     raise exc
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
E 04-27 13:23:32 httptools_impl.py:414]     await app(scope, receive, sender)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
E 04-27 13:23:32 httptools_impl.py:414]     await response(scope, receive, send)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 262, in __call__
E 04-27 13:23:32 httptools_impl.py:414]     with collapse_excgroups():
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/contextlib.py", line 153, in __exit__
E 04-27 13:23:32 httptools_impl.py:414]     self.gen.throw(typ, value, traceback)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
E 04-27 13:23:32 httptools_impl.py:414]     raise exc
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 266, in wrap
E 04-27 13:23:32 httptools_impl.py:414]     await func()
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/starlette/responses.py", line 246, in stream_response
E 04-27 13:23:32 httptools_impl.py:414]     async for chunk in self.body_iterator:
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/skypilot/sky/server/stream_utils.py", line 82, in log_streamer
E 04-27 13:23:32 httptools_impl.py:414]     async with aiofiles.open(log_path, 'rb') as f:
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/aiofiles/base.py", line 63, in __aenter__
E 04-27 13:23:32 httptools_impl.py:414]     return await self
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/aiofiles/base.py", line 59, in __await__
E 04-27 13:23:32 httptools_impl.py:414]     self._obj = yield from self._coro.__await__()
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/aiofiles/threadpool/__init__.py", line 92, in _open
E 04-27 13:23:32 httptools_impl.py:414]     f = await loop.run_in_executor(executor, cb)
E 04-27 13:23:32 httptools_impl.py:414]   File "/home/ubuntu/miniconda3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
E 04-27 13:23:32 httptools_impl.py:414]     result = self.fn(*self.args, **self.kwargs)
E 04-27 13:23:32 httptools_impl.py:414] FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/sky_logs/api_server/requests/cffae6c3-7840-4eba-95ea-c0b967673943.log'
@aylei aylei self-assigned this Apr 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aylei aylei

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aylei