Getting an exception while capturing an existing

[jainankit]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.13.0

Steps to Reproduce

We are using sentry-sdk 2.13.0 for python. We also alongside use "structlog-sentry ~= 2.1.0". We are seeing a large number of Sentry errors while capturing the exception stack-trace. It seems like there's an issue with capturing breadcrumbs, in what is being captured in timestamp:

...
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 106, in __call__
    response = self.app(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 86, in <lambda>
    lambda *a, **kw: old_app(self, *a, **kw),
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/opentelemetry/instrumentation/flask/__init__.py", line 390, in _wrapped_app
    result = wsgi_app(wrapped_app_environ, _start_response)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
    return self.app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1476, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 803, in handle_exception
    got_request_exception.send(self, _async_wrapper=self.ensure_sync, exception=e)
  File "/usr/local/lib/python3.12/site-packages/blinker/base.py", line 279, in send
    result = receiver(sender, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1730, in runner
    return sentry_patched_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 207, in _capture_exception
    sentry_sdk.capture_event(event, hint=hint)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/api.py", line 162, in capture_event
    return get_current_scope().capture_event(event, hint, scope=scope, **scope_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1139, in capture_event
    event_id = self.get_client().capture_event(event=event, hint=hint, scope=scope)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 745, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 453, in _prepare_event
    event_ = scope.apply_to_event(event, hint, self.options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 149, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1452, in apply_to_event
    self._apply_breadcrumbs_to_event(event, hint, options)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1309, in _apply_breadcrumbs_to_event
    event["breadcrumbs"]["values"].sort(key=lambda crumb: crumb["timestamp"])
TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 177, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 89, in sentry_patched_wsgi_app
    return middleware(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 113, in __call__
    reraise(*_capture_exception())
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 195, in _capture_exception
    sentry_sdk.capture_event(event, hint=hint)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/api.py", line 162, in capture_event
    return get_current_scope().capture_event(event, hint, scope=scope, **scope_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1139, in capture_event
    event_id = self.get_client().capture_event(event=event, hint=hint, scope=scope)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 745, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 453, in _prepare_event
    event_ = scope.apply_to_event(event, hint, self.options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 149, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1452, in apply_to_event
    self._apply_breadcrumbs_to_event(event, hint, options)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1309, in _apply_breadcrumbs_to_event
    event["breadcrumbs"]["values"].sort(key=lambda crumb: crumb["timestamp"])
TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'

Expected Result

No error is thrown while capturing the sentry exception

Actual Result

When this issue happens, the error is not captured in Sentry dashboard, so we are dropping events.

[jonas-debeuk]

Also seeing this, same versions of sentry and sentry-struct-logs, and error 🙈

[szokeasaurusrex]

@jainankit or @jonas-debeuk, could one of you please share a minimal reproduction? I am not very familiar with structlog or the structlog-sentry integration, since it is not an official Sentry integration.

[sentrivana]

Hey folks, the breadcrumb sorting code that the error originates from was added recently. If you revert to SDK version 2.10.0, you shouldn't see this anymore.

FWIW the structlog-sentry issue is here: kiwicom/structlog-sentry#159 But we should patch this in the SDK as well. I'll look into it.