Django cache_spans cause excessive overhead

[sl0thentr0py]

We noticed a drastic uptick in latency for a few key endpoints that utilize the cache (redis) starting on May 8th. After looking through what we deployed we noticed an update to sentry-sdk from 1.21.1 to 1.22.1. After looking at the change log the "add some overhead to your server" comment stood out and so we reverted this upgrade yesterday and the endpoint normalized.

This is our sentry init if it helps:

sentry.init(
        before_send=event_filter,
        dsn=settings.SENTRY_DSN if get_environment() in ['production', 'staging'] else '',
        environment=get_environment(),
        integrations=[CeleryIntegration(), DjangoIntegration(), RedisIntegration(), SqlalchemyIntegration()],
        release=get_release(),
        send_default_pii=True,
    )

Originally posted by @dvprz in #2116 (comment)

[jarekwg]

This knocked out our servers, causing a severe outage across many customers.
We had to roll back our latest release as it took a while to diagnose that the CPU shredding was a result of bumping sentry.
The irony of having an error monitoring framework causing a major outage....
Would it be possible to mark releases as beta or something when they're introducing this level of risk?

[HazAT]

@jarekwg you are entirely correct, and we apologize that this happened.
We already had an internal conversation about making sure we roll out changes similar to this with more care.

[antonpirker]

We added a parameter cache_spans to init() that defaults to False. See the documentation. This resolves this issue.