fix: remove broken tracing middleware

[iamemilio]

What does this PR do?

Removes the broken tracing middleware from llama stack core. This middleware duplicates what otel already does for fast api by default, but breaks tracing by incorrectly handling w3 trace headers.

Test Plan

Telemetry is currently not working. An attempt to run this by hand was made, but this is not the only thing that needs to change to make the telemetry work in this project, so traces did not show up. More changes to come to address this.

[leseb]

Ok, we want to remove the broken tracing middleware. Can you clarify with what we should replace it? Can you explain how do you intend to split your work and PRs that will follow?

Thanks!

[iamemilio]

Ok, we want to remove the broken tracing middleware. Can you clarify with what we should replace it? Can you explain how do you intend to split your work and PRs that will follow?

Thanks!
@leseb Thanks for the review!

Yeah, I am trying to find a way to make this change that makes sense, but its kinda a headache. The middleware we have right now interferes with other tracing. Would you prefer that I just replace all the tracing all at once?

I did a lot of testing, and discovered that we can use the auto instrumentation, but we need to do it programmatically due to a known quirk of using otel with uvicorn. This would mean that we would need telemetry installed and enabled by default, but we can disable it with environment variables. I am beginning to stage those WIP changes here: #3733

How do we feel about this design pattern? I made this comment in community the discord as well, I am happy to link you.

My goal with this PR is to make the telemetry we have work well enough. Then we can migrate services to the new pattern we want one service at a time. Once that is done, we can deprecate the telemetry API.

Once we merge this, and I finish implementing what is in the next PR, I can file tickets upstream for each place we capture custom instrumentation, and let you all help me with the migration. Its also an opportunity to go over what we capture with scrutiny to make sure what custom info we capture makes sense and isn't duplicated elsewhere.

[iamemilio]

@ehhuang take a look at this. I was able to get the integration test to work by doing this, but I am not 100% sure its right. I'd appreciate if you took a look and confirmed.

[ehhuang]

I don't either. Can we just kill this sqlite span processor alltogether and add tests analogous to those in test_*_telemetry but against OTEL?

[cdoern]

this is looking good, but +1 on leaving out unrelated changes.

[iamemilio]

Here is an example distributed trace with the changes in this PR from a client that was also instrumented sending a chat completion request to llama stack.

telemetry config:

  telemetry:
  - provider_id: meta-reference
    provider_type: inline::meta-reference
    config:
      service_name: llama-stack-server
      otel_exporter_otlp_endpoint: http://localhost:4318
      sinks:
        - console
        - otel_metric
        - otel_trace

[ehhuang]

did we lose some spans with these changes?