Investigate logging streaming HTTP responses in DevToolsPlugin

[waldekmastykarz]

Background

Currently, when an intercepted HTTP request returns a streaming response, it appears in Chrome DevTools (via the DevToolsPlugin) but no response body is shown. However, Dev Proxy is correctly intercepting these responses, as evidenced by mocks generated using the MockGeneratorPlugin.

Investigation

• Investigate how streaming HTTP responses are currently handled in the DevToolsPlugin when reporting/interacting with the browser's DevTools via the DevTools Protocol.
• Verify how the MockGeneratorPlugin is able to capture and display streaming responses.
• Research if there are additional or alternative methods in the DevTools Protocol that need to be used to properly log or display streaming responses in the DevTools UI.
• Document findings and propose code changes or protocol adjustments if required to enable proper logging of streaming HTTP responses.

Acceptance Criteria

• The investigation should clarify why streaming responses are not displayed in DevTools, despite being intercepted elsewhere.
• Suggestions or a plan for next steps to resolve the issue, if possible.

[pelikhan]

you could probably build a smaller repro with the openai node.js package and streaming enabled.

[waldekmastykarz]

I've got a mini repro with curl 😄

[bartizan]

Started an investigation into the issue. Here are the preliminary findings:

• DevProxy Streaming Response Behavior. DevProxy currently doesn't work transparently for streaming responses. The client doesn't receive the same response chunks as DevProxy, and it might receive nothing until the entire response is finished. This could be an issue in certain scenarios, so we might want to address it. The problem could be related to buffering configuration.

• DevToolsPlugin Functionality. DevToolsPlugin actually worked for me, successfully intercepting streaming responses (tested on Edge/Chrome). There's a trick to it, however: don't open the Response tab of the selected running request within DevTools > Network. The response will appear there once the request is completed. This behavior still needs to be investigated further.

[waldekmastykarz]

• DevProxy Streaming Response Behavior. DevProxy currently doesn't work transparently for streaming responses. The client doesn't receive the same response chunks as DevProxy, and it might receive nothing until the entire response is finished. This could be an issue in certain scenarios, so we might want to address it. The problem could be related to buffering configuration.

Correct, and this might be a bigger issue. Right now, we've got AfterResponse event which occurs after the whole response from the API was sent, but this isn't the case for streaming, so with Dev Proxy in between, Dev Proxy buffers the responses until the end and sends them in one go to the client. I suspect, that the proxy engine we're using, doesn't expose streaming and we might need to change it there if we wanted to support it. As long as we can have streamed responses appear in DevTools, I see support for streaming as a nice to have.

• DevToolsPlugin Functionality. DevToolsPlugin actually worked for me, successfully intercepting streaming responses (tested on Edge/Chrome). There's a trick to it, however: don't open the Response tab of the selected running request within DevTools > Network. The response will appear there once the request is completed. This behavior still needs to be investigated further.

Interesting. I'll need to give it another go then. For me, all requests lacked responses.

I appreciate you looking into it @bartizan

[waldekmastykarz]

How are you testing it @bartizan?

If I start Dev Proxy and run in console:

cucurl -ikx http://127.0.0.1:8000 'http://127.0.0.1:11434/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer ollama' \
  -H 'User-Agent: genaiscript' \
  -H 'Accept: */*' \
  -H 'Proxy-Connection: close' \
  -H 'Connection: close' \
  --data-raw $'{"model":"llama3.2","stream":true,"messages":[{"role":"user","content":"Why is the sky blue?"}],"stream_options":{"include_usage":true}}'

In DevTools I'm seeing the request but both Response and Event Stream are empty:

It does show the total size of the response and timing:

[bartizan]

@waldekmastykarz Thank you, I reproduced the issue.
My conclusion was based on a test server that provided Streams API of Web APIs while LLMs (openAI spec) implement Server-sent Events (SSE) of WebAPIs.

[waldekmastykarz]

I found the issue: it turns out we haven't sent the necessary information for streamed responses. Fix coming.