Closed
Description
Go version
go version go1.22.5 linux/amd64
Output of go env in your module/workspace:
N/AWhat did you do?
I have published a small reproducer to this repository: https://github.com/uhthomas/go-unsolicited-http
The instructions are in the README, but for brevity:
Run the server:
code
main.go
package main
import (
"context"
"flag"
"fmt"
"io"
"log"
"net/http"
"sync/atomic"
)
type Server struct {
requestCount uint64
}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
requestID := atomic.AddUint64(&s.requestCount, 1)
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
res, err := http.Get("https://go.dev")
if err != nil {
panic(err)
}
n, err := io.Copy(w, res.Body)
fmt.Printf("%d: %d bytes written, err=%v\n", requestID, n, err)
}
func Main(ctx context.Context) error {
addr := flag.String("addr", ":8080", "address to listen on")
flag.Parse()
log.Println("listening on", *addr)
return http.ListenAndServe(*addr, &Server{})
}
func main() {
if err := Main(context.Background()); err != nil {
log.Fatal(err)
}
}❯ go run github.com/uhthomas/go-unsolicited-http/cmd/server@mainRun the client:
code
main.go
package main
import (
"context"
"fmt"
"log"
"net/http"
"time"
)
func Main(ctx context.Context) error {
for i := 0; ; i++ {
fmt.Println("attempt", i)
if _, err := http.Head("http://localhost:8080"); err != nil {
return fmt.Errorf("head: %w", err)
}
time.Sleep(200 * time.Millisecond)
}
}
func main() {
if err := Main(context.Background()); err != nil {
log.Fatal(err)
}
}❯ go run github.com/uhthomas/go-unsolicited-http/cmd/client@mainWhat did you see happen?
Server:
1: 54 bytes written, err=<nil>
2: 0 bytes written, err=readfrom tcp [::1]:8080->[::1]:48336: write tcp [::1]:8080->[::1]:48336: write: broken pipe
3: 54 bytes written, err=<nil>
4: 54 bytes written, err=<nil>
5: 54 bytes written, err=<nil>
6: 54 bytes written, err=<nil>
7: 54 bytes written, err=<nil>
8: 54 bytes written, err=<nil>
9: 54 bytes written, err=<nil>
10: 0 bytes written, err=readfrom tcp [::1]:8080->[::1]:33742: write tcp [::1]:8080->[::1]:33742: write: broken pipeClient:
attempt 0
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n <head>\n <title>Thomas</tit"; err=<nil>
attempt 1
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n <head>\n <title>Thomas</tit"; err=<nil>
attempt 2
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n <head>\n <title>Thomas</tit"; err=<nil>
attempt 3
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n <head>\n <title>Thomas</tit"; err=<nil>
attempt 4
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n <head>\n <title>Thomas</tit"; err=<nil>
attempt 5
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n <head>\n <title>Thomas</tit"; err=<nil>
attempt 6
attempt 7
2024/07/26 16:17:18 do: Head "http://localhost:8080": net/http: HTTP/1.x transport connection broken: malformed HTTP status code "html>"
exit status 1What did you expect to see?
There should be no Unsolicited response received on idle HTTP channel starting with messages, and the request should not fail with net/http: HTTP/1.x transport connection broken: malformed HTTP status code "html>".
Metadata
Metadata
Assignees
Type
Projects
Relationships
Development
No branches or pull requests
Participants
Activity
gabyhelp commentedon Jul 26, 2024
Related Issues and Documentation
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
uhthomas commentedon Jul 26, 2024
#19895 is one of the only matches for "Unsolicited response received on idle HTTP channel starting with" and claims the HTTP server to be misbehaving.
seankhliao commentedon Jul 26, 2024
Your server is indeed misbehaving by writing data when it receives a
HEADrequest.uhthomas commentedon Jul 26, 2024
Sure, but it should not cause this sort of behavior at all. It's quite problematic, and inconsistent. Under many circumstances, there are no issues with writing a response to a HEAD request, even if it's wrong. It should not affect other connections to the server, which it does, and it should not leak data between connections, which it does. Please can this be reopened and considered more fairly?
mvdan commentedon Jul 26, 2024
@seankhliao where is that documented in the net/http docs? A quick skim doesn't show me anything, even if HEAD responses not containing a body could be common knowledge.
Assuming you're right about that assumption, I would imagine that calling Write on https://pkg.go.dev/net/http#ResponseWriter for a HEAD request should result in either a clear error or panic, and not anything like corruption or data races or confusing errors.
terinjokes commentedon Jul 26, 2024
To me the issue here looks like a Go HTTP client receiving a body in response to an HEAD request, and then the established HTTP/1.1 connection was reused for another request, where it tried to read the previously sent body as the start of the new response. This could happen regardless of the server implementation.
RFC 9110 section 9.3.2. recommends clients close and error when a HEAD response is received with a body to avoid request/response smuggling. Is that something we can consider here?
[-]net/http: data is written to idle connections[/-][+]net/http: Transport should discard connections from HEAD requests with a body[/+]seankhliao commentedon Jul 26, 2024
looks like the client (transport) assumes a HEAD request never has a body https://go.googlesource.com/go/+/3959d54c0bd5c92fe0a5e33fedb0595723efc23b/src/net/http/transport.go#2245
cc @neild
seankhliao commentedon Jul 26, 2024
I suppose #62015 would be the corresponding issue for discarding the body from the server side
uhthomas commentedon Jul 26, 2024
Thanks for the additional input, and thank you for reopening.
Something I'd really like to understand is why this doesn't happen when writing data in-memory? I could only ever reproduce this when making a HTTP request on the server and writing it back to the client.
neild commentedon Jul 26, 2024
Some of the above is talking about a body in a HEAD request, and some is talking about a body in the response to a HEAD request. These are completely different things.
HEAD requests may contain a body, but the body has no defined semantics. (RFC 9110, section 9.3.2)
The response to a HEAD request does not contain a body. RFC 9112, section 6.3:
Improperly writing a body leaves a connection in an invalid state, since the client is expecting the next bytes read to be the status line for the next request on the connection.
Thanks for the report, @uhthomas. This is a bug in net/http. The problem is that while an
http.ResponseWriternormally discards the body for HEAD requests,ResponseWriter.ReadFromfails to do so. Triggering this bug requires either callingReadFromdirectly, or usingio.Copyto copy to aResponseWriterfrom a source with noWriteTomethod.uhthomas commentedon Jul 26, 2024
@neild Ohh, that answers my previous question, thank you so much for the insight.
22 remaining items