Existing HTTP/3 connections become unusable during config update

[timmclean]

When a new config is pushed to caddy using POST http://localhost:2019, clients with existing HTTP/3 connections to the caddy server lose the ability to connect for a short period of time, causing active users to experience downtime.

I was able to reproduce this with a clean install on a new EC2 instance:

1. Create a t4g.nano instance on EC2 with Ubuntu 24.04 (arm64).
2. Open TCP ports 22, 80, 443 and UDP port 443.
3. Install Caddy v2.9.1 for Linux arm64
4. Create a config.json file on the server (replace TESTDOMAIN with a domain you control):

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [ ":443" ],
          "routes": [
            {
              "match": [ { "host": [ "TESTDOMAIN" ] } ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "group": "group1",
                      "handle": [
                        {
                          "body": "hello world",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [ "TESTDOMAIN" ]
          }
        ]
      }
    }
  }
}

5. Add an A record for TESTDOMAIN that matches the public IPv4 address of the EC2 instance.
6. Run curl -d @config.json -H 'content-type: application/json' http://localhost:2019/load on the server
7. Go to https://TESTDOMAIN in Google Chrome (tested with both Chrome for Android and desktop).
8. "hello world" should be shown.
9. Refresh the page a bunch. It should load just fine.
10. Edit config.json, changing "hello world" to "hello world 2".
11. Run sleep 3 && curl -d @config.json -H 'content-type: application/json' http://localhost:2019/load on the server
12. Quickly switch to Google Chrome and start refreshing the page rapidly. The refreshes will work until sleep 3 finishes, at which point the page will hang and refuse to load.
13. Navigate to https://TESTDOMAIN from a different browser and observe that the page does in fact load.
14. Refresh Google Chrome again, and observe that the page still does not load.
15. Wait 1-2 minutes.
16. Refresh Google Chrome again. The page will now load correctly.

Blocking UDP port 443 seems to fix this problem.

Log output during config change:

{"level":"info","ts":1742505571.2683778,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"40602","headers":{"Accept":["*/*"],"Content-Length":["885"],"Content-Type":["application/json"],"User-Agent":["curl/8.5.0"]}}
{"level":"info","ts":1742505571.2690651,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1742505571.269525,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1742505571.269545,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1742505571.2697465,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"warn","ts":1742505571.2697577,"msg":"quic listener tls configs are more than 2","number of configs":3}
{"level":"info","ts":1742505571.2697651,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1742505571.269796,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1742505571.2698002,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1742505571.269803,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1742505571.2698061,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["caddybugtest.timmclean.net"]}
{"level":"info","ts":1742505571.2698162,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1742505571.2710686,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
{"level":"info","ts":1742505571.2711222,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1742505571.2733417,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

[mholt]

Thanks for the instructions; I will give it a try soon... in the meantime, does this only happen with Chrome?

(What about curl with http/3, or Firefox?)

[timmclean]

I just tried Firefox and the behaviour there is a bit better. When the config update goes through, it seems to hesitate for a moment and then fallback to HTTP/2 transparently. When I check the requests in the Network tab, I can see that they are HTTP/3 before the config change, and then HTTP/2 after the config change. I'm guessing the hesitation is because Firefox is setting up a new connection.

I haven't observed any issues with curl --http3-only so far in my testing. It recreates connections every time, so that makes sense to me.

Client versions (desktop):

• Chrome 134.0.6998.117
• Firefox 136.0.2
• curl 8.12.1 musl

[crrodriguez]

Chrome stable http3 support is spotty.. I have achieved better results with chrome-dev where it seems to be more consistent/stable.

[jph00]

We're seeing this issue as well with Chrome and also Arc (which is Chromium-based IIRC).

We'll just turn off http 3 in caddy for now -- is that the recommended workaround?

[mholt]

That should work around it by avoiding the glitchy code paths, yeah. I haven't had a chance yet to dig into this, but it does seem odd that it's mainly Chrome-only.

[jph00]

That should work around it by avoiding the glitchy code paths, yeah. I haven't had a chance yet to dig into this, but it does seem odd that it's mainly Chrome-only.

We also found that changing "experimental quic protocol" from "default" to "disabled" in Chrome fixes the problem btw. (It surprised me that it was enabled by default in the release version of Chrome, despite being experimental.)