handle cookies on redirection manually

[krichprollsch]

This PR fixes the cookies handling on redirection.

The original problem was:

It seems to work correctly except one case: if you set manually a cookie (A=A) and the server sets one with the same name (A=B) and it redirects, then the redirect request will send the cookie twice with the 2 values (eg. Cookie: A=A; A=B).

This PR de-activate the Curl's cookie engine.
We now set cookies manually in redirection to ensure only once Cookie: is set.

relates with #940

[krichprollsch]

I'm not sure about using this allocator...

[karlseguin]

Yes. I've had the need for a transfer.arena a few times before, but I've always managed to work around it.

I'm not opposed to adding one. There's code like https://github.com/lightpanda-io/browser/pull/946/files#diff-897697f048d76304958bca64b03d999c1b5f27c29a1e6d23d9f9a96386bd3b43R261 that could use it

[krichprollsch]

I added an arena to Transfer + I removed the arena from the Client which was useless AFAIK

[krichprollsch]

Using lightpanda-io/demo#57 test file

on main the redirection URL sends:

> GET /cookies HTTP/1.1
Host: httpbin.io
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: manual=B; manual=A
User-Agent: Lightpanda/1.0

Which have manual cookie set twice.

This branch sends correctly

> GET /cookies HTTP/1.1
Host: httpbin.io
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: manual=B
User-Agent: Lightpanda/1.0

new behavior

> GET /cookies/set?manual=B HTTP/1.1
Host: httpbin.io
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: manual=A
User-Agent: Lightpanda/1.0

* Request completely sent off
< HTTP/1.1 302 Found
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Location: /cookies
< Set-Cookie: manual=B; HttpOnly
< Date: Mon, 18 Aug 2025 10:16:31 GMT
< Content-Length: 0
* Ignoring the response-body
* setting size while ignoring
<
* Connection #0 to host httpbin.io:443 left intact
* Issue another request to this URL: 'https://httpbin.io/cookies'
* Reusing existing https: connection with host httpbin.io
> GET /cookies HTTP/1.1
Host: httpbin.io
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: manual=B
User-Agent: Lightpanda/1.0

* Request completely sent off
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Date: Mon, 18 Aug 2025 10:16:31 GMT
< Content-Length: 20
<
* Connection #0 to host httpbin.io:443 left intact

previous behavior

> GET /cookies/set?manual=B HTTP/1.1
Host: httpbin.io
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: manual=A
User-Agent: Lightpanda/1.0

* Request completely sent off
< HTTP/1.1 302 Found
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Location: /cookies
* Added cookie manual="B" for domain httpbin.io, path /cookies/, expire 0
< Set-Cookie: manual=B; HttpOnly
< Date: Mon, 18 Aug 2025 10:15:06 GMT
< Content-Length: 0
* Ignoring the response-body
* setting size while ignoring
<
* Connection #0 to host httpbin.io:443 left intact
* Issue another request to this URL: 'https://httpbin.io/cookies'
* Reusing existing https: connection with host httpbin.io
> GET /cookies HTTP/1.1
Host: httpbin.io
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: manual=B; manual=A
User-Agent: Lightpanda/1.0

* Request completely sent off
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Date: Mon, 18 Aug 2025 10:15:06 GMT
< Content-Length: 20
<
* Connection #0 to host httpbin.io:443 left intact

[karlseguin]

In the Request Interception PR (#946), the Transfer.Request now has a resource_type enum. I think you would want to to is_navigation = req.resource_type == .document, because I think it should be false for scripts and xhr requests (the other 2 types of resource_types we currently have)

[karlseguin]

Yes. I've had the need for a transfer.arena a few times before, but I've always managed to work around it.

I'm not opposed to adding one. There's code like https://github.com/lightpanda-io/browser/pull/946/files#diff-897697f048d76304958bca64b03d999c1b5f27c29a1e6d23d9f9a96386bd3b43R261 that could use it

[karlseguin]

LGTM.

Things to do in RI PR after this is merged:
1 - use the transfer.arena from this PR in CDP where we need to dupe/own various CDP inputs (and possibly other places, need to re-check if ScriptManager, PAge and XHR do any allocations that might better belong to the Transfer's arena).
2 - use the new request.resource_type from that PR to set is_navigation