curl error with cabal upload -d (regression in 3.14)

[andreasabel]

It looks like uploading documentation got broken:

$ cabal-3.12.1.0 upload -d /Users/abel/bin/src/test-framework/dist-newstyle/test-framework-0.8.2.2-docs.tar.gz
Uploading documentation
/Users/abel/bin/src/test-framework/dist-newstyle/test-framework-0.8.2.2-docs.tar.gz...
Documentation successfully uploaded for package candidate...

$  cabal-3.14.2.0 upload -d /Users/abel/bin/src/test-framework/dist-newstyle/test-framework-0.8.2.2-docs.tar.gz
Uploading documentation
/Users/abel/bin/src/test-framework/dist-newstyle/test-framework-0.8.2.2-docs.tar.gz...
Error: cabal-3.14.2.0: '/usr/bin/curl' exited with an error:
curl: (56) chunk hex-length char not a hex digit: 0xd

$  curl --version                                                                            
curl 8.7.1 (x86_64-apple-darwin24.0) libcurl/8.7.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.64.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe UnixSockets

(I also tried cabal-3.14.1.0/1, same problem.)

curl: (56) chunk hex-length char not a hex digit: 0xd

[andreasabel]

Recent PR that could be related:

• cabal-install: update curl transport to support Basic authentication #10089

[andreasabel]

Is there a workaround for this issue?

[ulysses4ever]

Does this look helpful?

#10920

[ulysses4ever]

Sorry, wrong link. Here you go:

• cabal upload throwing curl errors #10252

[Bodigrim]

Is there a workaround for this issue?

Does cabal --http-transport=plain-http upload -d docs.tar.gz --publish work for you?

[andreasabel]

@Bodigrim : Thanks, this works indeed.

@ulysses4ever : Thanks for linking to #10252 . It shows the same symptoms, but it is reported against 3.12, whereas I only see the failure in 3.14.

[andreasabel]

I can confirm again that I am suffering from a regression in 3.14:

Uploading documentation
/Users/abel/bin/src/hackage-security/dist-newstyle/hackage-security-0.6.3.0-docs.tar.gz...
Running: /usr/bin/curl --config - 'https://hackage.haskell.org/package/hackage-security-0.6.3.0/candidate/docs' --request PUT --data-binary '@/Users/abel/bin/src/hackage-security/dist-newstyle/hackage-security-0.6.3.0-docs.tar.gz' --write-out '
%{http_code}' --user-agent 'cabal-install/3.14.2.0 (osx; aarch64)' --silent --show-error --location --header 'Accept: text/plain' --header 'Content-Type: application/x-tar' --header 'Content-Encoding: gzip'
...
curl: (56) chunk hex-length char not a hex digit: 0xd

stdin input:
--anyauth
--user AndreasAbel:PASSWORD
CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:447:12 in Cabal-3.14.2.0-inplace:Distribution.Simple.Utils

The -v3 for 3.12 is:

...
Running: /usr/bin/curl --config - 'https://hackage.haskell.org/package/hackage-security-0.6.3.0/candidate/docs' --request PUT --data-binary '@/Users/abel/bin/src/hackage-security/dist-newstyle/hackage-security-0.6.3.0-docs.tar.gz' --write-out '
%{http_code}' --user-agent 'cabal-install/3.12.1.0 (osx; aarch64)' --silent --show-error --location --header 'Accept: text/plain' --header 'Content-Type: application/x-tar' --header 'Content-Encoding: gzip'
...
Documentation successfully uploaded for package candidate. You can now preview
the result at
'https://hackage.haskell.org/package/hackage-security-0.6.3.0/candidate'.

[andreasabel]

I now made the experiment of reverting #10089 and it indeed fixes the regression for me.

Running: /usr/bin/curl --config - 'https://hackage.haskell.org/package/hackage-security-0.6.3.0/candidate/docs' --request PUT --data-binary '@/Users/abel/bin/src/hackage-security/dist-newstyle/hackage-security-0.6.3.0-docs.tar.gz' --write-out '
%{http_code}' --user-agent 'cabal-install/3.15.0.0 (osx; aarch64)' --silent --show-error --location --header 'Accept: text/plain' --header 'Content-Type: application/x-tar' --header 'Content-Encoding: gzip'

(However, in the light of #10252 there seems to be an underlying problem and likely #10089 just exposed it for me.)

[andreasabel]

Hackage server issue:

• Is curl --anyauth supported properly when uploading documentation with cabal? hackage-server#1389

[hasufell]

Is there a workaround for this issue?

cabal --http-transport=wget upload works for me.

cabal upload is really in an absolutely disgraceful state.

hackage doc builders are broken and now I have to jump through hoops to be able to upload documentation for core libraries.

[alt-romes]

I was bit by this just now as well.

[MangoIV]

There's another workaround - the hackage web interface allows for directly uploading tarballs from your disk.

[mpickering]

@hasufell Could you please refrain from using such hyperbolic language. All software has bugs, and this is one of many examples.

To summarise, --http-transport=plain-http can be passed to workaround the issue.

• It is reported that reverting cabal-install: update curl transport to support Basic authentication #10089 works for some users but cabal upload throwing curl errors #10252 seems similar (and reproducible with earlier cabal versions)
• It seems that @andreasabel suggests that the server might be misconfigured in ghc-lib-parser fails to build with -j1 #10963
• @gbaz suggests that the problem may be in happstack (cabal upload throwing curl errors #10252 (comment))

I am looking into how to fix the issue.

[hasufell]

Could you please refrain from using such hyperbolic language.

The issues with cabal upload affect many users. They are very visible. They started surfacing almost a year ago and now also affect maintainers of core libraries negatively, where the absence of documentation is a major problem.

I think my language simply reflects the state of the user experience. This was not a statement related to volunteers and their efforts. We need to be able to express matters as they are and I will continue to call them out. I think we should all be able to differentiate between technology and people.

I am looking into how to fix the issue.

Terrific!

[mpickering]

I have looked into this and it appears to be an issue with the nginx configuration. @bgamari is briefing the infrastructure team.

@hasufell Thanks for your thoughtful reply, I completely agree it's an important issue to fix, but that cause isn't helped with hyperbole and exaggeration. Your message shows me it is possible to express the same sentiment but in a more sensitive way.

I think there are two sides to the same coin. One side asks why is something so broken, the other side asks what can be done to make things better. Both people are motivated by the desire to improve a situation but the latter seems more positive to me.

[mpickering]

What I have tried:

• Uploading documentation to hackage.haskell.org, fails
• Uploading documentation to hackage-origin.haskell.org fails
• Uploading documentation to a local hackage instance (works)

You also need to use an upload which is quite "big", my documentation was 300kb, but it worked with --data 'test".

I was testing using the command:

curl --anyauth --user admin:admin   'http://127.0.0.1:8080/package/ghc-debug-client-0.7.0.0/docs'   --request PUT   --data-binary '@dist-docs/ghc-debug-client-0.7.0.0-docs.tar.gz'   --write-out '\n%{http_code}'   --user-agent 'cabal-install/3.14.2.0 (linux; x86_64)'   --silent --show-error --location   --header 'Accept: text/plain'   --header 'Content-Type: application/x-tar'   --header 'Content-Encoding: gzip' --trace-ascii response.trace

[geekosaur]

Sounds like nginx is configured for default max request size. When I was configuring the bridge there was a warning in the documentation to adjust it.

[gbaz]

hackage has client_max_body_size 40M; -- i think the issue is that when the request is big enough, it turns on chunked encoding, and the issue is in how hackage handles the chunks...

[bgamari]

I have done a fair bit of digging into this; indeed the problem is chunked encoding. Specifically, the problem appears to be an interaction between nginx's chunked-transfer implementation and that of hackage-server: The latter is able to handle directly-issued chunked requests without any trouble.

I have disabled chunking in our nginx configuration for the moment but this deserves more investigation.