http: migrate to fsspec & aiohttp

[isidentical]

Resolves #6413

Differences compared to the old HTTPFileSystem (requests based):

• We no longer have digest authentication since it is not supported by the aiohttp, at least not yet. (there is an old PR, ~3 years old that is still open). Instead of a deprecation period, now the config option auth with the value digest will raise an error saying that the option is not available. In the future, if this feature gets accepted to the aiohttp we could bring back the support, but it doesn't seem to be going to happen anytime soon.

[efiop]

@isidentical Btw, since we are depending on fsspec's core more and more (e.g. that lexist issue you've mentioned today), let's setup testing for upstream fsspec? E.g. as a daily build.

[isidentical]

Makes sense!

[efiop]

@isidentical Also, could you please elaborate in the PR description on the things we are missing here and other differences introduced by this? E.g. digest auth.

[skshetry]

+1 for httpx.

[efiop]

@skshetry could you elaborate, please?

[pared]

LGTM, one question.

[efiop]

@isidentical Please clarify the path we will take about digest auth. I know that we've talked about it privately, but we need it to be visible for everyone :)

[isidentical]

@efiop updated the PR description.

[efiop]

For the record: digest auth from our perception seems to be used rare enough that it makes sense for us to take the risk and to drop it for now. If there will be demand to bring it back, we will consider investing time in it. This is bad to drop a feature like that, but in this case it is worth it to move our filesystems/objects forward.

[jorgeorpinel]

Hi. Do we know in what exact release (DVC version) this will be applied? For iterative/dvc.org#2802 (comment)

[efiop]

@jorgeorpinel 2.7.1

[jorgeorpinel]

Thanks