fix HttpResource.resolve() with preprocessing

[pmeier]

#5282 removed the preprocess flag from OnlineResource. In #5584 and #5667 it was discovered that this broke the use case if either decompress or extract is set on a HttpResource and the URL redirects. @YosuaMichael provided a solution in #5667, but I think it will be much better to have a more general solution.

This PR removes the decompress and extract parameters and re-adds preprocess. It can be any Callable[[pathlib.Path], pathlib.Path]. To keep the old convenience, one can also pass preprocess="decompress" or preprocess="extract".

[pmeier]

We could also go for an enum here, but not sure if that would be overkill.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit 8f07a4e (more details on the Dr. CI page):

---

✅ None of the CI failures appear to be your fault 💚

---

• 3/3 broken upstream at merge base 647016b since Mar 21

---

🚧 3 ongoing upstream failures:

These were probably caused by upstream breakages that are not fixed yet.

• binary_linux_conda_py3.7_cu115 since Mar 21 (fbc8ea4)
  • 🔁 rerun
• binary_linux_conda_py3.8_cu115 since Mar 21 (fbc8ea4)
  • 🔁 rerun
• binary_linux_conda_py3.10_cu115 since Mar 21 (fbc8ea4)
  • 🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[NicolasHug]

Thanks for the PR @pmeier

If i remember correctly, the way redirections are done is fairly hacky. Would it be worth thinking of a roboust way of doing these?

[pmeier]

If i remember correctly, the way redirections are done is fairly hacky. Would it be worth thinking of a roboust way of doing these?

Yes, but not now. The thing that doesn't work is redirecting of mirrors and we only have a single dataset with mirrors. In general nothing has changed from the last time we evaluated this. I think we should wait for your investigations on how we want to load data from remote sources, e.g. do we use iopath or something else. If that decision is made, I'm all for refactoring this properly.

[NicolasHug]

Approving to unblock but if we're going to revisit everything anyway, perhaps there are more minimal changes that would work too (and that would be easier to review)? Not sure how tied the proposed changes are to the actual fix here.

[pmeier]

more minimal changes that would work too (and that would be easier to review)?

IMO, the changes are fairly minimal. As stated in my top comment, have a look at #5667 for an alternative solution. The gist from there is that we internally track whether decompress or extract was set to be able to pass the right values when instantiating a new resource after resolving.