fs.upload: add support for optional callback #6546
Conversation
skshetry
force-pushed
the
upload_callback
branch
5 times, most recently
from
8524c36 to
32c4cb3
Compare
| ExtraArgs=self.fs_args.get("s3_additional_kwargs"), | ||
| Config=self._transfer_config, | ||
| ) | ||
| callback.set_size(os.path.getsize(from_file)) |
There was a problem hiding this comment.
I am not sure if we need self.makedirs here in the put_file these days, cc @isidentical.
| s3 = rem.fs | ||
| s3.fs.mkdir("another/") | ||
| s3 = rem.fs.s3 | ||
| s3.create_bucket(Bucket="another") |
There was a problem hiding this comment.
The s3.fs.mkdir was failing locally for me, though I am not sure why. I replaced this with the one that shares the intention in a better way, which passes both locally and in CI.
skshetry
force-pushed
the
upload_callback
branch
from
32c4cb3 to
9426ade
Compare
skshetry
force-pushed
the
upload_callback
branch
2 times, most recently
from
a6f21be to
28e8ba0
Compare
| def _upload( | ||
| self, from_file, to_info, name=None, no_progress_bar=False, **pbar_args | ||
| @staticmethod | ||
| def put_file_compat( |
There was a problem hiding this comment.
Temporary, I'm planning to get rid of _upload/_download from CallbackMixin and then this CallbackMixin will just be a way to provide compat support for callback. Naming is bad, but will just be replace by put_file after I get rid of _download.
skshetry
force-pushed
the
upload_callback
branch
from
be304ff to
07496c5
Compare
| super().absolute_update(value) | ||
|
|
||
|
|
||
| def tdqm_or_callback_wrapped( |
There was a problem hiding this comment.
This codebase is a bit weird, as we need to support both tqdm and FsspecCallback for now.
| @@ -182,9 +181,29 @@ def set_size(self, size): | |||
| size = self.fs.info(self.path_info)["size"] | |||
| self.progress_bar.total = size | |||
| self.progress_bar.refresh() | |||
| super().set_size(size) | |||
There was a problem hiding this comment.
These will be useful in tests, as we can introspect size and value easily.
Also removes _upload methods in other fses, and replaces them with fsspec-compatible put_file which supports fsspec callback even if they are not fsspec based filesystem.
skshetry
force-pushed
the
upload_callback
branch
from
f311ca6 to
7d3cf02
Compare
for more information, see https://pre-commit.ci
efiop
added
refactoring
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.