-
Notifications
You must be signed in to change notification settings - Fork 24.3k
Serialization of data within a tensor is slow #9168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
the standard pickler will be slow. if you use Have a look here: https://github.com/pytorch/pytorch/blob/master/torch/serialization.py#L212-L286 |
Right, I'm suggesting making Torch's implementation of pickle fast. |
Separately, what is the most efficient way to convert a torch model/tensor into a set of bytes? Pass an |
@mrocklin the fastest way is to do |
|
I understand. Would the PyTorch community accept a PR that uses numpy within the |
For others, looks like using io.BytesIO gets up to about 1GB/s. In [1]: from torchvision.models.resnet import resnet18
...: model = resnet18(pretrained=True)
...:
...:
In [2]: import torch
In [3]: import io
In [4]: bio = io.BytesIO()
In [5]: %%time
...: torch.save(model, bio)
...: b = bio.getvalue()
...:
CPU times: user 32 ms, sys: 16.4 ms, total: 48.4 ms
Wall time: 47.7 ms
In [6]: len(b) / 0.047 / 1e6 # MB/s
Out[6]: 996.619 And then we can reconstitute io = io.BytesIO(b)
model2 = torch.load(io) |
For context, I maintain a parallel computing library, Dask, and users are naively passing around PyTorch objects and getting poor performance. I can special-case PyTorch models to use the trick above, but it might be a good idea to make PyTorch's general serialization solution decently fast for other libraries that run into the same problem. I think that this is probably pretty easy to do. |
I'll discuss with the team and get back to you in a couple of days. We've avoided a dependence on numpy for functionality so far, but it's been a while since we discussed this.
for the moment, this seems like a good idea. |
Alternatively is there a nice way to get a memoryview out of a storage
object?
…On Wed, Jul 4, 2018, 2:45 PM Soumith Chintala ***@***.***> wrote:
Would the PyTorch community accept a PR that uses numpy within the
*reduce* methods in order to improve serialization performance
I'll discuss with the team and get back to you in a couple of days. We've
avoided a dependence on numpy for functionality so far, but it's been a
while since we discussed this.
I can special-case PyTorch models to use the trick above
for the moment, this seems like a good idea.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9168 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AASszNVqXOMxYD6Qao-TLr7RoxIvJm-Vks5uDQ0_gaJpZM4VCxH2>
.
|
This would be the pure python native solution
…On Wed, Jul 4, 2018, 2:54 PM Matthew Rocklin ***@***.***> wrote:
Alternatively is there a nice way to get a memoryview out of a storage
object?
On Wed, Jul 4, 2018, 2:45 PM Soumith Chintala ***@***.***>
wrote:
> Would the PyTorch community accept a PR that uses numpy within the
> *reduce* methods in order to improve serialization performance
>
> I'll discuss with the team and get back to you in a couple of days. We've
> avoided a dependence on numpy for functionality so far, but it's been a
> while since we discussed this.
>
> I can special-case PyTorch models to use the trick above
>
> for the moment, this seems like a good idea.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#9168 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszNVqXOMxYD6Qao-TLr7RoxIvJm-Vks5uDQ0_gaJpZM4VCxH2>
> .
>
|
MemoryView requires us to implement the Py_buffer interface, which we haven't. Implementing Py_buffer interface across Py2 and Py3 is really complicated (notes from last time we tried doing it). |
Would you welcome a PR implementing the buffer protocol?
…On Wed, Jul 4, 2018, 3:00 PM Soumith Chintala ***@***.***> wrote:
MemoryView requires us to implement the Py_buffer interface, which we
haven't. Implementing Py_buffer interface across Py2 and Py3 is really
complicated (notes from last time we tried doing it).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9168 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AASszLPxJIswjHCgw8X5wHrqvAJxcDdYks5uDRDGgaJpZM4VCxH2>
.
|
Alternatively, PyTorch clearly has code to turn storage objects efficiently into bytestreams. It must do this with For example, a fully usable solution for pickle would be to call the |
Short term, here is a possible workaround that reuses |
Previously this used the ``.toliist`` method, which converted the storage object into a list of Python objects, and then sent those to pickle. For storgae objects of non-trivial size, this was very slow. Now we reuse the logic of the ``torch.save`` function to efficiently turn the Storage object into bytes, and send those instead. This reduces the semantic information (it's harder to interpret the bytes) but should be orders of magnitude more efficient when serializing data with the pickle protocol. For future work it would be nice to develop a mechanism to get a buffer of bytes out of a Storage object, and use that alongside the current ``from_buffer`` method. See pytorch#9168 for context
Summary: Previously this used the ``.toliist`` method, which converted the storage object into a list of Python objects, and then sent those to pickle. For storage objects of non-trivial size, this was very slow. Now we reuse the logic of the ``torch.save`` function to efficiently turn the Storage object into bytes, and send those instead. This reduces the semantic information (it's harder to interpret the bytes) but should be orders of magnitude more efficient when serializing data with the pickle protocol or with copy For future work it would be nice to develop a mechanism to get a buffer of bytes out of a Storage object, and use that alongside the current ``from_buffer`` method. See #9168 for context Closes #9184 Differential Revision: D8747794 Pulled By: soumith fbshipit-source-id: ac598e660c043788ed1ffab3d0303812886edf79
@mrocklin we'll discuss and get back to you |
Summary: Previously this used the ``.toliist`` method, which converted the storage object into a list of Python objects, and then sent those to pickle. For storage objects of non-trivial size, this was very slow. Now we reuse the logic of the ``torch.save`` function to efficiently turn the Storage object into bytes, and send those instead. This reduces the semantic information (it's harder to interpret the bytes) but should be orders of magnitude more efficient when serializing data with the pickle protocol or with copy For future work it would be nice to develop a mechanism to get a buffer of bytes out of a Storage object, and use that alongside the current ``from_buffer`` method. See pytorch#9168 for context Closes pytorch#9184 Differential Revision: D8747794 Pulled By: soumith fbshipit-source-id: ac598e660c043788ed1ffab3d0303812886edf79
Not to reanimate this zombie, but this issue really isn't solved. It seems like it is solved for pickling models (in the above PR) but it is not solved for tensors themselves. If I pickle a tensor ".numpy()" i get much smaller pickles and much faster reads of those pickles (at least 12x faster reads). |
…-pickle 2. greydanus starred a repository - https://github.com/zwimpee/cursivetransformer 3. Benoît Legat's Practical 1 – Linear regressions Url- https://blegat.github.io/ccir/practical1/ 4. The Python Pickle Module 4a. Matthew Rocklin's Blog Post Url: https://matthewrocklin.com/blog/work/2018/07/23/protocols-pickle 4b. Guillaume Lemaitre 5. WhyThisRepo component 6. PR - pytorch/pytorch#9168
Uh oh!
There was an error while loading. Please reload this page.
Issue description
When naively serializing a pytorch tensor (or model) with pickle the process takes longer than it probably should when comparing it to Numpy serialization of the same data. It appears that at some point we're converting things into a list rather than returning the buffers directly.
Code example
The majority of this time is spent in converting the
t.storage()
object into a listInstead, we might consider passing around a numpy array,
buffer
, ormemoryview
, each of which will serialize much more quickly than converting to many Python objectsThe text was updated successfully, but these errors were encountered: