Use old-style buffer on Python 2 #119
Conversation
As Python 2 objects can be comfortably coerced to the old-style buffer interface, which can cleanly be converted to a `memoryview`, go ahead and coerce everything to an old-style buffer in Python 2. Then it is a straightforward matter to get a `memoryview` from the object and handle it the same as one might on Python 3.
Once we have an old-style buffer object in Python 2, use the same `memoryview` handling code for both Python 2 and Python 3.
In Python 2, everything can be comfortably coerced to the old-style buffer interface, which is easily coerced to the new buffer interface. This is a nice path to proceed down as it can cleanly get a `memoryview` on Python 2 without copying. This strategy works well on Python 3 as well as long as we make `buffer` a no-op.
As a minor note, we eliminated a conditional branch that would have been ~25ns (not factored into the previous benchmark), so this is more accurately ~75ns slowdown for Python 3.
Similarly we did not include the |
|
Thanks @jakirkham, didn't know about this. FWIW I wouldn't be too concerned about performance of I'm being picky but I'd be slightly reluctant to introduce |
|
Yeah, it's not well known. Just one of the subtleties of how the new buffer interface got added to Python 2 (much like the fact that Python 2's Sure. Not actually concerned personally. Just trying to be transparent is all. Was using largish (100s to 1000s) of elements to make sure there wasn't a copy. Completely understandable. Actually thought you might say that. Have reverted the last commit, which leaves a Python 2 branch in there. Technically this makes Python 3 slightly faster (though I don't want to micro-optimize 😉). |
|
So one thing that is a little concerning is the In [1]: for i in range(9):
...: n = 10 ** i
...: r = 50
...: t = timeit.timeit(
...: "memoryview(a).tobytes()",
...: "import array; a = array.array('d', {n} * [1.0]);".format(n=n),
...:
...: number=r
...: )
...:
...: print("Took %f secs for %i elements." % (t / float(r), n))
Took 0.000000 secs for 1 elements.
Took 0.000001 secs for 10 elements.
Took 0.000001 secs for 100 elements.
Took 0.000001 secs for 1000 elements.
Took 0.000004 secs for 10000 elements.
Took 0.000065 secs for 100000 elements.
Took 0.001999 secs for 1000000 elements.
Took 0.029316 secs for 10000000 elements.
Took 0.628060 secs for 100000000 elements. |
|
I expect that tobytes() will make a memory copy. IIUC a Python bytes needs
to own its own memory, it can't be shared with another object. Hence we
only resort to creating a bytes object if we know the underlying
compression library cannot receive anything else. Normally we hope that the
underlying compression library can accept any object exposing the new-style
buffer interface, in which case we can pass objects straight through and
don't have to create bytes objects or do anything else that might imply a
memory copy. Hope that makes sense.
…On Thu, 8 Nov 2018, 23:34 jakirkham ***@***.*** wrote:
So one thing that is a little concerning is the tobytes method of
memoryview. This seems fine for most sizes of arrays, but if the array
gets really large this starts to take a startling amount of time, which
makes me wonder if there is a copy happening. Benchmarks taken on Python 3.
In [1]: for i in range(9):
...: n = 10 ** i
...: r = 50
...: t = timeit.timeit(
...: "memoryview(a).tobytes()",
...: "import array; a = array.array('d', {n} * [1.0]);".format(n=n),
...:
...: number=r
...: )
...:
...: print("Took %f secs for %i elements." % (t / float(r), n))
Took 0.000000 secs for 1 elements.
Took 0.000001 secs for 10 elements.
Took 0.000001 secs for 100 elements.
Took 0.000001 secs for 1000 elements.
Took 0.000004 secs for 10000 elements.
Took 0.000065 secs for 100000 elements.
Took 0.001999 secs for 1000000 elements.
Took 0.029316 secs for 10000000 elements.
Took 0.628060 secs for 100000000 elements.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#119 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAq8QoahQCjZ6gtga8T8yHYWcybvgzLRks5utL-dgaJpZM4YVDs3>
.
|
|
Guess that makes sense. Would think that if the data was already read-only (e.g. another |
In the GZip and Zlib encoders, coerce data to old-style buffers on Python 2 as both GZip and Zlib can handle these inputs. By doing this, we are able to avoid inducing a copy as would have been the case when converting the data to `bytes`. We also are able to handle a wide variety of inputs include `bytearray`s and `array`s.
Basically just define `buffer` in the Python 2 world as itself so that `flake8` thinks it is defined and doesn't raise false positives.
This is sort of a lie. That said, they are pretty close to one another and most compat documentation suggests to do this to migrate Python 2 code to Python 3. We only need this so importing `buffer` to appease `flake8` doesn't become even more of a mess, which seems a bit odd.
|
Have reworked this a bit in PR ( #121 ), which seems like a nicer approach. So will close this out. |
Makes use of the old-style buffer on Python 2 to more easily coerce objects to
memoryviews on Python 2. The code here is a bit cleaner by smoothing over Python 2/3 differences. That said, it results in a ~100ns slowdown for Python 3 (due to thelambda) and a ~400ns slowdown for Python 2. This is pretty negligible on the grand scheme of things, but it is worth noting. Figured this might be worth discussion.TODO:
tox -e py37passes locallytox -e py27passes locallytox -e docspasses locally