ENH: sample chunk type optimized for vectorized reading

[tstenner]

Current state of chunk 3

The current sample chunk type (tag 3) requires at least one branch per sample (2 for string samples) so both the Python and Matlab implementation require compiled extensions.

Also, the current code to "compress" timestamps isn't helping much due to the usual floating point equality problems, even under perfect conditions:

In [1]: import numpy as np

In [2]: ts = np.arange(151500, 151600, 1/1000)

In [3]: np.sum(np.diff(ts)==1/1000)
Out[3]: 0

In [4]: np.diff(ts)
Out[4]: array([0.001, 0.001, 0.001, ..., 0.001, 0.001, 0.001])

In [5]: np.diff(ts)-1/1000
Out[5]: 
array([-1.07102096e-11, -1.07102096e-11, -1.07102096e-11, ...,
       -1.07102096e-11, -1.07102096e-11, -1.07102096e-11])

In [6]: np.mean(np.diff(np.arange(0, 10, 1/100))==1/100)
Out[6]: 0.002 # 0.2% gets actually left out

Proposal: new chunk type

My proposal for a new chunk type therefore looks as follows:

[Tag 7, uint16] [streamid, uint32] [n_samples uint32] [n_channels uint32] [typeid uint8] # chunk header
[timestamps, n_samples * double64]                    # timestamps, 0: left out
[samples, n_channels * n_samples * channel_type]      # sample data

Each chunk would require at most 3 reads:

streamid, n_samples, n_channels, typeid = struct.unpack('<IIIB', f.read(13))
stream_type = stream_info[streamid].type
timestamps = np.frombuffer(f.read(num_samples*8), np.float64)
data = np.frombuffer(f.read(num_samples * stream.samplebytes), stream.dtype).reshape(n_samples, n_channels)

The chunk includes sample count, channel count and data type for increased robustness (errors can be detected earlier) and so data can be read without an XML parser.

For the timestamp array, a value of 0 (0x0000000000000000 in IEEE 754 representation) indicates a left out timestamp that would be recomputed as usual. Since this chunk format is more compression friendly due to the reduced number of reads / writes, a gzip compressed array would also require less space.

Implementation

The writer is implemented in https://github.com/labstreaminglayer/App-LabRecorder/tree/xdfwriter_chunk7. The main serialization code looks as follows:

_write_chunk_header(chunk_tag_t::samples_v2, len, &streamid);
write_little_endian(file_, n_channels);
write_little_endian(file_, n_samples);
write_little_endian(file_, sampletype);
write_sample_values(file_, timestamps);
write_sample_values(file_, chunk, n_samples * n_channels);

The reader is implemented in https://github.com/tstenner/xdf/commit/412998f3ed66cc7e85a93d77a1d7c96f81073ff0

Benchmark (more below)

A single XDF file with one 64-channel EEG (double) stream was created with 30 chunks, each containing 10K samples.

          Reading (xdfz) Reading (xdf)
Chunk 3:  3.094s         1.704
Chunk 7:  0.961          0.399

In both cases, the new chunks are smaller (156.3 vs. 156.0 MiB, 36.0 vs. 34.9 MiB compressed with gzip -6).

Open questions

[ ] layout for string chunks
[ ] optional padding so data starts at page boundaries (e.g. for mmap)?
[ ] should the redundant type index be included?

[tstenner]

CC @chkothe who wanted to see an example implementation.

[tstenner]

Also CC @agricolab, @cbrnr since this might be useful for a pure python reader.

[cbrnr]

I'm not sure a new chunk type is the best solution. Have you tried Cythonizing the existing type 3 reader?

[tstenner]

If experimented with both Cython and pybind11 (C++ to parse the data chunks), and while the speedups were impressive they were still slower (not to mention more complex) than the pure python implementation sketched above. I'll upload the test code and data files some time next week.

With liblsl we've had our fair share of binary distribution problems (both for Python and Matlab), so a chunk type that enables both fast loading with the base language as well as the option to mmap (parts of) streams into memory would allow for a more portable use of the format, not to mention that the additional fields allow for a lot more error checking than currently possible.

[cboulay]

There’s a lot of concern about feature parity with the matlab importer.

Since it is still early stages, libxdf can be created in a way that keeps it manylinux2010 compatible.

[cboulay]

Neither of my comments above are reasons in themselves to not change the chunk format. As long as things stay backwards compatible then I am for it.

[tstenner]

From my point of view, it's absolutely necessary that all three readers (C++ libxdf, xdf-Python and xdf-Matlab) have a reference implementation before anything is set in stone. For Matlab, the fread function should make the chunk reader code about as simple and fast as the python code I posted above.

[tstenner]

Testing with 10 chunks, 64 channels and 100000 samples each. You can find the code at https://github.com/tstenner/xdf/tree/chunk7 (readers) and https://github.com/labstreaminglayer/App-LabRecorder/tree/xdfwriter_chunk7 (writer; run test_chunk7 as test_chunk7 10 100000 0 0).

		64ch floats	strings
Python	plain	7.10
	C++ ext.	2.67
	chunk7	1.64
Matlab	plain	17.66	28.4
	mex	1.32	0.62
	chunk7	0.69	8.6 1.61

So even though compiled extensions speed everything up considerable, the builtin compiled binary readers are even faster.

Neither of my comments above are reasons in themselves to not change the chunk format. As long as things stay backwards compatible then I am for it.

Most readers either print an error message about the new chunk format or skip the chunks, but a conversion utility could help in the immediate transition. The (as far as I can see) feature complete matlab reader for the chunk 7 is only 37 lines long, so it's quite easy to implement it.

[tstenner]

Added data for the Matlab string chunk reader

[cbrnr]

Can you summarize again why chunk3 reading is slow? I didn't really understand it from you initial post. Is it the nested nature of the samples structure (i.e. that you can't read the whole chunk in on go because you don't know how many bytes it consists of, mainly because of the presence/absence of timestamps)? Can LSL outlets stream chunk7 data? Is there more buffering going on for chunk7 vs. chunk3? What are the pros and cons of both chunk types?

Having said that, XDF is extensible as the name implies, so if there are good reasons why we need something in addition to chunk3 then we should go for it. Personally, I could live with 7 seconds loading time of the plain Python chunk3 reader, but I guess there will be much longer files.

The compiled extensions (Cython/MEX) are already much faster, so it is worth discussing if these are already fast enough. It is a bit odd that MEX is twice as fast as Cython, so I'm assuming that the Cython implementation could be further tweaked.

Furthermore, why is the MATLAB chunk7 reader more than twice as fast as the Python implementation?

[dmedine]

Just a comment on the speed, Matlab has very well optimized matrix handling and is always working to improve performance there, so it's probably always going to be better than python's. As suggested in this post: https://stackoverflow.com/questions/2133031/is-matlab-faster-than-python Numpy can be compiled against ATLAS which will help. Don't know why it isn't distributed this way by default.

Still, 2x faster seems like a lot.

I still think that having a(n optimized) C backend is the best option here. For my money, what's lost in ease of distribution is gained in maintenance and performance. But, I'm not doing any work on this, so my vote doesn't count for much :P

[cbrnr]

Just a comment on the speed, Matlab has very well optimized matrix handling and is always working to improve performance there, so it's probably always going to be better than python's. As suggested in this post: https://stackoverflow.com/questions/2133031/is-matlab-faster-than-python Numpy can be compiled against ATLAS which will help. Don't know why it isn't distributed this way by default.

NumPy is also heavily optimized and they even use the same LAPACK/BLAS libraries in the background. Anaconda links NumPy against Intel MKL, which speeds up things even more (in most use cases). So like you're saying, I'm a bit skeptical that when MATLAB is 2x faster than Python/NumPy I tend to think that the Python code could probably be improved (and maybe it is the code outside of NumPy that could be optimized).

I still think that having a(n optimized) C backend is the best option here. For my money, what's lost in ease of distribution is gained in maintenance and performance. But, I'm not doing any work on this, so my vote doesn't count for much :P

Distributing C bindings is a lot of work for Python, because you need to create binary wheels for all platforms if you want pip install to just work. That's why I normally prefer pure Python or if necessary Cython. But of course then you need to write separate readers for all languages (like we currently do). So yes, having only one C reader with bindings to other languages would be a good option (LSL does that BTW) - but currently we don't have anyone with the time to work on that.