Zarr.js - Zarr in the browser (and Node)

[gzuidhof]

I created this issue to discuss Zarr.js - a typescript implementation of Zarr I've been working on for a while now. It has reached a state in which it is useful in the real world and worth discussing :)

Motivation

More and more data visualization and exploration tools are moving to the browser. Why is getting your n-dimensional data into the browser still a pain? Right now getting your data into the browser requires a server that serves data in just the right format for your web application (in bite-sized chunks as memory is limited), Zarr promises to be a format that can remove that complexity. Its chunking and compression is exactly what is required for browsers to handle too-big-for-memory datasets.

I foresee a future in which something similar to Jupyter notebooks can just run in the browser entirely, perhaps dropping into WebAssembly when performance is critical. This would make (results of) experiments much more sharable, interactive and hackable.

Links

• GitHub repository
• Documentation
• Python-to-browser example in the docs
• NPM package

[jakirkham]

cc @freeman-lab (for awareness 😉)

[joshmoore]

As a side note, just before the holidays I had reached out to @FrancescAlted about maybe joining one of the community calls to discuss the state of blosc in the browser (cf. Blosc/c-blosc#238) and how that impacts zarr in the browser. He suggested lopping in @wolfv, and I was going to, but holidays, yada yada. The regular meeting slot (Wed. 8PM UK) is available to anyone who would like it.

[wolfv]

Hi all, yep, I have experimented (very briefly) with compiling blosc to webassembly. it was pretty simple and I got some test cases to run.

It's been a while that I have done this but I would be happy to join some call or similar. I can't make it tonight though.

Cheers! Wolf

[joshmoore]

With Zarr.js having just joined the ZIC (zarr-developers/governance#28) I'm going to assume it's alright to close this issue. Lovely times.