Read Zarr Mesh Files

[fm3]

• Similar to Read Zarr Agglomerate Files #8633 we now support reading mesh files in zarr format. Note that there are also neuroglancerPrecomputed meshes, so this time there are three services the meshFileService delegates to.

• The frontend no longer needs to specially handle neuroglancerPrecomputed meshes, since the lookUpMeshFileKey function abstracts from that. Because of that, I also simplified the frontend types.

• Note that zarr meshfiles must be registered in the datasource-properties.json, they are not explored.

• Also fixed a small error in job result status reporting

• Also fixed a small error in mag selection for animation job.

Steps to test:

• Load some precomputed meshes (both zarr, hdf5, neuroglancerPrecomputed; talk to me for example datasets), should show meshes correctly
• Create animations via wk worker, should also work correctly (I tested this locally)
• Note that neuroglancerPrecomputed meshes now need to be registered as attachments in the datasource-properties.json. You may need to re-explore the relevant datasets or edit the json.

TODOs:

• introduce and look up MeshFileKey
• delegate to zarr, hdf5 or neuroglancer meshfile services
• explore remote neuroglancer meshes
• parse new meshfile attributes zarr group format
• do we still need the null vs none meshfile-mapping attribute?
• test with frontend (zarr,hdf5,neuro)
• adapt frontend to no longer need path + fileType for meshfiles
• clear caches
• unify fullMeshStl route
• unify spelling (meshfile vs meshFile)
• When exploring remote neuroglancer meshes, add credentialId, and use it? (or create follow-up issue)
• re-add singletons to module after merge of zarr-agglomerate’s latest changes
• unify function names with zarr-agglomerate’s latest changes

Issues:

• contributes to Support reading new S3 Artifacts #8618
• contributes to Attachments in datasource-properties.json #8567

---

• Updated changelog
• Considered common edge cases
• Needs datastore update after deployment

[MichaelBuessemeyer]

Hej, I have a high level question regarding:

The frontend no longer needs to specially handle neuroglancerPrecomputed meshes, since the lookUpMeshFileKey function abstracts from that. Because of that, I also simplified the frontend types.

Does this mean, that the frontend no longer needs to pay attention to the meshfile type? Or is this still relevant and returned by the backend when the meshfiles are requested?

[fm3]

Does this mean, that the frontend no longer needs to pay attention to the meshfile type?

Correct! And because of that, I removed fileType and path from the ApiMeshFileInfo. formatVersion remains (but only to warn that meshfiles with version <3 are not supported anymore)

[MichaelBuessemeyer]

Here are my first comments :)

[MichaelBuessemeyer]

Here is the rest of the review :)

Thanks for implementing all this so fast 🏎️ 💨

[MichaelBuessemeyer]

Here is the rest of the review :)

Thanks for implementing all this so fast 🏎️ 💨

[MichaelBuessemeyer]

Here are my testing results:

• I can no longer explore the data gs://fafb-ffn1-20190805/segmentation/ which I think has precomputed meshes.On master (wk.org) this worked. So I thikn this is cause by this PR,
• The rest works very nicely (hdf5 mesh, ad hoc mesh, zarr mesh)

[fm3]

Thanks for the detailed testing! Huh, exploring gs://fafb-ffn1-20190805/segmentation/ on this branch works for me, I can then also load its meshes 🤔 Maybe let’s have a look at this together today

[MichaelBuessemeyer]

Nice, looks good. The problem I had with dataset gs://fafb-ffn1-20190805/segmentation/ was that I first explored gs://fafb-ffn1-20190805/section_to_section_xcorr/ and then wanted to add gs://fafb-ffn1-20190805/segmentation/. But they seem to have incompatible mags:

Found 1 Neuroglancer Precomputed layers at gs://fafb-ffn1-20190805/segmentation/.
Error when exploring as layer set: Could not extract common voxel size from layers <~ voxel sizes for layers are not uniform, got List(VoxelSize((8.0, 8.0, 40.0),nanometer)) <~ invalid mag: (0, 0, 1). Must all be powers of two

But this is currently expected and the same behaviour as on the master. Therefore, it should not block this pr and can be solved in a later pr or deemed to be expected.