Fix format in volume annotation download zarr.json

[fm3]

Fixes a regression introduced in #7995 – the volume buckets are blosc compressed, but that was no longer mentioned in the zarr header.

Also, the chunks had the c. name prefix, but chunk_key_encoding was already v2 in the zarr header. Now I actually remove the c. name prefix so that it is actually v2 compliant.

Also, fixes passing the voxel size to the volume zip download so that it is correct in the included datasource-properties.json

Steps to test:

• Create volume annotation, download it in zarr format
• Unpack the zip and the contained data_Volume.zip
• place the result in the binaryData folder, should be readable as standalone dataset
• reupload the downloaded annotation, should still be uploadable as an annotation
• Also, upload volume annotation downloaded on the master code, should also still be uploadable as annotation

Issues:

• fixes Migrate all exposed / exported zarr3 configs to use chunk encoding version 2 #7961

---

• Added changelog entry (create a $PR_NUMBER.md file in unreleased_changes or use ./tools/create-changelog-entry.py)
• Considered common edge cases

[coderabbitai]

📝 Walkthrough

Walkthrough

This change updates the handling of Zarr3 array metadata and chunk paths for volume annotation downloads. It ensures compression codec information is included in the generated zarr.json, adjusts the chunk path naming convention, and updates a route parameter. The Zarr3 array header logic is extended to accept additional codec configurations.

Changes

Files/Paths	Change Summary
webknossos-datastore/app/com/.../zarr3/Zarr3ArrayHeader.scala	Updated fromDataLayer to accept an optional additionalCodecs parameter, included in the header codecs.
webknossos-tracingstore/app/com/.../volume/VolumeDataZipHelper.scala	Modified regex to make the "c." prefix in chunk paths optional for Zarr chunk parsing.
webknossos-tracingstore/app/com/.../volume/Zarr3BucketStreamSink.scala	Added compressorConfiguration for Blosc codec; passed as codec to header; removed "c" prefix from chunk file path.
webknossos-tracingstore/conf/tracingstore.latest.routes	Renamed route query parameter from voxelSize to voxelSizeFactor.
unreleased_changes/8800.md	Added changelog entry describing the bug fix for missing compression codec metadata in zarr.json.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• Don't require zarr 3 Bytes Codec configuration key #8282: Both PRs modify codec handling in Zarr3ArrayHeader.scala, with this PR extending codec configuration support and the related PR improving codec reading and validation logic.

Poem

A rabbit hops through Zarr array,
Adding codecs in a clever way.
"No more missing compression!" it cheers,
As chunk paths lose their extra "c" in frontiers.
With headers neat and routes renamed,
The metadata game is now well-tamed! 🐇✨

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b283045 and caa9215.

📒 Files selected for processing (5)

• unreleased_changes/8800.md (1 hunks)
• webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/zarr3/Zarr3ArrayHeader.scala (2 hunks)
• webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeDataZipHelper.scala (1 hunks)
• webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/Zarr3BucketStreamSink.scala (3 hunks)
• webknossos-tracingstore/conf/tracingstore.latest.routes (1 hunks)

🧰 Additional context used

🧠 Learnings (3)

webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeDataZipHelper.scala (1)

Learnt from: frcroth
PR: #8609
File: app/models/dataset/Dataset.scala:753-775
Timestamp: 2025-05-12T13:07:29.637Z
Learning: In the updateMags method of DatasetMagsDAO (Scala), the code handles different dataset types distinctly:

1. Non-WKW datasets have magsOpt populated and use the first branch which includes axisOrder, channelIndex, and credentialId.
2. WKW datasets will have wkwResolutionsOpt populated and use the second branch which includes cubeLength.
3. The final branch is a fallback for legacy data.
   This ensures appropriate fields are populated for each dataset type.

webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/zarr3/Zarr3ArrayHeader.scala (1)

Learnt from: frcroth
PR: #8609
File: app/models/dataset/Dataset.scala:753-775
Timestamp: 2025-05-12T13:07:29.637Z
Learning: In the updateMags method of DatasetMagsDAO (Scala), the code handles different dataset types distinctly:

1. Non-WKW datasets have magsOpt populated and use the first branch which includes axisOrder, channelIndex, and credentialId.
2. WKW datasets will have wkwResolutionsOpt populated and use the second branch which includes cubeLength.
3. The final branch is a fallback for legacy data.
   This ensures appropriate fields are populated for each dataset type.

webknossos-tracingstore/conf/tracingstore.latest.routes (2)

Learnt from: philippotto
PR: #8602
File: frontend/javascripts/oxalis/model/volumetracing/volume_annotation_sampling.ts:365-366
Timestamp: 2025-05-07T06:17:32.810Z
Learning: The parameter in applyVoxelMap was renamed from sliceCount to sliceOffset to better reflect its purpose, but this doesn't affect existing call sites since JavaScript/TypeScript function calls are position-based.

Learnt from: dieknolle3333
PR: #8168
File: frontend/javascripts/oxalis/model/sagas/proofread_saga.ts:1039-1039
Timestamp: 2024-11-22T17:18:04.217Z
Learning: In frontend/javascripts/oxalis/model/sagas/proofread_saga.ts, when calling getMagInfo, the use of volumeTracingLayer.resolutions is intentional and should not be changed to volumeTracingLayer.mags.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: build-smoketest-push
• GitHub Check: frontend-tests
• GitHub Check: backend-tests

🔇 Additional comments (9)

unreleased_changes/8800.md (1)

1-3: LGTM! Clear and accurate changelog entry.

The changelog entry correctly documents the specific bug fix for missing compression codec metadata in zarr.json files during volume annotation downloads.

webknossos-tracingstore/conf/tracingstore.latest.routes (1)

21-21: All client code updated to use voxelSizeFactor
No occurrences of the old voxelSize parameter were found in API consumers. The remote client (app/models/annotation/WKRemoteTracingStoreClient.scala) now uses voxelSizeFactor and voxelSizeUnit. No further changes are needed.

webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeDataZipHelper.scala (2)

98-100: Good backward-compatible regex update.

Making the "c." prefix optional allows parsing both old and new chunk path formats during the transition period. The comment clearly explains the rationale for this change.

---

103-103: Regex usage correctly handles the optional prefix.

The regex capture groups are used consistently - the third group captures the optional "c." prefix, which is then ignored in the parsing logic as intended.

webknossos-tracingstore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/Zarr3BucketStreamSink.scala (3)

44-46: Excellent fix for missing compression codec metadata.

The addition of additionalCodecs = Seq(compressorConfiguration) ensures that compression codec information is properly included in the Zarr array header, addressing the core issue described in the PR.

---

105-105: Chunk path format correctly updated.

Removing the "c" prefix aligns with the v2 chunk key encoding standard and works with the updated regex parsing in VolumeDataZipHelper that makes this prefix optional.

---

121-128: Well-configured Blosc codec settings.

The compressorConfiguration uses appropriate default Blosc settings that match the existing compressor instance below. This ensures consistency between the metadata and actual compression used.

webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/zarr3/Zarr3ArrayHeader.scala (2)

261-263: Clean and backward-compatible API extension.

The addition of the optional additionalCodecs parameter with a default empty sequence maintains backward compatibility while enabling codec injection from calling code.

---

292-292: Correct implementation of codec sequence concatenation.

Appending the additional codecs to the existing codec sequence ensures the proper processing order and maintains the expected codec pipeline structure.

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch zarr-json-in-volume-download

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[MichaelBuessemeyer]

Thanks for fixing all this.
The changes lgtm, but when I tested the download and using the zipped data as a dataset by putting it into the binaryData folder, the voxelISzeFactor was still at [1,1,1]. I also peeked at the datasource-properties.json and it said the same. Therefore, sorry, but this part seems not to be fully fixed. But the scale in the nml was correct

[MichaelBuessemeyer]

oh no 🙈 Good find!

[MichaelBuessemeyer]

Ok nvm 🙈

I just noticed that during testing I accidentally revoked your changes of webknossos-tracingstore/conf/tracingstore.latest.routes locally. Leading to the regression. Now with your version of the file the downloaded datasource-properties.json has the correct voxelScaleFactor 👍