Metadata update

[ieivanov]

This PR updates the format of the metadata file saved by recorder. The main motivation was to add a field for logging LC voltages (and not just retardance), and to round float numbers before saving. In the process, I updated the structure of the metadata dictionary, adding the Calibration field (in addition to Summary, Notes, and Microscope parameters), changed some entries to more sensible naming (e.g. Calibration scheme instead of Acquired Using), and to more consistent capitalization (e.g. Extinction ratio instead of Extinction Ratio).

Saving LC voltage in the metadata file will be implemented in calibration_data_class_v2 branch, which will be tested and merged soon.

If we like this updated metadata file we should make sure to update recon pipelines that rely on it accordingly, keeping backward compatibility, if possible. @talonchandler, @Soorya19Pradeep, @mattersoflight let me know what you think.

Update:
Introduced MetadataReader class, which handles different versions of the calibration metadata we save. This class allows for backward compatibility and reconstructing data acquired with previous versions of the metadata format. Updated the repo to use the MetadataReader instead of loading the json file directly.

PR of the calibration_data_class_v2 branch will include logging of LC voltage in the calibration metadata (to 4 decimal places) and logging of the voltage interpolation method (e.g. interpolation vs Schoor fit)

[ieivanov]

The old format metadata looks like this

{
 "Summary": {
  "Timestamp": "2022-05-25 18:37:38.910646",
  "recOrder-napari version": "recOrder-napari 0.1.1.dev1+g148b9c3",
  "waveorder version": "waveorder 1.0.0b0",
  "Acquired Using": "5-State",
  "Swing (fraction)": 0.1,
  "Wavelength (nm)": 532,
  "BlackLevel": 144.91,
  "Extinction Ratio": 254.22,
  "ChNames": [
   "State0",
   "State1",
   "State2",
   "State3",
   "State4"
  ],
  "[LCA_Ext, LCB_Ext]": [
   0.30634075285030704,
   0.5834
  ],
  "[LCA_0, LCB_0]": [
   0.20634075285030704,
   0.5834
  ],
  "[LCA_45, LCB_45]": [
   0.3063,
   0.4952139933415637
  ],
  "[LCA_90, LCB_90]": [
   0.3984260299461079,
   0.5834
  ],
  "[LCA_135, LCB_135]": [
   0.3063,
   0.6532008989501689
  ],
  "Swing0": 0.1,
  "Swing45": 0.08818601607486636,
  "Swing90": 0.09208527709580083,
  "Swing135": 0.06980091084682555,
  "ROI Used (x, y, width, height)": [
   0,
   0,
   1232,
   1026
  ],
  "Instrument_Matrix": [
   [
    1.0,
    0.0,
    0.0,
    -1.0
   ],
   [
    1.0,
    0.5877852522924731,
    0.0,
    -0.8090169943749475
   ],
   [
    1.0,
    0.0,
    0.5877852522924731,
    -0.8090169943749475
   ],
   [
    1.0,
    -0.5877852522924731,
    0.0,
    -0.8090169943749475
   ],
   [
    1.0,
    0.0,
    -0.5877852522924731,
    -0.8090169943749475
   ]
  ]
 },
 "Notes": "",
 "Microscope Parameters": null
}

The new metadata format looks like this

{
 "Summary": {
  "Timestamp": "2022-05-31 20:14:30.027134",
  "recOrder-napari version": "recOrder-napari 0.1.1.dev1+g148b9c3",
  "waveorder version": "waveorder 1.0.0b0"
 },
 "Calibration": {
  "Calibration scheme": "5-State",
  "Swing (waves)": 0.1,
  "Wavelength (nm)": 532,
  "Black level": 119.73,
  "Extinction ratio": 237.08,
  "ROI (x, y, width, height)": [
   0,
   0,
   2448,
   2048
  ],
  "Channel names": [
   "State0",
   "State1",
   "State2",
   "State3",
   "State4"
  ],
  "LC retardance": {
   "LCA_ext": 0.30566,
   "LCB_ext": 0.5789,
   "LCA_0": 0.20566,
   "LCB_0": 0.5789,
   "LCA_45": 0.3057,
   "LCB_45": 0.48902,
   "LCA_90": 0.40126,
   "LCB_90": 0.5789,
   "LCA_135": 0.3057,
   "LCB_135": 0.64786
  },
  "LC voltage": {},
  "Swing_0": 0.1,
  "Swing_45": 0.09,
  "Swing_90": 0.096,
  "Swing_135": 0.069,
  "Instrument matrix": [
   [
    1.0,
    0.0,
    0.0,
    -1.0
   ],
   [
    1.0,
    0.58779,
    0.0,
    -0.80902
   ],
   [
    1.0,
    0.0,
    0.58779,
    -0.80902
   ],
   [
    1.0,
    -0.58779,
    0.0,
    -0.80902
   ],
   [
    1.0,
    0.0,
    -0.58779,
    -0.80902
   ]
  ]
 },
 "Notes": "",
 "Microscope parameters": null
}

[deprecated-napari-hub-preview-bot]

Preview page for your plugin is ready here:
https://preview.napari-hub.org/mehta-lab/recOrder/117
Updated: 2022-06-21T21:36:20.057965

[talonchandler]

Thanks @ieivanov! I think these are big improvements in clarity, and I really like the tightened getattr writers.

I have two questions:

1 - Since these files are the only record of the calibration, is rounding the right move here? You're keeping five digits of retardance and three digits of swing. My intuition says that that should be more than enough, but I don't have enough experience with the reconstructions to be sure without testing. Is there any chance that the fifth digit matters when you're on the steepest part of the voltage-retardance curve?

I know you're prioritizing readability here---maybe we should save all digits to file then round only during saving and loading?

2 - Can you update/remind me what you expect the advantages of voltage-space calibration will be? IIRC you're hoping to perform the optimization in voltage space (seems like a clear advantage to me) and avoid the interpolation errors by changing to a closed-form voltage-retardance mapping (again, seems like a clear advantage).

Do these changes necessitate saving in voltage space, though? Could we get away with saving in retardance space for simplicity+backwards compatibility reasons?

[ieivanov]

Thanks @talonchandler, these are good points

• At the steepest point of the retardance vs voltage curve the slope is about -0.3 nm/mV, which roughly correspond to -5 x 10-4 waves / mV. I think the LC controllers would have accuracy on the order of 0.1 mV, so at best we can control the retardance in waves to 6 sig figs. I'll bump the rounding from 5 to 6 digits, 17 is a bit of an overkill here. The much better thing to do is to keep accurate record of the applied voltage.

• Voltage-space calibration and saving the voltage applied to each LC are a bit unrelated. Here I'm creating space in the metadata file for a record of the voltage that is applied to each LC (to be filled in later since I have more convenient functions in the new branch), independent of the calibration method. That is the value that we actually know and can control, the LC retardance is a guess/interpolation that relies on the calibration curve. Makes sense?

[talonchandler]

Great! Thanks for the specs on the number of digits we can control...makes complete sense.

Thanks for the clarification on saving voltage. If I understand correctly, you're planning to always save the calibration result in both retardance and voltage space? Are you planning to save the voltage-retardance mapping as well (in the form the closed-form fit parameters you mentioned)?

These questions are low priority and not that relevant to this PR. Approving now...

[talonchandler]

Minor note: changing 'Ext' to 'ext' here will likely need to be updated in the calibration file reader as well. I'll add a commit shortly.

[ieivanov]

@talonchandler I'm finished working on this PR and I think it's ready to be merged, but it's still failing some of the checks. Can you please help me troubleshoot it? I just ran it again, but I'm expecting it to fail.

[ieivanov]

Since these files are the only record of the calibration, is rounding the right move here

I increased the retardance rounding to 6 decimal places. LC voltage will be rounded to 4 decimal places (0.1 mV) in upcoming PR.

If I understand correctly, you're planning to always save the calibration result in both retardance and voltage space?

Correct, we'll save both LC retardance and voltage, independent of the calibration method

Are you planning to save the voltage-retardance mapping as well (in the form the closed-form fit parameters you mentioned)?

I think it will be useful to have some record of how retardance was converted into voltage, I'll add that in the next PR.

[talonchandler]

Checks are passing now (dropped 3.10 checks for now...will reinstate when numcodecs pushes 3.10 to PyPI zarr-developers/numcodecs#308).

I fixed one typo, and checked that everything opens and is visible from my local machine. I haven't done any tests w/ hardware, but I think this is a WIP that doesn't affect anything beyond the calibration.

Ready to merge from my perspective!

[ieivanov]

Thanks for looking into this! Merging...