See GitHub releases page for the complete history.
This release includes a bug fix for changes from the previous release and a few functionality enhancements.
Add parser support for EK80 MRU1 datagram (#1242)
Add support for consolidate
subpackage functions to accept both in-memory or stored datasets (#1216)
Add test for ES60 spare field decoding issue (#1233)
Add test for EK80 missing receiver_sampling_freq
error (#1234)
Fixed reshape bug in pad_shorter_ping
that was remnant from use_swap
full refactoring (#1234)
This release includes a number of important performance enhancements, bug fixes, and under-the-hood refactoring to improve code readability.
Full refactor of the functionality to directly storing parsed data to zarr before set_groups_* (#1185, #1218, #1070)
This avoids memory expansion during NaN padding across channel
Improve compute_MVBS
using flox (#1124)
Revive and improve compute_NASC
using flox (#1167)
Refactor AZFP XML parser
Improved XML parsing by using xml.etree.ElementTree
to be consistent with other XML parsers in echopype (#1135)
Parse more parameters and store in the resulting EchoData
objects (#1135)
Add partial support for AZFP multiple phase settings (#1182)
Overhaul AZFP Environment group
Set mandatory variables not in data to NaN (sounds_speed_indicative
, absorption_indicative
) (#1226)
Parse and store AZFP pressure data when exist (#1189, #1226)
Add utils.misc.depth_from_pressure
function for converting pressure to depth (#1207)
Remove unused mechanism to selectively parse some EK datagrams (#1214)
Enhancements of EchoData.update_platform
method
Factor out utility functions to a submodule (#1209)
Auto-assign timestamp for fixed-location external data using first ping_time
(#1196)
Add support for consolidate
subpackage functions to accept both in-memory or stored datasets (#1216) [NOT MERGED]
Fix EchoData.to_zarr
encoding (#1128)
Eliminate preferred chunks
Ensuring chunk alignment and encoding only when handling dask arrays
Fix scaling bugs in compute_NASC
(#1167)
Fix UnicodeDecodeError
for ES60 files (#1215)
Handle missing receiver_sampling_frequency
for EK80 data (#1219)
Add support for running individual test files (#1166)
Add module based testing (#1180)
Update CI to barebone python, without conda (#1192)
This release includes a bug fix and a few infrastructure upgrades and enhancements.
More intuitive input syntax for frequency_differencing
(#1106)
Set EK60/80 Platform and NMEA nan timestamp to first ping_time
value (#1154)
This avoids warnings due to xarray’s new behavior with nan timestamp in coordinate
Remove automatic version conversion in open_converted
(#1143)
Bump testing minimum Python version to 3.9 (#1146)
Add Python 3.11 in testing (#1148)
Some tests related to parsed2zarr
fail due to more_itertools
in python 3.11. parsed2zarr
is being overhauled in #1070 so these are deferred to the next release.
Remove logging during testing (#1141)
Remove NASC import causing failure (#1145)
This release includes important updates to the raw-converted data format based on adaptation of convention, an efficiency boost for combining multiple EchoData
objects, packaging and infrastructure upgrades, and other updates necessitated by these changes.
Overhaul the combine_echodata
function (#1042, #1116)
No longer require spinning up dask client under the hood during combine
Use xr.concat
directly compatible with delayed computation
No longer require destination path as input argument
Overhaul EchoData.update_platform
(#1060)
Allow adding external variables with arbitrary number of time dimensions, with new time dimensions following an increasing order (e.g., time3
, time4
, etc)
Only allow updating standardized, sonar model-dependent Platform group variables
Drop the beam
and ping_time
dimensions for some parameters (#1056, #1083)
The dimensions of these parameters were previously expanded to conform with convention requirements, but this inflated the data volume, especially for in memory operations, and made the calibration code confusing
Standardize backscatter_r/i
long_name in the Sonar/Beam_groupX
group, and correct units (#1047)
Standardize use of transmit_frequency_start/stop
for all echosounder models (#1091)
Move filter coefficients and decimation factor to variables in EK80 Vendor_specific
group (#1044, #1046, #1105)
Add new Provenance
group combination_*
attributes to combined EchoData
object, mirroring the conversion_*
attributes (#1113)
Bring more consistency in the Platform
group across sensors on conversion (#1058, #1061)
Add missing mandatory variables and other data format changes (#1094, #1099, #1101, #1102, #1103, #1104, #1107, #1114)
Set time arrays explicitly to datetime64[ns] in parsers and remove duplicated timestamp stuff (#1117)
Remove Pandas<2 pinning (#1080)
Pin netcdf to >1.6 and add explicit encoding (#1112)
Update CI to prepare for python 3.11 (#1108)
Switch github action for get-changed-files (#1118)
Fix discrepancy on test log output (#1119)
Set and fix encoding for variable encoding (#1072)
Specify handling for expected string type
Sanitize and set encoding as part of EchoData
Handling problematic NMEA messages when setting lat/lon in Platform group (#1067)
Update permission check filename to be UUID to make it thread safe (#1110)
use pulse_form
for EK80 transmit type checks (#1091)
Factor out a normalization factor from pulse compression function for flexible use (#1105)
Add existence checking for angle parameters in consolidate.add_splitbeam_angle
(#1105)
Revise qc function to clean up reversed time (#1065)
Update matplotlib call to register EK500 colormap (#1068)
Temporarily remove compute_NASC
for debugging and overhaul (#1136)
Remove the deprecated preprocess
subpackage (#1077, #1082)
Remove the ZarrCombine
module since it is no longer used (#1117)
Remove outdated ui_param
(#1071)
Add rendering for example EchoData
objects for all supported echosounders to show the updated format (#1089)
Restructure TOC to have separate Raw data formats and Data Processing sections, each with new subpages (#1093, #1132, #1134)
Add discussion on z-axis vertical coordinate system and variables (#1133)
Add bibliography page linked from inline citations (#1137)
This release includes important updates to the requirements, small bug fixes, and refactoring of AZFP parser code to handle files from glider deployment and improve code coherence.
AZFP conversion enhancements
Add error catch to handle AZFP files with no temperature or tilt, such as those from glider deployments (#1020)
Refactor AZFP parser code to improve code coherence (#1024)
Correct and add missing SONAR-netCDF4 variable attributes for AZFP (#1023)
Improve impedance parameter handling for EK80 files (#1019)
allow impedance data type to be float
unify naming of impedance parameters
Allow only one chirp generation option for EK80 broadband processing (#1019)
previously there was an option to use the implementation from Matlab echolab
starting from this version only the implementation identical to those from Lars Andersen (repo)
Remove unneeded return self in processing level decorator wrapper-inner function for class method (#1037)
Unpin xarray version and pin pandas < 2 (#981)
allow v0.7.x in map_ep_version
checks
This release includes new features to interface with Echoview ECS files for computing Sv, reorganization of computing functions into new subpackages, addition of data processing level attributes to data products, and other improvements and bug fixes.
Allow using ECS for calibrating Simrad echosounders (#996, #1004)
This functionality is in a beta testing stage
Details of implementation may change and bugs are possible
Expand ECS parser to accept frequency-dependent values in EK80 ECS files
Overhaul env_params
to ensure correct intake for calibration (#985)
Now allows using env_params
entries that are xr.DataArrays
Move functions previously in preprocess
subpackage to new subpackages (#993). Calling these functions from preprocess
is deprecated and will be removed in v0.7.1.
clean
: remove_noise
, estimate_noise
commongrid
: compute_MVBS
, compute_MVBS_index_binning
Add commongrid.compute_NASC
(#1005)
The current implementation uses brute force looping for mean Sv computation, this will be refactored and optimized together with other functions requiring the same pattern in an upcoming release
Add global attributes for data processing levels (#1001).
This functionality is in a beta testing stage
See data processing level specifications for functions and conditions under which such attributes are added
Expand mask.apply_mask
to handle multi-channel Sv datasets (#1010)
Standardize sonar metadata for EK80 data (#992)
sonar_serial_number
is now an empty global attribute, no longer a variable, as in the EK60 case
transducer_name
, transducer_serial_number
, transceiver_serial_number
based on parser parameter transducer_name
, transducer_serial_number
, and serial_number
, respectively
Fix scaling bug for beamwidth_alongship
and beamwidth_athwartship
from CW-based parameters to values corresponding to center frequency of broadband transmit signals (#998)
Add more comprehensive tests for add_location
(#1000)
Add test for splitbeam angle ek80_CW_power
case (#994)
Add unit and integration tests for env_params
intake for calibration (#985)
Exclude test_data
folder in codespell
pre-commit hook (#1016)
This is a minor release that includes a small bug fix and an enhancement, which removes excessive warnings due to changes in v0.6.4.
Fix a bug that prevented passing in env_params
for EK60 calibration (#987)
Handling non-positive values that causes log10 warnings on EK calibration (#986)
This is a release that includes important performance enhancements that allow user-provided calibration parameters for broadband and narrowband calibration, new functionalities to compute and attach split-beam angles to calibrated Sv dataset, perform frequency-differencing and masking, as well as a number of bug fixes and other improvements.
Allow passing in cal_params
as a dictionary for narrowband and broadband calibration (#955)
Add default chunk encoding for zarr output (#939)
Add add_splitbeam_angle
function to the consolidate
subpackage (#916, #971)
Add a new mask
subpackage
Add apply_mask
function to the mask
subpackage (#912)
Add frequency-differencing
function to the mask
subpackage (#901)
Allow selection of a subset of channels when combining multiple echodata objects (#892)
This is done via the added channel_selection
input argument to combine_echodata
Add default consolidated flag for echodata.to_zarr
(#855)
Generalize and improve efficiency of compute_MVBS
(#878)
Allow echo_range
that vary with ping_time
Allow Sv
data that are dask arrays
Add .nbytes
to obtain that data size of an echodata
object (#874)
Add new data variables from the raw data files to facilitate EK80 calibration (#944)
transceiver type, transmit impedance, receive impedance, receiver sampling frequency
Unify the order of dimensions of echo_range
across all sonar models (#968)
Create and use the default echopype home directory (#896, #954)
This default directory is at ~/.echopype
Refactor the calibrate
subpackage (#904)
routines for env_params
and cal_params
intake and range computation are now in different modules rather than methods in each classes
Tidy up cal_params
related routines (#953)
Revise env_params
related routines (#952)
Improving pulse compressed broadband Sv computation and echo range computation (#944)
The resulting values are tested against pyEcholab and Echoview outputs
Note there is unresolved discrepancy between pyEcholab/Echopype outputs with Echoview outputs for the first section of Sv values
Change input argument names for open_raw
(#962)
Change offload_to_zarr
to use_swap
Change max_zarr_mb
to max_mb
Handling of provenance attributes in apply-mask and add-depth, especially for testing (#930)
Fix bugs in combining multiple echodata objects (the combine_echodata
function)
Fix meta_source_filenames bug and enable (meta)source_filenames appending of path and list (#913)
Fix bugs in env_params
intake for calibration (#952)
Update docker build for arm64 mac silicon chips (#964)
Fix windows github actions workflow for utils module (#947)
Remove mamba dependency (#946)
Fix pre-commit ci and update RTD config for jupyter-book (#934)
This is a minor release that includes an important performance enhancement for combining large volumes of data residing in individual files into a single entity, a number of bug fixes, and other smaller improvements.
Overhaul combine_echodata
function
Allow combine a large number of EchoData
objects exceeding memory limits (#808, #824, #830)
Remove reversed time check from combine_echodata (#835)
Add minimal ZarrCombine test (#826)
Order the channel coordinate in file conversion to ensure consistent combination across files (#818)
Revise outdated data combination behavior (#797, #799)
Clean up all coordinate and attribute details under combine_echodata
function (#848, #849)
Track provanance for filenames of raw data files and auxiliary files
Propagate xml_path
as meta_source_filenames
to combined echodata (#814, #852)
Standardize handling of source_filenames
in individual and combined echodata as well as downstream datasets (#804, #806)
Clean up functions for setting encoding in the converted files (#851)
Add requests
and aiohttp
to dependency (#844)
Pin netcdf4
to be <1.6 for pypi package due to ongoing netcdf-c
problem (#843)
Write Parsed2Zarr
generated files to temp_echopype_output/parsed2zarr_temp_files
(#832)
Change isel
to sel
to fix compute_Sv
to allow working with dask array (#828)
Add open_raw(offload_to_zarr=True)
integration tests (#794)
Fix regression bug with interpolating environmental variables to ping_time
grid (#837, #856)
Fix WindowsPath error with compute_Sv when run on Windows (#829)
Fix logic problem in open_raw(offload_to_zarr=True)
that sometimes cause problems (#794, #853)
Update CI to use micromamba (#805)
Fix version string on CI (#804, #820)
Rename ci.yaml
to build.yaml
for clarity (#807)
This is a minor release that includes a few new features and memory efficiency-related changes that make echopype better.
Add a new subpackage consolidate
that contains functions to consolidate data across the calibrated Sv dataset and the corresponding raw-converted file
Add function consolidate.add_location
to interpolate location to calibrated dataset (#749)
Add function consolidate.add_depth
to convert range_meter
to depth
with information on transducer tilt and depth (#738)
Move function that swaps the channel with frequency dimension (now as consolidate.swap_dims_channel_frequency
) (#738)
Add new functionality to allow control of logging outputs (#772)
Improve memory usage while converting files that require significant NaN-padding and previously would incur very large memory expansion (#774)
This is achieved by directly writing variables that may incur a large memory expansion into a temporary zarr store
Beta function that will benefit from user feedback
Overhaul access pattern for EchoData (#762)
Remove previous access pattern for different groups in the raw-converted file
Starting from this release all groups are accessed with echodata["GROUP_PATH"]
, e.g., echodata["Platform"]
, echodata["Sonar/Beam_group1"]
, etc.
Make long_name in ds_power for EK80 consistent with other sonar model (#771)
Modify set_beam()
so it returns a list (#780)
Change the order in _save_groups_to_file
so the order of groups is preserved when opening a converted netcdf file (#779)
Remove the user option to select NMEA sentences in open_raw
to ensure raw data is preserved (#778)
Update CI set up to avoid exceeding GitHub actions memory limitation (#761)
decrease the number of workers from 4 to 2
This is a minor release that includes important bug fixes, a number of new features, and some leftover data format changes specific to parsed AD2CP data format.
We use datatree
under the hood for the EchoData
object, but datatree
v0.0.4 had a bug in accessing subgroups in netCDF data model in Windows OS. This was fixed in datatree
v0.0.6, and hence we updated our dependency and made other associated changes to support cross platform users (#732, #748)
Fix a bug in compute_MVBS
in selecting echo_range
for specific frequency. This is from the process of converting the data to be aligned with dimension frequency
to channel
in v0.6.0 (#736)
Allow parsing data from EK60 with split-beam transducers but without phase/angle data (#491, #718)
Fix invalid timestamp issue in AD2CP data conversion (#733)
Check filter coeffs existence in SetGroupsEK80.set_vendor
before saving (#720, #724)
Fix empty Sv problem related to renaming time coordinate associated with environmental parameters used for calibration (#755)
Fix the check in compute_MVBS
for handling different variations of NaN entries in the Sv dataset (#753)
Enhance update_platform
to support a new use case (location data from fixed location) and add more consistency (#741)
Ability to parse and store RAW4 datagram for EK80 data (#714)
Add utility function for swapping channel
coordinate with frequency_nominal
(#710)
Add ES70, ES80, EA640 to allowed data type for calibration (#759)
Reorganize AD2CP data variables into different Sonar/Beam_groupX
s and different first-level groups in a form consistent with v0.6.0 changes for all other sonar models (#731); some variables remain to be discussed and may change in future releases (#719)
Refactor AD2CP conversion to improve speed and memory usage through removal of xr.merge (#505)
Update Python requirements in docs to >=3.8 (#744)
Update PR action to use PR title [all tests ci] to run the entire suite of tests and [skip ci] to skip all tests (#721)
This is a major release that contains changes that enhances the compliance of echopype data model (and hence generated file structure) to the SONAR-netCDF4 convention Version 1.0 . In addition, some variables were renamed to improve intuitive understanding of sonar data, provenance and standardized attributes are added to the processed dataset (e.g. Sv), the deprecated old API (<0.5.0) was removed, and some bugs were fixed.
Move and rename the original Beam
and Beam_power
group to be subgroups under the Sonar
group, in order to comply with the structure defined in the convention (#567, #574, #605, #606, #611)
Beam
–> Sonar/Beam_group1
: contains either raw power or power/angle data for all sonar models other than EK80. For EK80, if only complex or power/angle data exist, all data are in this group; if both complex and power/angle data exist, the complex data are in this group.
Beam_power
–> Sonar/Beam_group2
: contains power/angle when complex data occupies Sonar/Beam_group1
; only exists for EK80 data when both power/angle data and complex data bothexist in the file
Rename the coordinate range_bin
to range_sample
to make it obvious that this coordinate indicates the digitization sample number for the sound waveform or intensity time series, and hence it takes a form of sequential integers 0, 1, 2, 3, … (#595)
Rename the data variable range
in the calibrated Sv or TS dataset to echo_range
, so that it is not confused with the python built-in function (#590)
Rename the coordinate quadrant
for EK80 data to beam
(#619)
Add coordinate beam
with length 1 for all sonar models, except for AD2CP (#638, #646)
Rename the data variable Sp
to TS
since “point backscattering strength” is a Simrad terminology and target strength (TS) is clearly defined and widely used. (#615)
Rename time dimensions in the Platform
group (location_time
: time1
, mru_time
: time2
) (#518, #631, #647)
Rename the coordinate frequency
to channel
for all groups, to be more flexible (can accommodate channels with identical frequencies #490) and reasonable (since for broadband data the channel frequency is only nominal #566) (#657)
Rename the data variable heave
to vertical_offset
in the Platform group (#592, #623)
Change src_filenames
string attribute to source_filenames
list-of-strings variable (#620, #621)
Bring consistency to the names of the time coordinates for the Platform
and Environment groups
(#656, #672)
The groups can now be accessed via a path in the form echodata["Sonar/Platform"]
, echodata["Sonar/Platform"]
, echodata["Sonar/Beam_groupX"]
, etc. using DataTree v0.0.4 functionalities. (#611)
The previous access pattern echodata.platform
, echodata.sonar
, echodata.beam
, etc. is deprecated and will be removed in v0.6.1.
Add indexing info for Beam_groupX
as data variable under the Sonar
group (#658)
Add missing coordinate and variable attributes in the processed datasets Sv, MVBS, TS(#594)
Add water_level
to processed datasets (Sv, MVBS, TS) for convenient downstream conversion to depth (#259, #583, #615)
Add additional environment variables for EK80 data (#616)
Add missing platform data variables (#592, #631, #649, #654)
Add parser for Echoview ECS file (#510)
Add provenance to raw-converted and processed datasets (#621)
Consolidate convention specs into a single yml file for pre-loading when creating EchoData
objects (#565)
Extra visualization module can now handle both frequency
and channel
filtering, as well as files with duplicated frequencies (#660)
Improve selection of Vendor-specific calibration parameters for narrowband EK data (#697)
Upgrade python minimum to 3.8 and tidy up packaging (#604, #608, #609)
Upgrade echopype development status to Beta (#610)
Update setup-services.py
to include images & volumes subtleties (#651)
Remove the deprecated old (<0.5.0) API (#506, #601)
Update README in the echopype/test_data
folder (#584)
Add documentation for visualization (#655)
Add development roadmap to documentation (#636, #688)
Restructure and expand data format section (#635)
This is a minor release that contains an experimental new feature and a number of enhancements, clean-up and bug fixes, which pave the way for the next major release.
(beta) Allow interpolating CTD data in calibration (#464)
Interpolation currently allowed along the ping_time
dimension (the "stationary"
case) and across latitude
and longitude
(the "mobile"
case).
This mechanism is enabled via a new EnvParams
class at input of calibration functions.
Make visualize module fully optional with matplotlib
, cmocean
being optional dependency (#526, #559)
Set range entries with no backscatter data to NaN in output of echodata.compute_range()
(#547) and still allows quick visualization (#555)
Add codespell
GitHub action to ensure correct spellings of words (#557)
Allow sonar_model="EA640"
for open_raw
(before it had to be “EK80”) (#539)
Allow using sonar_model="EA640"
(#538, #539)
Allow flexible and empty environment variables in EA640/EK80 files (#537)
Docstring overhaul and fix bugs in utils.uwa
(#525)
Upgrade echopype docs to use jupyter book (#543)
Change the RTD latest
to point to the dev
branch (#467)
Update convert tests to enable parallel testing (#556)
Overhaul tests (#523, #498)
use pytest.fixture
for testing
add ES70/ES80/EA640 test files
add new EK80 small test files with parameter combinations
reduce size for a subset of large EK80 test data files
Add packaging testing for the dev
branch (#554)
This is a minor release that includes new features, enhancements, bug fixes, and linking to an echopype preprint.
Allow converting ES60/70/80 files and handle various datagram anomaly (#409)
Add simple echogram plotting functionality (beta) (#436)
update_platform
method for EchoData
now include proper variable attributes and correctly selects time range of platform data variables corresponding to those of the acoustic data (#476, #492, #493, #488)
Improve testing for preprocess.compute_MVBS
by running through real data for all supported sonar models (#454)
Generalize handling of Beam group coordinate attributes and a subset of variable attributes (#480, #493)
Allow optional kwargs when loading EchoData
groups to enable delaying operations (#456)
The gain factor for band-integrated Sv is now computed from broadband calibration data stored in the Vendor group (when available) or use nominal narrowband values (#446, #477)
Fix time variable encoding for combine_echodata
(#486)
Fix missing ping_time
dimension in AZFP Sv dataset to enable MVBS computation (#453)
Fix bugs re path when writing to cloud (#462)
Improvements to the “Contributing to echopype” page: Elaborate on the git branch workflow. Add description of PR squash and merge vs merge commit. Add instructions for running only a subset of tests locally (#482)
Add documentation about output_storage_options
for writing to cloud storage (#482)
Add documentation and docstring for sonar_model
in open_raw
(#475)
Improve documentation of EchoData object by adding a sample of the xarray Dataset HTML browser (#503)
Zenodo badge update (#469)
Add github citation file (#496), linking to echopype preprint on arXiv
This is a minor release that contains a few bug fixes and new functionalities.
The repo has migrated to use main
instead of master
after this release.
Adding external platform-related data (e.g., latitude, longitude) to the EchoData
object via the update_platform
method (#434)
Allow converting and storing data with duplicated ping times (#433)
Add simple functions to compute summary statistics under the metrics
subpackage (#444)
Allow string info in AD2CP data packet header (#438)
Re-attach sonar_model
attribute to outputs of combine_echodata
(#437)
Handle exception in open_converted
due to potentially empty beam_power
group in Zarr files (#447)
Warn users of removal of old API in the next release (#443)
This is a minor release that adds a few new functionalities, in particular a method to combine multiple EchoData
objects, addresses a few bugs, improves packaging by removing pinning for dependencies, and improving the testing framework.
Add a new method to combine multiple EchoData objects (#383, #414, #422, #425 )
Potential time reversal problems in time coordinates (e.g., ping_time
, location_time
) are checked and corrected as part of the combine function
The original timestamps are stored in the Provenance
group
Add a new method compute_range
for EchoData
object (#400)
Allow flexible extensions for AZFP files in the form “.XXY” where XX is a number and Y is a letter (#428)
Fix the bug/logic problems that prevented calibrating data in EK80 files that contains coexisting BB and CW data (#400)
Fix the bug that prevented using the latest version of fsspec
(#401)
Fix the bug that placed echosounder_raw_transmit_samples_i/q
as the first ping in echosounder_raw_samples_i/q
as they should be separate variables (#427)
Consolidate functions that handle local/remote paths and checking file existence (#401)
Unpin all dependencies (#401)
Improve test coverage accuracy (#411)
Improve testing structure to match with subpackage structure (#401, #416, #429 )
Expand Contributing to echopype
page, including development workflow and testing strategy (#417, #420, #423)
This is a minor release that addresses issues related to time encoding for data variables related to platform locations and data conversion/encoding for AD2CP data files.
Fixed the location_time
encoding in the Platform
group for latitude and longitude data variables (#393)
Fixed the location_time
encoding in the Platform/NMEA
group (#395)
Updated EchoData
repr to show Platform/NMEA
(#395, #396)
Improved AD2CP data parsing and conversion (#388)
Cleaned up organization of data from different sampling modes and their corresponding time coordinates
Fixed parsing issues that generated spikes in parsed echosounder mode amplitude data
Removed the Beam_complex
group and put raw IQ samples in the Vendor
group per convention requirements
Populated the Sonar
group with AD2CP information
This is a minor release that addresses a couple of issues from the last major version (0.5.0) and improves code maintenance and testing procedures.
Added experimental functions to detect and correct ping_time
reversals.
See qc
subpackage (#297)
Fixed ADCP encoding issues (#361)
Updated SetGroupsBase
to use
ABC (Abstract Base Classes) Interface (#366)
Whole code-base linted for pep8 (#317)
Removed old test data from the repository (#369)
Updated package dependencies (#365)
Simplified requirements for setting up local test environment (#375)
Added code coverage checking (#317)
Added version check for echopype install (#367, #370)
#v0.5.0 (2021 May 17)
This major release includes:
major API updates to provide a more coherent data access pattern
restructuring of subpackages and classes to allow better maintenance and future expansion
reorganization of documentation, which also documents the API changes
overhaul and improvements of CI, including removing the use of Git LFS to store test data
new features
bug fixes
The existing API for converting files from raw instrument formats to a standardized format, and for calibrating data and performing operations such as binned averages and noise removal has been updated.
The new API uses a new EchoData
object to encapsulate all data and metadata related to/parsed from a raw instrument data file. Beyond the calibration of backscatter quantities, other processing functions follow a consistent form to take an xarray Dataset as input argument and returns another xarray Dataset as output.
The major changes include:
change from an object-oriented method calls to functional calls for file conversion (using the new convert
subpackage), and deprecate the previous Convert
class for handling file parsing and conversion
deprecate the previous Process
class, which use object-oriented method calls for performing both calibration and data processing
separate out calibration functions to a new calibrate
subpackage
separate out noise removal and data reduction functions to a new preprocess
subpackage
create a new EchoData
object class that encapsulates all raw data and metadata from instrument data files, regardless of whether the data is being parsed directly from the raw binary instrument files (returned by the new function open_raw
) or being read from an already converted file (returned by the new function open_converted
)
The subpackages and classes were restructured to improve modularity that will help will future expansion and maintenance. The major restructuring includes: (“SONAR” below is used to indicate the sonar model, such as EK60, EK80 or AZFP)
consolidate overlapping EK60/EK80 components, deprecate the previous Convert
classes that handled file parsing and serialization, and revise new ParseSONAR
and SetGroupsSONAR
classes for file parsing and serialization
consolidate all calibration-related components to a new calibrate
submodule, which uses CalibrateSONAR
classes under the hood
consolidate all preprocessing functions into a a new preprocess
submodule, which will be later expanded to include other functions with similar use in a workflow
Added github workflows for testing, building test docker images, and publishing directly to PyPI
Deprecated usage of Travis CI
Test run is now selective on Github, to run tests only on changed/added files. Or run all locally with run-test.py
script. (#280, #302)
Re-organization of pages with better grouping
Added “What’s New” page
Added “Contributing to echopype” page
Overhaul “API reference” page
Add interfacing capability to read from and write to cloud object storage directly. (#216, #240)
Allow environmental and calibration parameters to be optionally used in calibration in place of the values stored in data file
Mean volume backscattering strength (MVBS) can now be computed based on actual time interval (specified in seconds) and range (specified in meters) (#54)
Add NMEA message type as a data variable in the Platform
group (#232), which allows users to freely select the suitable ones depending on use
Add support to convert .ad2cp
files generated by Nortek’s Signature series ADCP (#326)
Fix EK80 config XML parsing problem for files containing either PulseDuration
or PulseLength
(#305)
Fix time encoding discrepancy in AZFP conversion (#328)
Fix problematic automatic encoding of AZFP frequency (previously as int
) to float64
(#309)
Overhaul EK80 pulse compressed calibration (current implementation remaining in beta, see #308)
Patches and enhancements to file conversion
This minor release includes the following changes:
Fix bug in top level .nc output when combining multiple AZPF .01A
files
Correct time stamp for .raw
MRU data to be from the MRU datagram, instead of those from the RAW3 datagrams (although they are identical from the test files we have).
Remove unused parameter sa_correction
from broadband .raw
files
Make sure import statement works on Google colab
Parse Simrad EK80 config XML correctly for data generated by WBAT and WBT Mini, and those involving the 2-in-1 “combi” transducer
Parse Simrad .raw
files with NME1
datagram, such as files generated by the Simrad EA640 echosounder
Handle missing or partially valid GPS data in .raw
files by padding with NaN
Handle missing MRU data in .raw
files by padding with NaN
Parse .raw
filename with postfix beyond HHMMSS
Allow export EK80 XML configuration datagram as a separate XML file
To increase maintenance efficiency and code readability we are refactoring the convert
and process
modules. Some usage of these modules will change in the next major release.
Add EK80 conversion, rename subpackage model to process
Add EK80 support:
File conversion from EK80 .raw
files to netCDF and zarr formats
“Simple” calibration to frequency-average Sv based on pulse compression output is implemented but needs to be thoroughly tested.
Rename subpackage echopype.model
to echopype.process
The new name better describes the subpackage’s function to process data for further analysis
Also rename class EchoData
to Process
to mirror the structure in Convert
better.
Importing using the old names will be deprecated in the next release.
Overhaul converting multiple files with combine_opt=True
If target format is netCDF, temporary files will be created and finally combined to a single netCDF. This is due to current restriction that xarray does not allow simply appending new data to an existing file.
If target format is zarr, data in each file are unpacked and appended to the same output file.
Allow reading Zarr into Process
in addition to netCDF: thanks @lsetiawan!
Add a logo!
Fix bugs in slicing NMEA group data based on the same time base when range_bin
is changed