Data processing functionalities

Summary of subpacakges

Echopype currently contains the following data processing subpacakges:

The clean and commongrid subpackages were introduced in version 0.7.0. They contain functions previously found in the deprecated preprocess subpackage; preprocess was removed in version 0.8.0.

The use of these processing functions are summarized below:

Calibration

nc_path = './converted_files/file.nc'  # path to a converted nc file
echodata = ep.open_converted(nc_path)  # create an EchoData object
ds_Sv = ep.calibrate.compute_Sv(echodata)  # obtain a dataset containing Sv, echo_range, and
                                           # the calibration and environmental parameters

The calibration functions can require different input arguments depending on the echosounder instrument (sonar_model). See data-processing:calibration for detail.

Noise removal

# Remove noise
ds_Sv_clean = ep.clean.remove_noise(    # obtain a denoised Sv dataset
    ds_Sv,                # calibrated Sv dataset
    range_sample_num=30,  # number of samples along the range_sample dimension for estimating noise
    ping_num=5,           # number of pings for estimating noise
)

Reduce data by computing MVBS

# Reduce data based on physical units
ds_MVBS = ep.commongrid.compute_MVBS(
    ds_Sv,               # calibrated Sv dataset
    range_meter_bin=20,  # bin size to average along echo_range in meters
    ping_time_bin='20S'  # bin size to average along ping_time in seconds
)

# Reduce data based on sample number
ds_MVBS = ep.commongrid.compute_MVBS_index_binning(
    ds_Sv,                # calibrated Sv dataset
    range_sample_num=30,  # number of sample bins to average along the range_sample dimensionm
    ping_num=5            # number of pings to average
)

Saving results

Typically echopype functions do not save the calculation resuls to disk, but the returned xarray Dataset can be saved using native xarray method such as to_netcdf and to_zarr. For the EchoData object we have implemented identical methods to allow saving raw converted data to disk. For example:

ed = ep.open_raw("PATH_TO_RAW_FILE", sonar_model="SONAR_MODEL")
ed.to_zarr("RAW_CONVERTED_FILENAME.zarr")  # save converted raw data to Zarr format

# Some processing functions that results in an xarray Dataset ds_Sv
ds_Sv.to_netcdf("PROCESSED_FILENAME.nc")  # save data to netCDF format

Parameter considerations for calibrating data

Calibration transforms raw data collected by the instrument into physically meaningful units by incorporating:

Echopype also requires correct input argument combinations to calibrate data, due to intrinsic differences of the echosounders.

These considerations are explained below.

For more information on calibration, see [DBB+15].

Input arguments

AZFP and EK60 data

For data from the AZFP and EK60 echosounders, since these instruments can only transmit narrowband data, you do not have to specify any additional argument when calibrating data, and the following should always work:

ed = ep.open_raw("PATH_TO_RAW_FILE", sonar_model="EK60")
ds_Sv = ep.calibrate.compute_Sv(ed)

EK80 data

For data from the EK80 echosounder, since both narrowband and broadband transmissions are possible for different channels, and for narrowband transmissions the data can be stored in two forms, you need to specify argument combinations corresponding to the data you want to calibrate.

For computing band-averaged Sv from broadband EK80 data, use:

ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="BB", encode_mode="complex")

Here, waveform_mode="BB" indicates that the data you want to calibrate are the channels set to do broadband transmissions. encode_mode="complex" indicates that these data are stored as complex samples. The function will raise an error if there are no broadband data found in the provided EchoData object (ed).

For computing narrowband Sv from narrowband EK80 data:

ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="CW", encode_mode="complex")
ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="CW", encode_mode="power")

Calibration parameters

Echopype by default uses calibration parameters stored in the converted files. However, since careful calibration is often done separately from the data collection phase of the field work, accurate calibration parameters are often supplied in the post-processing stage. This can be done via passing in a dictionary cal_params containing calibration parameters needed to overwrite values stored in the raw data files like below:

ds_Sv = ed.calibrate.compute_Sv(
    ed,
    cal_params={
        sa_correction: sa_vals,
        equivalent_beam_angle: eba_vals,
    }
)

Calibration parameters are instrument specific, and currently we support the following:

EK60_CAL_DICT = {
    "sa_correction",
    "gain_correction",
    "equivalent_beam_angle",
    "angle_offset_alongship",
    "angle_offset_athwartship",
    "angle_sensitivity_alongship",
    "angle_sensitivity_athwartship",
    "beamwidth_alongship",
    "beamwidth_athwartship",
}

EK80_CAL_DICT = {
    "sa_correction",
    "gain_correction",
    "equivalent_beam_angle",
    "angle_offset_alongship",
    "angle_offset_athwartship",
    "angle_sensitivity_alongship",
    "angle_sensitivity_athwartship",
    "beamwidth_alongship",
    "beamwidth_athwartship",
    "impedance_transducer",   # z_et
    "impedance_transceiver",  # z_er
    "receiver_sampling_frequency",
}

AZFP_CAL_DICT = {
    "EL",
    "DS",
    "TVR",
    "VTX",
    "equivalent_beam_angle",
    "Sv_offset",
}

One could choose to bypass the above by updating only specific values in the EchoData object (calibration parameters are in the Sonar/Beam_groupX and the Vendor_specifc group). This may be useful if only one value needs to be updated among all channels.

Environmental parameters

Environmental parameters, including temperature, salinity, pressure, and pH, are critical in processing raw data collected by the echosounders. They influence calibration through sound speed and absorption coefficients, both central to calibrating raw data to physically meaningful quantities:

Default behavior

By default, echopype uses environmental parameters stored in the EchoData object parsed from the raw file and users need to supply any required parameters that are missing from the data file.

Echopype does not currently handle calculation based on a sound speed profile.

Formulae used

When custom environmental values are supplied, echopype calculates sound speed and absorption coefficients based on the following:

To supply custom environmetnal parameters, use input argument env_params:

env_params = {
    "temperature": 8,  # temperature in degree Celsius
    "salinity": 30,    # salinity in PSU
    "pressure": 50,    # pressure in dbar
}
ds_Sv = ep.calibrate.compute_Sv(ed, env_params=env_params)

Sanity checks

Values of the calibration and environmental parameters, as well as the resultant sound speed and absorption coefficints are stored in the output of compute_Sv for easy access and verification:

# Check environmental parameters used in compute_Sv
print(
    f"Temperature used is: {ds_Sv["temperature"]}"
    f"Salinity used is: {ds_Sv["salinity"]}",
    f"Pressure used is: {ds_Sv["pressure"]}",
)
# Check absorption coefficients computed and used
ds_Sv['sound_absorption']  # absorption coefficients [dB/m]
# Check sound speed absorption coefficients computed and used in compute_Sv
ds_Sv['sound_speed']  # sound speed [m/s]

Interfacing Echoview ECS file

Echopype is able to interface with Echoview ECS file for EK60 and EK80 echosounder starting from v0.7.0. The ECS file contains specifications for both calibration parameters and environmental parameters. When an ECS file is provided, parameters in cal_params and env_params, if provided, are ignored. To provide an ECS file, use:

ds_Sv = ep.calibrate.compute_Sv(ed, ecs_file="PATH_TO_ECS_FILE")

The vocalbury currently implemented was assembled from example files we have access to and may not include all aliases. We have connected with Echoview and will add all aliases in a future release, as well as ECS support for the AZFP echosounder.