Invalid argument error with libnetcdf 4.5.0 but not 4.4.1

[spencerkclark]

Per the recommendation of @dopplershift (cc: @spencerahill), I am migrating an issue (Unidata/netcdf4-python#742) initially posted in netcdf4-python over to here.

Environment Information

• I have reproduced this behavior on my local machine (using Mac OS version 10.13)
• This behavior was first identified with our test suite run using Travis CI (see here for the platform information)
• On both platforms libnetcdf was installed from the conda-forge channel (e.g. conda install -c conda-forge libnetcdf).

Summary of Issue

Since libnetcdf version 4.5.0 appeared on conda-forge, we have encountered some puzzling errors in our CI test suite. Namely, some files that we used to be able to open with xarray using netCDF4 now lead to errors like the following:

aospy/test/test_calc_basic.py:60: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
aospy/test/test_calc_basic.py:35: in _test_files_and_attrs
    _test_output_attrs(calc, dtype_out)
aospy/test/test_calc_basic.py:18: in _test_output_attrs
    data = xr.open_dataset(calc.path_out[dtype_out])
../../../miniconda/envs/test_env/lib/python3.6/site-packages/xarray/backends/api.py:282: in open_dataset
    autoclose=autoclose)
../../../miniconda/envs/test_env/lib/python3.6/site-packages/xarray/backends/netCDF4_.py:210: in __init__
    self.ds = opener()
../../../miniconda/envs/test_env/lib/python3.6/site-packages/xarray/backends/netCDF4_.py:185: in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
netCDF4/_netCDF4.pyx:2015: in netCDF4._netCDF4.Dataset.__init__
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   OSError: [Errno -36] NetCDF: Invalid argument: b'/home/travis/build/spencerahill/aospy/aospy/test/data/objects/test-files/example_proj/example_model/example_run/condensation_rain/condensation_rain.ann.av.from_monthly_ts.example_model.example_run.0004-0006.nc'

In doing some investigation offline, it seems that the command line tool ncdump provided by libnetcdf version 4.5.0 is also unable to open these files. E.g. I get the following response:

$ ncdump sphum.ann.reg.av.vert_int.from_monthly_ts_sigma.example_model.example_run.0006.nc
ncdump: sphum.ann.reg.av.vert_int.from_monthly_ts_sigma.example_model.example_run.0006.nc: NetCDF: Invalid argument

However, using libnetcdf version 4.4.1.1, ncdump is able to handle the file (and using version 4.4.1.1 also enables downstream tools like netcdf4-python to work):

$ ncdump -h sphum.ann.reg.av.vert_int.from_monthly_ts_sigma.example_model.example_run.0006.nc
netcdf sphum.ann.reg.av.vert_int.from_monthly_ts_sigma.example_model.example_run.0006 {
variables:
	double raw_data_start_date ;
		raw_data_start_date:units = "days since 1673-01-01" ;
		raw_data_start_date:long_name = "time axis boundaries" ;
		raw_data_start_date:calendar = "NOLEAP" ;
		raw_data_start_date:_FillValue = 1.e+20 ;
	int subset_end_date ;
		subset_end_date:units = "days since 1678-01-31 00:00:00" ;
		subset_end_date:calendar = "proleptic_gregorian" ;
	double globe ;
		globe:units = "(vertical integral of quantity with unspecified units)" ;
		globe:_FillValue = NaN ;
		globe:description = "specific humidity" ;
	double raw_data_end_date ;
		raw_data_end_date:units = "days since 1673-01-01" ;
		raw_data_end_date:long_name = "time axis boundaries" ;
		raw_data_end_date:calendar = "NOLEAP" ;
		raw_data_end_date:_FillValue = 1.e+20 ;
	int subset_start_date ;
		subset_start_date:units = "days since 1678-01-01 00:00:00" ;
		subset_start_date:calendar = "proleptic_gregorian" ;

// global attributes:
		:coordinates = "raw_data_start_date subset_end_date raw_data_end_date subset_start_date" ;
}

I have posted a link to this file as an example. It would be helpful to know if you can reproduce the issues we've encountered here with files like the one provided, and what about them led to this problem. Thanks in advance for your help!

Steps to reproduce the behavior

Using the latest version of libnetcdf from the conda-forge channel, download the linked file and try to run ncdump.

[dopplershift]

I can reproduce with conda-forge libnetcdf 4.5.0 on my osx machine as well.

[WardF]

I believe we have fixed this with a pull to revert to 4.4.1.1 behavior based on a bug report issued by @czender. Let me confirm and follow up with specifics.

[WardF]

It is not corrected, looking at it now.

[WardF]

Ok, the issue relates to how padding is being calculated in v1hpg.c:v1h_get_NC_string(). This code was contributed to us by @wkliao so taking a bit to read it and understand it.

[spencerkclark]

Many thanks for investigating this @dopplershift and @WardF. We have no issues sticking with version 4.4.1.1 for now, but we just wanted to bring this to your attention (also in case it impacts anyone else).

[dopplershift]

Thanks for the report!

[wkliao]

FYI. @WardF is referring to PR #332, which checks the null-byte padding in file header.

After reading this PR again, I found that returning error code NC_ENOTNC is better than NC_EINVAL in this case, as the error message corresponds to NC_ENOTNC is "NetCDF: Unknown file format", which includes the case when file fails to conform with the classic CDF file format specification. In this case, it is the null-byte padding.

FYI. PnetCDF returns NC_ENOTNC.
@WardF, let me know if you agree with this change. I can create a PR.

[dopplershift]

But...the file is a valid, so I'm not clear on why it's an error at all.

[WardF]

@wkliao I can make that change if need be but I need to determine why the file is invalid and, if it is, is it invalid in a way that contradicts the spec but became valid through common usage and the spec not being enforced historically. I will follow up when I get in to the office and can inspect the file more closely.

[wkliao]

I guess the nc file in question violates only the null-byte padding requirement by the CDF format specification. The reason it can be read by ncdump of earlier versions is those ncdump programs (and most likely other NetCDF utility programs) fail to check null-byte padding.

I consider this is not a serious bug in NetCDF library and its utilities, but strictly speaking the file does NOT conform with the CDF specification.

FYI, the files generated by NetCDF with non-null byte padding will get NC_ENOTNC from PnetCDF. Because PnetCDF was developed independently from NetCDF, sometimes PnetCDF conforms with the CDF specification more strictly. In fact, as early as the first release of PnetCDF 0.8.0, this null-byte padding checking has been implemented.

One possible solution is to develop a new utility program to detect and fix this padding for nc files.

CCed to @roblatham00

[wkliao]

Forgot to mention that there is a PnetCDF utility program named ncmpivalid. It can be used to validate a CDF file and report the exact location that fails to conform with the file specification.
https://trac.mcs.anl.gov/projects/parallel-netcdf/browser/tags/v1-9-0-pre1/src/utils/ncmpivalid

[WardF]

Given that netCDF hasn't enforced this up to this point, and that apparently enforcing this causes some issues with extant workflows, I am wondering if we need to revert this. I really want to avoid causing disruption to our users, especially when the practical positive impact seems to be limited. It can be reimplemented once we have an easy way for users to fix otherwise valid files, and will be the least disruptive to existing workflows, etc.

[WardF]

Alternatively this could be made an option instead of removed altogether, and have it turned off for now. I would also add a more specific error message, NC_EHDRPAD or some such, to make it clear what the issue is if the option is turned on. Just thinking out loud.

--enable-enforce-null-byte-padding perhaps.