Fix loading hdf5 files with new versions of pandas #514
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I think I have found all of the places where we pass the format kwarg to pandas.read_hdf(), which is not allowed anymore by new versions of pandas. This also removes commented-out lines that I had left in an earlier commit.
nickponvert
requested review from
matchings,
dougollerenshaw,
mochic and
jeromelecoq
|
@nickponvert do both of the notebooks run with the newer version of pandas? specifically, 1) the notebook that loads the dataset object, and 2) the notebook that reads the multi session dataframes, such as the one here: "\allen\programs\braintv\workgroups\nc-ophys\visual_behavior\visual_behavior_production_analysis\multi_session_summary_dfs\mean_flashes_image_name_engaged_df.h5" |
|
Both of the notebooks Yazan sent us work after this fix. One of them uses Edit: I had to change a few lines in the second notebook that were calling pandas.read_hdf() with the format kwarg |
|
Ok great, as long as those work, I am good with this change. |
matchings
approved these changes
Apr 11, 2019
|
Circle CI tests are failing. |
|
Those linting issues are probably my fault, I will try to fix them now. |
nickponvert
added a commit
that referenced
this pull request
Apr 11, 2019
This addresses issue #513 and makes PR #514 unnecessary. For future reference, here's the explanation for the #513 fix: This PR addresses issue #513 by changing all of the calls to pandas.read_hdf() to no longer pass 'format' as a kwarg. We used to do this, and the passed kwarg was ignored by HDFStore.open because it wasn't a defined keyword arg for that method, but no alarms were raised. See: pandas-dev/pandas#13291 In newer versions of pandas there is an exception raised if you try to pass 'format' to the read_hdf() method. Removing this kwarg: - Won't change the output of read_hdf, as the kwarg was being ignored before, and - Allows me to run Marina's old notebooks with VBA analysis code and pandas==0.24.2
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses issue #513 by changing all of the calls to
pandas.read_hdf()to no longer pass 'format' as a kwarg. We used to do this, and the passed kwarg was ignored by HDFStore.open because it wasn't a defined keyword arg for that method, but no alarms were raised.See: pandas-dev/pandas#13291
In newer versions of pandas there is an exception raised if you try to pass 'format' to the
read_hdf()method. Removing this kwarg:read_hdf, as the kwarg was being ignored before, andpandas==0.24.2TODO: