BUG: Series.corr/cov raising with masked dtype #51422
Conversation
lukemanley
added
Bug
ExtensionArray
| from scipy import stats | ||
|
|
||
| datetime_series = datetime_series.astype(dtype) |
There was a problem hiding this comment.
what happens here if there are NAs?
There was a problem hiding this comment.
Both left and right are filtered for notna here:
Line 1600 in 95a087d
so this works:
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: ser1 = pd.Series(np.random.randn(100), dtype="Float64")
In [4]: ser2 = pd.Series(np.random.randn(100), dtype="Float64")
In [5]: ser1[1] = pd.NA
In [6]: ser2[5:7] = pd.NA
In [7]: ser1.corr(ser2)
Out[7]: 0.09774253881093414
There was a problem hiding this comment.
cool. what about if there is an nan?
There was a problem hiding this comment.
Do you mean an nan within a masked array? e.g.
In [8]: ser1[1] = 0.0
In [9]: ser1 /= (ser1 != 0)
In [10]: ser1
Out[10]:
0 0.148073
1 NaN
2 0.556972
3 -0.554886
4 1.216938
...
95 0.33919
96 0.528683
97 1.590215
98 0.84015
99 0.333666
Length: 100, dtype: Float64
In [11]: ser1.corr(ser2)
Out[11]: 0.09774253881093414
It works either way since the notna is applied to the ndarray which will capture both np.nan and pd.NA
doc/source/whatsnew/v2.0.0.rst
Outdated
| @@ -1230,6 +1230,7 @@ Numeric | |||
| - Bug in arithmetic operations on :class:`Series` not propagating mask when combining masked dtypes and numpy dtypes (:issue:`45810`, :issue:`42630`) | |||
| - Bug in :meth:`DataFrame.sem` and :meth:`Series.sem` where an erroneous ``TypeError`` would always raise when using data backed by an :class:`ArrowDtype` (:issue:`49759`) | |||
| - Bug in :meth:`Series.__add__` casting to object for list and masked :class:`Series` (:issue:`22962`) | |||
| - Bug in :meth:`Series.corr` and :meth:`Series.cov` raising ``AttributeError`` for masked dtypes (:issue:`51422`) | |||
pandas/core/series.py
Outdated
| @@ -2702,8 +2705,10 @@ def cov( | |||
| this, other = self.align(other, join="inner", copy=False) | |||
| if len(this) == 0: | |||
| return np.nan | |||
| this_values = np.asarray(this.values) | |||
There was a problem hiding this comment.
Yes I think so. For my knowledge, if we need an ndarray for nanops.nancov, is there a benefit to using np.asarray(this._values) vs np.asarray(this.values)?
There was a problem hiding this comment.
its probably harmless in this case but in general .values can be either a) lossy (dt64tz) or b) convert to object (period) while ._values is safe from those failure modes
There was a problem hiding this comment.
got it, thanks! I'll update just in case
There was a problem hiding this comment.
values also creates references and is read_only (in the future) for CoW. Better to use ._values
|
thx @lukemanley |
doc/source/whatsnew/v2.0.0.rstfile if fixing a bug or adding a new feature.