Skip to content

FloatingPointError on ewm().std() #13529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chrism2671 opened this issue Jun 29, 2016 · 3 comments
Closed

FloatingPointError on ewm().std() #13529

chrism2671 opened this issue Jun 29, 2016 · 3 comments
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Usage Question

Comments

@chrism2671
Copy link

I'm not sure if this is a bug, but I'm pretty sure I shouldn't be seeing these warnings. Any clarification would be a real help, as I need to ensure the accuracy of the calculations in all circumstances (including situations where the deviation is effectively zero)

Paste-ready code:

import traceback
import warnings
import sys
import pandas as pd
import numpy as np
np.seterr(all='raise')
def warn_with_traceback(message, category, filename, lineno, file=None, line=None):
    traceback.print_stack()
    log = file if hasattr(file,'write') else sys.stderr
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback

pd.Series(np.random.randn(50)).pct_change().ewm(span=35, min_periods=35).std()

Expected output:
pd.Series of exponentially weighted standard deviation.

output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.2.0-34-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.2
setuptools: 23.0.0
Cython: 0.24
numpy: 1.10.4
scipy: 0.17.1
statsmodels: 0.6.1
xarray: None
IPython: 4.2.0
sphinx: 1.4.1
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.2
lxml: 3.6.0
bs4: 4.4.0
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: 0.2.1

@jreback
Copy link
Contributor

jreback commented Jun 29, 2016

you are overriding pandas by doing this: np.seterr(all='raise')
pandas by definition will catch things like operating on nans and do the right thing. see #13109
so in theory a user could eventually change this. But to be honest its not very useful.

In [3]: np.random.seed(1234)

In [4]: pd.options.display.max_rows=8

In [5]: pd.Series(np.random.randn(50)).pct_change().ewm(span=35, min_periods=35).std()
Out[5]: 
0           NaN
1           NaN
2           NaN
3           NaN
        ...    
46    27.070401
47    26.253511
48    27.533031
49    26.703128
dtype: float64

@jreback jreback closed this as completed Jun 29, 2016
@jreback jreback added Usage Question Numeric Operations Arithmetic, Comparison, and Logical operations labels Jun 29, 2016
@chrism2671
Copy link
Author

Thanks, that's useful insight. I just wanted to make sure I wasn't seeing something unexpected.

@chrism2671
Copy link
Author

One further question. The reason I was investigating this is because this warning is still getting thrown out when using the ThreadPoolExecutor to do this calculation (concurrent.futures), as:

/projects/anaconda3/lib/python3.5/site-packages/pandas/core/ops.py:716: RuntimeWarning: invalid value encountered in less
  result = getattr(x, name)(y)

I do believe the numbers are coming out as expected (I haven't checked meticulously), but did not want to suppress this warning without at least checking where it's coming from.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Usage Question
Projects
None yet
Development

No branches or pull requests

2 participants