Skip to content

TST: Parse dates with empty space (#6428) #14862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 14, 2016
Merged

TST: Parse dates with empty space (#6428) #14862

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
36bc9b7
TST: Parse dates with empty space (#6428)
mroeschke Dec 12, 2016
23a4cb1
Add link to to_datetime
jorisvandenbossche Dec 14, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions doc/source/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -867,6 +867,12 @@ data columns:
index_col=0) #index is the nominal column
df

.. note::
If a column or index contains an unparseable date, the entire column or
index will be returned unaltered as an object data type. For non-standard
datetime parsing, use :func:`to_datetime` after ``pd.read_csv``.


.. note::
read_csv has a fast_path for parsing datetime strings in iso8601 format,
e.g "2000-01-01T00:01:02+00:00" and similar variations. If you can arrange
Expand Down
4 changes: 4 additions & 0 deletions pandas/io/parsers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,10 @@
* dict, e.g. {'foo' : [1, 3]} -> parse columns 1, 3 as date and call result
'foo'

If a column or index contains an unparseable date, the entire column or
index will be returned unaltered as an object data type. For non-standard
datetime parsing, use ``pd.to_datetime`` after ``pd.read_csv``

Note: A fast-path exists for iso8601-formatted dates.
infer_datetime_format : boolean, default False
If True and parse_dates is enabled, pandas will attempt to infer the format
Expand Down
13 changes: 13 additions & 0 deletions pandas/io/tests/test_date_converters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,19 @@ def date_parser(date, time):
names=['datetime', 'prn']))
assert_frame_equal(df, df_correct)

def test_parse_date_column_with_empty_string(self):
# GH 6428
data = """case,opdate
7,10/18/2006
7,10/18/2008
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should actually parse as datetime and make the space a NaT. So this 'works' in that it doesn't convert as today's date, but it is not registering as missing here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently it looks like errors='ignore' is set when parsing datetimes with read_csv (example). I can change this to errors='coerce'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I think we did that on purpose..

that whole clause is bogus, errors='ignore' means it will never raise.

but its a bit tricky, you want to try to parse, then fallback on a softer-parse if necessary.

try playing around with this and see what happens.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah thanks for the insight. I'll tinker with it later tonight.

Copy link
Member Author

@mroeschke mroeschke Dec 13, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were correct; current tests require errors='ignore'.

This is tricky since errors='ignore' and errors='coerce' both terminate the parse with NaTs or returning the input. Would it be possible to introduce a new kwarg (e.g. date_errors) to pass ignore/coerce/raise, into to_datetime within read_csv? It could default to ignore.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to introduce a new kwarg (e.g. date_errors) to pass ignore/coerce/raise, into to_datetime within read_csv? It could default to ignore.

This would make the API even more complicated. Generally supporting non-standard datetime parsing should simply use pd.to_datetime post-read_csv.

@jorisvandenbossche any thoughts here.

Copy link
Member Author

@mroeschke mroeschke Dec 13, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point.

I could clarify in the docs that if parse_dates cannot parse a date the column is returned untouched (which seems like the current behavior) and the user can coerce a space or any unparseable date to NaT post-read_csv with to_datetime

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's prob the best soln here (pls add to doc-string & a small warning/note section in io.rst if you don't mind).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, I agree that we should not add kwargs to read_csv for something like this. If you want the date parser, your dates should just be able to be parsed with the default settings, otherwise you can just use to_datetime. PR for doc clarification certainly welcome!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR for doc clarification certainly welcome!

which you already included here :-)

621, """
result = read_csv(StringIO(data), parse_dates=['opdate'])
expected_data = [[7, '10/18/2006'],
[7, '10/18/2008'],
[621, ' ']]
expected = DataFrame(expected_data, columns=['case', 'opdate'])
assert_frame_equal(result, expected)

if __name__ == '__main__':
nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],
exit=False)
12 changes: 12 additions & 0 deletions pandas/tseries/tests/test_timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -947,6 +947,18 @@ def test_to_datetime_on_datetime64_series(self):
result = to_datetime(s)
self.assertEqual(result[0], s[0])

def test_to_datetime_with_space_in_series(self):
# GH 6428
s = Series(['10/18/2006', '10/18/2008', ' '])
tm.assertRaises(ValueError, lambda: to_datetime(s, errors='raise'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is right

result_coerce = to_datetime(s, errors='coerce')
expected_coerce = Series([datetime(2006, 10, 18),
datetime(2008, 10, 18),
pd.NaT])
tm.assert_series_equal(result_coerce, expected_coerce)
result_ignore = to_datetime(s, errors='ignore')
tm.assert_series_equal(result_ignore, s)

def test_to_datetime_with_apply(self):
# this is only locale tested with US/None locales
_skip_if_has_locale()
Expand Down