BUG: NaT in Timestamp ignored by sort_values with na_position='last'

[mficek]

Code Sample, a copy-pastable example if possible

This is an excerpt from tests/frame/test_sorting.py:

from pandas import (DataFrame, Timestamp, NaT)
from pandas.util.testing import assert_frame_equal

# test_sort_nat_values_in_int_column(self):
# GH 14922: "sorting with large float and multiple columns incorrect"

float_values = (2.0, -1.797693e308)
# and now check if NaT is still considered as "na" for datetime64
# columns:
df = DataFrame(dict(datetime=[Timestamp("2016-01-01"), NaT],
                    float=float_values), columns=["datetime", "float"])

df_reversed = DataFrame(dict(datetime=[NaT, Timestamp("2016-01-01")],
                             float=float_values[::-1]),
                        columns=["datetime", "float"],
                        index=[1, 0])

df_sorted = df.sort_values(["datetime", "float"], na_position="first")
assert_frame_equal(df_sorted, df_reversed)

df_sorted = df.sort_values(["datetime", "float"], na_position="last")
assert_frame_equal(df_sorted, df_reversed)

Output

Traceback (most recent call last):
  File "/home/mficek/repos/pandas-mficek/pandas/tests/frame/bugreport.py", line 24, in <module>
    assert_frame_equal(df_sorted, df_reversed)
  File "/home/mficek/repos/pandas-mficek/pandas/util/testing.py", line 1363, in assert_frame_equal
    obj='{0}.index'.format(obj))
  File "/home/mficek/repos/pandas-mficek/pandas/util/testing.py", line 939, in assert_index_equal
    obj=obj, lobj=left, robj=right)
  File "pandas/_libs/testing.pyx", line 59, in pandas._libs.testing.assert_almost_equal
  File "pandas/_libs/testing.pyx", line 173, in pandas._libs.testing.assert_almost_equal
  File "/home/mficek/repos/pandas-mficek/pandas/util/testing.py", line 1102, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.index are different

DataFrame.index values are different (100.0 %)
[left]:  Int64Index([0, 1], dtype='int64')
[right]: Int64Index([1, 0], dtype='int64')

Problem description

I'm working on #17111 trying to change sorting algorithm for better performance. I noticed that in tests sorting with NaT, interpreted as int, should ignore na_position argument in sort_values function. But not for datetime64 type, which should consider NaT as not-a-number value.

Therefore the output above is "expected" for me, because I think sort_values simply does not consider NaT in datetime64 type as nan.

In my opinion, there should be the following change in the test suite:

df_sorted = df.sort_values(["datetime", "float"], na_position="last")
assert_frame_equal(df_sorted, df)

which however means, that version 0.20.3 wouldn't pass.

Am I wrong? Do I miss something here, please?

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-87-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 3.1.3
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.1
scipy: None
xarray: None
IPython: 5.4.1
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None

[mficek]

Hm... after some effort I accidentally found this behaviour to be a bug already issued: #16995