BUG: MultiIndex.unique incorrect when NA value is present

[jbrockmendel]

drop_duplicates looks right, while unique (and _get_unique_index) keep a duplicate entry containing an NA value. Example based on test_intersection_with_missing_values_on_both_sides. The analogous problem crops up for every entry in nulls_fixture

nulls_fixture = np.nan
mi1 = pd.MultiIndex.from_arrays([[3, nulls_fixture, 4, nulls_fixture], [1, 2, 4, 2]])

>>> mi1.unique()
MultiIndex([(3.0, 1),
            (nan, 2),
            (4.0, 4),
            (nan, 2)],
           )

>>> mi1.drop_duplicates()
MultiIndex([(3.0, 1),
            (nan, 2),
            (4.0, 4)],
           )

[jbrockmendel]

@realead i think one of your recent PRs closes this, not sure which one

[realead]

It was closed with #41952

[realead]

The difference is , that mi1.unique operates on self.values (

pandas/pandas/core/base.py

Line 969 in 648eb40

def unique(self):

)

(i.e. [(3.0, 1) (nan, 2) (4.0, 4) (nan, 2)]) , while drop_duplicates operates on self.codes

pandas/pandas/core/indexes/multi.py

Line 1592 in 648eb40

ids = get_group_index(self.codes, shape, sort=False, xnull=False)

where nans are replaced by -1.

This explains why drop_duplicates worked, but not unique.