BUG: Fix Series.reindex losing values when reindexing to MultiIndex

[Roline-Stapny]

• closes #60923
• [X ] Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v3.0.0.rst file if fixing a bug or adding a new feature.

##Series.reindex()

#Before

# Create a Series with a named Index
series = pd.Series([26.73, 24.255], index=pd.Index([81, 82], name='a'))

# Create a MultiIndex with level names 'a', 'b', 'c'
target = pd.MultiIndex.from_product(
    [[81, 82], [np.nan], ["2018-06-01", "2018-07-01"]], 
    names=["a", "b", "c"]
)

# This would incorrectly set all values to NaN
series.reindex(target)
# a   b    c         
# 81  NaN  2018-06-01   NaN
#          2018-07-01   NaN
# 82  NaN  2018-06-01   NaN
#          2018-07-01   NaN

# But this works correctly
series.reindex(target, level="a")
# a   b    c         
# 81  NaN  2018-06-01    26.73
#          2018-07-01    26.73
# 82  NaN  2018-06-01    24.255
#          2018-07-01    24.255

#After

# Same setup as before
series = pd.Series([26.73, 24.255], index=pd.Index([81, 82], name='a'))
target = pd.MultiIndex.from_product(
    [[81, 82], [np.nan], ["2018-06-01", "2018-07-01"]], 
    names=["a", "b", "c"]
)

# Now both produce the same correct result
series.reindex(target)  # Automatically detects level='a'
# a   b    c         
# 81  NaN  2018-06-01    26.73
#          2018-07-01    26.73
# 82  NaN  2018-06-01    24.255
#          2018-07-01    24.255

##Datafram.reindex()

df = pd.DataFrame({
    'value': [26.73, 24.255],
    'other': ['A', 'B']
}, index=pd.Index([81, 82], name='a'))

target = pd.MultiIndex.from_product(
    [[81, 82], [np.nan], ["2018-06-01", "2018-07-01"]], 
    names=["a", "b", "c"]
)

Before

df.reindex(index = target)
                   value other
a  b   c
81 NaN 2018-06-01    NaN   NaN
       2018-07-01    NaN   NaN
82 NaN 2018-06-01    NaN   NaN
       2018-07-01    NaN   NaN

After

df.reindex(index = target)
                    value other
a  b   c
81 NaN 2018-06-01  26.730     A
       2018-07-01  26.730     A
82 NaN 2018-06-01  24.255     B
       2018-07-01  24.255     B

[mroeschke]

Does DataFrame.reindex also need the same handling?

[Roline-Stapny]

• reviewers

Does DataFrame.reindex also need the same handling?

Yes, Dataframe with single index is having the same issue

df = pd.DataFrame({
    'value': [26.73, 24.255],
    'other': ['A', 'B']
}, index=pd.Index([81, 82], name='a'))

# Create a MultiIndex with level names 'a', 'b', 'c'
target = pd.MultiIndex.from_product(
    [[81, 82], [np.nan], ["2018-06-01", "2018-07-01"]], 
    names=["a", "b", "c"]
)



df.reindex(target)
                   value other
a  b   c
81 NaN 2018-06-01    NaN   NaN
       2018-07-01    NaN   NaN
82 NaN 2018-06-01    NaN   NaN
       2018-07-01    NaN   NaN

df.reindex(target, level="a")
                    value other
a  b   c
81 NaN 2018-06-01  26.730     A
       2018-07-01  26.730     A
82 NaN 2018-06-01  24.255     B
       2018-07-01  24.255     B

How its the same scenario for multiindex, reindex only works if all index are matching. Infact specifying level for multiIndex dataframe is raising TypeError

raise TypeError("Join on level between two MultiIndex objects is ambiguous")
TypeError: Join on level between two MultiIndex objects is ambiguous

    source_idx = pd.MultiIndex.from_product(
        [[81, 82], ["2018-06-01"]],
        names=["a", "c"]
    )
    df = pd.DataFrame(
        {"value": [26.73, 24.255]},
        index=source_idx
    )

    # Create target with same level names but different structure
    target_idx = pd.MultiIndex.from_product(
        [[81, 82], [np.nan], ["2018-06-01", "2018-07-01"]],
        names=["a", "b", "c"]
    )

    
>>> df.reindex(target_idx)  # Reindexing doesnt copy matching index values
                   value
a  b   c
81 NaN 2018-06-01    NaN
       2018-07-01    NaN
82 NaN 2018-06-01    NaN
       2018-07-01    NaN

Reindex MultiIndex dataframe works iff all indexes match.

I will leave the multiIndex dataframe functionality as is and address the issue in single index dataframe like the example above. lmk what you think.

[Roline-Stapny]

@mroeschke could you please review it when you get a chance