np.nan comparison to pd.NA Equality Semantics differ elementwise in IntervalArray

[WillAyd]

Discovered in #31799

I think this is probably a deeper rooted problem than just IntervalArray but not sure how else to describe. @jorisvandenbossche maybe has thoughts

>>> from pandas.core.arrays import IntervalArray
>>> arr = IntervalArray.from_arrays([0., 1., 2., np.nan], [1., 2., 3., np.nan])
>>> arr == pd.NA
array([False, False, False, False])
>>> arr[-1] == pd.NA
<NA>

[jorisvandenbossche]

Yes, also some other array types such as Categorical or DatetimeArray have this behaviour.

In the pd.NA object's ops, we handle np.ndarray, so we get behaviour like this:

In [26]: np.array([1, 2]) == pd.NA 
Out[26]: array([<NA>, <NA>], dtype=object)

See

pandas/pandas/_libs/missing.pyx

Lines 308 to 310 in 50ebb24

	elif isinstance(other, np.ndarray):
	out = np.empty(other.shape, dtype=object)
	out[:] = NA

but this doesn't handle our own array types.

I think we need to decide where this should be handled (should pd.NA keep returning NotImplemented to have it handled correctly by the array object, or should pd.NA be able to handle those array types?) In general, handling this on the array might be better, since the output dtype might differ depending on the exact operation.

---

In general, we basically didn't put much effort in making pd.NA already working with array types that don't use this as the missing value indicator (eg DatetimeArray - pd.NA also raises TypeError). Also because it is not always clear what the result should be. Eg we might decide that DatetimeArray - pd.NA should result in TimedeltaArray, but TimedeltaArray cannot yet hold pd.NA, so returning that is not yet possible.