BUG: infer_dtype_from_scalar infers nanosecond dtype for second input

[MarcoGorelli]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In [1]: from pandas.core.dtypes.cast import infer_dtype_from_scalar

In [2]: infer_dtype_from_scalar(pd.Timestamp('2020-01-01').as_unit('s'))
Out[2]: (dtype('<M8[ns]'), numpy.datetime64('2020-01-01T00:00:00.000000000'))

Issue Description

nanosecond resolution is inferred for second input

Expected Behavior

Out[2]: (dtype('<M8[s]'), numpy.datetime64('2020-01-01T00:00:00'))`

Installed Versions

INSTALLED VERSIONS

commit : b2a26ec
python : 3.11.1.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.0.0.dev0+1430.gb2a26ecc53
numpy : 1.25.0.dev0+446.ga701e87ce
pytz : 2022.7
dateutil : 2.8.2
setuptools : 65.7.0
pip : 22.3.1
Cython : 0.29.33
pytest : 7.2.0
hypothesis : 6.52.1
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.8.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.0
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.0.dev0+1336.f5f64c5
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : None
qtpy : None
pyqt5 : None
None

[MarcoGorelli]

This may have unintended consequences here:

pandas/pandas/core/indexes/interval.py

Lines 545 to 549 in 1c17d94

	key_dtype, key_i8 = infer_dtype_from_scalar(key, pandas_dtype=True)
	if lib.is_period(key):
	key_i8 = key.ordinal
	elif isinstance(key_i8, Timestamp):
	key_i8 = key_i8.value

Test which hits these lines: pytest pandas/tests/indexes/interval/test_interval.py::TestIntervalIndex::test_maybe_convert_i8