BUG: `Index.difference` is returning too many values

[michaelpradel]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

index1 = pd.period_range('2022-01-01', periods=5, freq='M')
index2 = pd.Index(['2022-02', '2022-03'])
result = index1.difference(index2)
print(result)

Issue Description

The example creates index1 with 5 months, and index2 with two months. The difference should be an index with three months. However, I'm getting the following:

PeriodIndex(['2022-01', '2022-02', '2022-03', '2022-04', '2022-05'], dtype='period[M]')

Looks like this issue was introduced by https://github.com/pandas-dev/pandas/pull/55108.

Expected Behavior

The example should print the following:

Index(['2022-02', '2022-03'], dtype='object')

Pandas 2.1.4 gives the expected behavior.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 6dbeeb4009bbfac5ea1ae2111346f5e9f05b81f4 python : 3.10.8.final.0 python-bits : 64 OS : Linux OS-release : 6.5.0-35-generic Version : #35-Ubuntu SMP PREEMPT_DYNAMIC Fri Apr 26 11:23:57 UTC 2024 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.0rc0+28.g6dbeeb4009 numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0.post0 setuptools : 63.2.0 pip : 24.0 Cython : 3.0.10 pytest : 8.2.0 hypothesis : 6.100.2 sphinx : 7.3.7 blosc : None feather : None xlsxwriter : 3.2.0 lxml.etree : 5.2.1 html5lib : 1.1 pymysql : 1.4.6 psycopg2 : 2.9.9 jinja2 : 3.1.3 IPython : 8.24.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : 1.3.8 fastparquet : 2024.2.0 fsspec : 2024.3.1 gcsfs : 2024.3.1 matplotlib : 3.8.4 numba : 0.59.1 numexpr : 2.10.0 odfpy : None openpyxl : 3.1.2 pyarrow : 16.0.0 pyreadstat : 1.2.7 python-calamine : None pyxlsb : 1.0.10 s3fs : 2024.3.1 scipy : 1.13.0 sqlalchemy : 2.0.29 tables : 3.9.2 tabulate : 0.9.0 xarray : 2024.3.0 xlrd : 2.0.1 zstandard : 0.22.0 tzdata : 2024.1 qtpy : None pyqt5 : None