BUG: `groupby.agg` with `as_index=False` and duplicate columns in `by` fails with `pd.NamedAgg`.

[sfc-gh-rdurrani]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame(
           {
               "col1": [2, 1, 1, 0, 2, 0],
               "col2": [4, 5, 36, 7, 4, 5],
               "col3": [3.1, 8.0, 12, 10, 4, 1.1],
               "col4": [17, 3, 16, 15, 5, 6],
               "col5": [-1, 3, -1, 3, -2, -1],
           }
)

df.groupby(by=["col1", "col1", "col2"], as_index=False).agg({"col1": [min, max], "col2": "count"}) # works
df.groupby(by=["col1", "col1", "col2"], as_index=False).agg(new_col=pd.NamedAgg('col1', min), new_col1=pd.NamedAgg('col1', max), new_col2=('col2', 'count')) # Fails

Issue Description

df.groupby with duplicate columns in by works with agg when a dictionary is passed in for aggregation function, but fails when pd.NamedAgg is used to provide aggregation functions.

Expected Behavior

Should work, and have a similar output as passing in a dictionary.

Installed Versions

INSTALLED VERSIONS

commit : bdc79c146c2e32f2cab629be240f01658cfb6cc2 python : 3.10.14.final.0 python-bits : 64 OS : Darwin OS-release : 23.4.0 Version : Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.1 numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0.post0 setuptools : 68.2.2 pip : 23.3.1 Cython : None pytest : 7.4.4 hypothesis : None sphinx : 5.0.2 blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.3 IPython : 8.23.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : 2024.3.1 gcsfs : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 16.0.0 pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.13.0 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None

[rhshadrach]

Thanks for the report. You checked the box I have confirmed this bug exists on the main branch of pandas. I am seeing the following output on main:

print(df.groupby(by=["col1", "col1", "col2"], as_index=False).agg(new_col=pd.NamedAgg('col1', min), new_col1=pd.NamedAgg('col1', max), new_col2=('col2', 'count')))
#    col1  col2  new_col  new_col1  new_col2
# 0     0     5        0         0         1
# 1     0     7        0         0         1
# 2     1     5        1         1         1
# 3     1    36        1         1         1
# 4     2     4        2         2         2

Can you confirm if you are seeing an error or the above output on main?

[sfc-gh-rdurrani]

I can't seem to install pandas from source - I keep getting the following error:

ModuleNotFoundError: No module named 'pandas._libs.pandas_parser'

but I did notice that the bug is happening on the latest version of pandas - 2.2.2.

[VISWESWARAN1998]

@sfc-gh-rdurrani Your code runs fine without any errors in main branch for me.

I can't seem to install pandas from source - I keep getting the following error:

ModuleNotFoundError: No module named 'pandas._libs.pandas_parser'

but I did notice that the bug is happening on the latest version of pandas - 2.2.2.

Try creating an isolated environment using mamba (recommended) or pip, install the requirements in requirements-dev.txt and try installing pandas from source. More information here: https://pandas.pydata.org/docs/dev/development/contributing_environment.html

[rhshadrach]

Thanks @sfc-gh-rdurrani - there isn't a requirement to try it out on main (but you're certainly welcome to!). I just wanted to make sure I wasn't making a mistake when I see this is fixed on the main branch.

[jasonmokk]

take