pytest closes io.BytesIO objects innappropiately (using pandas to_parquet)

[EHYPI]

• [x] a detailed description of the bug or problem you are having
• [x] output of pip list from the virtual environment you are using
• [x] pytest and operating system versions
• [x] minimal example if possible

Description

If I write a pandas dataframe to a BytesIO object that object should remain readable. Indeed OUTSIDE of a pytest session it does. However inside of a pytest session it gets closed.

MRE

import pytest
import pandas as pd
from io import BytesIO


def test_bytes_io_bug():
    df = pd._testing.makeDataFrame()
    _buffer = BytesIO()
    df.to_parquet(_buffer)
    assert _buffer.readable(), "PYTEST python 3.11 bug, erroneously closes buffers while in pytest"
    print("no failure")

generated error

 xxx@yyy:~/$ pytest "test_bytesIO_bug.py"
================================================================================================================================================ test session starts =================================================================================================================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: ./
plugins: rerunfailures-13.0, cov-4.1.0
collected 1 item                                                                                                                                                                                                                                                                                                     

../../../../test_bytesIO_bug.py F                                                                                                                                                                                                              [100%]

====================================================================================================================================================== FAILURES ======================================================================================================================================================
_________________________________________________________________________________________________________________________________________________ test_bytes_io_bug __________________________________________________________________________________________________________________________________________________

    def test_bytes_io_bug():
        """
        pip show pandas
        Name: pandas
        Version: 2.1.3
    
        pip show pytest
        Name: pytest
        Version: 7.4.3
    
        python --version
        Python 3.11.6
        """
        df = pd._testing.makeDataFrame()
        _buffer = BytesIO()
        df.to_parquet(_buffer)
>       assert _buffer.readable(), "PYTEST python 3.11 bug, erroneously closes buffers while in pytest"
E       ValueError: I/O operation on closed file.

test_bytesIO_bug.py:22: ValueError
============================================================================================================================================== short test summary info ===============================================================================================================================================
FAILED ../../../../test_bytesIO_bug.py::test_bytes_io_bug - ValueError: I/O operation on closed file.
================================================================================================================================================= 1 failed in 0.83s ==================================================================================================================================================

System info

relevant versions

pip show pandas
Name: pandas
Version: 2.1.3

pip show pytest
Name: pytest
Version: 7.4.3

python --version
Python 3.11.6

cat /etc/os-release
NAME="Ubuntu" 
VERSION="20.04.5 LTS (Focal Fossa)"

NOTE: this is WSL2, but error also occurs on latest MAC OS

Full pip list

Package                  Version
------------------------ ------------
asttokens                2.4.1
boto3                    1.33.5
botocore                 1.33.5
certifi                  2023.11.17
cffi                     1.16.0
charset-normalizer       3.3.2
clarabel                 0.6.0
coverage                 7.3.2
cramjam                  2.7.0
cryptography             41.0.7
decorator                5.1.1
ecos                     2.0.12
executing                2.0.1
fastparquet              2023.10.1
fsspec                   2023.10.0
idna                     3.6
iniconfig                2.0.0
ipython                  8.18.1
jedi                     0.19.1
Jinja2                   3.1.2
jmespath                 1.0.1
joblib                   1.3.2
MarkupSafe               2.1.3
matplotlib-inline        0.1.6
moto                     4.2.10
numpy                    1.26.2
osqp                     0.6.3
packaging                23.2
pandas                   2.1.3
parso                    0.8.3
peppercorn               0.6
pexpect                  4.9.0
pip                      23.2.1
pluggy                   1.3.0
prompt-toolkit           3.0.41
ptyprocess               0.7.0
pure-eval                0.2.2
pyarrow                  14.0.1
pybind11                 2.11.1
pycparser                2.21
pycryptodome             3.19.0
Pygments                 2.17.2
pyparsing                3.1.1
pytest                   7.4.3
pytest-cov               4.1.0
pytest-rerunfailures     13.0
python-dateutil          2.8.2
pytz                     2023.3.post1
PyYAML                   6.0.1
qdldl                    0.1.7.post0
requests                 2.31.0
responses                0.24.1
s3fs                     0.4.2
s3transfer               0.8.2
scikit-learn             1.3.2
scipy                    1.11.4
scs                      3.2.4.post1
setuptools               65.5.0
six                      1.16.0
stack-data               0.6.3
threadpoolctl            3.2.0
traitlets                5.14.0
tzdata                   2023.3
urllib3                  2.0.7
wcwidth                  0.2.12
Werkzeug                 3.0.1
xmltodict                0.13.0

[nicoddemus]

Thanks for the example!

I cannot reproduce this on Windows (same pytest and pandas version, Python 3.11.3).

[EHYPI]

Thanks for the example!

I cannot reproduce this on Windows (same pytest and pandas version, Python 3.11.3).

Same pip list? We've got it reproduced in MAC and WSL.

[onsh]

I virtually reproduced your environment and the test passed without any issues. If you could give me more details about the error message you received, I might be able to help you better.

Here is the information about my environment:

• Ubuntu 22.04.3 LTS on Windows with WSL2
• Python 3.11.6
• pytest 7.4.3
• pandas 2.1.4

(bugfix) onsh@XXX:~/sandbox/bugfix20231215$ pytest test_bytesIO_bug.py 
====================================================================== test session starts =======================================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/onsh/sandbox/bugfix20231215
collected 1 item                                                                                                                                             

test_bytesIO_bug.py .                                                                                                                                  [100%]

===================================================================== 1 passed in 0.46s ======================================================================

To make it easier to troubleshoot the issue, I installed the minimum configuration using pip install. The following libraries and their dependencies were installed: pytest, pandas, pyarrow, and fastparquet.

(bugfix) onsh@XXX:~/sandbox/bugfix20231215$ pip list
Package         Version
--------------- ------------
cramjam         2.7.0
fastparquet     2023.10.1
fsspec          2023.12.2
iniconfig       2.0.0
numpy           1.26.2
packaging       23.2
pandas          2.1.4
pip             23.3.1
pluggy          1.3.0
pyarrow         14.0.1
pytest          7.4.3
python-dateutil 2.8.2
pytz            2023.3.post1
setuptools      69.0.2
six             1.16.0
tzdata          2023.3

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity.

[obestwalter]

Stale for a long time and no-follow up, just helping the bot here a bit and closing manually.