asdf
asdf copied to clipboard
asdf-format
Ghost namespace package from old installation?
@bsipocz , @saimn, and I ran into this mystery. We have an environment with asdf installed. After uninstalling it, ghost files remained in such a way that it registered as an empty namespace package.
>>> import asdf
>>> asdf.__path__
_NamespacePath(['.../asdf'])
Here is an example listing of the ghost directory in site-packages:
.../asdf/commands:
__pycache__/
.../asdf/compat:
__pycache__/
.../asdf/extern:
__pycache__/
.../asdf/__pycache__:
asdf.cpython-37-pytest-5.3.0.pyc
block.cpython-37-pytest-5.3.0.pyc
compression.cpython-37-pytest-5.3.0.pyc
constants.cpython-37-pytest-5.3.0.pyc
_convenience.cpython-37-pytest-5.3.0.pyc
_display.cpython-37-pytest-5.3.0.pyc
exceptions.cpython-37-pytest-5.3.0.pyc
extension.cpython-37-pytest-5.3.0.pyc
fits_embed.cpython-37-pytest-5.3.0.pyc
generic_io.cpython-37-pytest-5.3.0.pyc
__init__.cpython-37-pytest-5.3.0.pyc
_internal_init.cpython-37-pytest-5.3.0.pyc
reference.cpython-37-pytest-5.3.0.pyc
resolver.cpython-37-pytest-5.3.0.pyc
schema.cpython-37-pytest-5.3.0.pyc
search.cpython-37-pytest-5.3.0.pyc
stream.cpython-37-pytest-5.3.0.pyc
tagged.cpython-37-pytest-5.3.0.pyc
treeutil.cpython-37-pytest-5.3.0.pyc
type_index.cpython-37-pytest-5.3.0.pyc
types.cpython-37-pytest-5.3.0.pyc
util.cpython-37-pytest-5.3.0.pyc
version.cpython-37-pytest-5.3.0.pyc
versioning.cpython-37-pytest-5.3.0.pyc
yamlutil.cpython-37-pytest-5.3.0.pyc
.../asdf/schemas:
stsci.edu/
.../asdf/tags:
core/ __pycache__/
.../asdf/tests:
__pycache__/
In my case, I use conda and I don't remember when I first installed asdf into this environment but it was a while ago. Once I manually deleted the ghost directory, pip install asdf and pip uninstall asdf did not bring it back.
However, @saimn claims otherwise for pyenv. I'll let him elaborate here if he so desires.
I am able to reliably reproduce this with the following recipe (from my astropy root directory, which is checked out at astropy/astropy@2511c48):
conda create -n fresh-environment python==3.7
conda activate fresh-environment
pip install -e .[test]
pip install asdf
pytest --collect-only astropy/nddata
pip uninstall asdf
How bizarre! Without the pytest command, asdf uninstalls cleanly. The collected directory doesn't seem to be relevant either.
Seems to be an issue with any package that defines a pytest plugin. I see the same residue left over after uninstalling pytest_remotedata, pytest_xdist, etc.
Turns out, the .pyc files are artifacts of pytest assertion rewriting. Adding this to astropy's conftest.py:
import sys
sys.dont_write_bytecode = True
prevents the .pyc files from being written, which then allows pip to remove the asdf directory on uninstall. More details here.
could the plugin live in a separate repo?
I don't think there's any reason why it needs to live in the asdf repo/package, the plugin could be a separate package that lists asdf as a dependency. I imagine developer convenience is the strongest argument for keeping it where it is?
People might want to use asdf without wanting the plugin, so I see a reason to move it out.
Thanks for the Sherlock level investigation and detailed explanation, @eslavich !
What is the side effect of using sys.dont_write_bytecode = True? Will pytest run slower?
I timed collection of astropy on my laptop, and without sys.dont_write_bytecode, it takes ~ 23 seconds the first time and ~ 11.5 seconds on subsequent collections. With sys.dont_write_bytecode = True each collection takes ~ 12 seconds (with no up-front cost).
It must be that writing all those .pyc files is expensive? That would account for the extra time on first collection. Doesn't seem worthwhile for only a ~ 5% improvement on subsequent test runs, but maybe it's more important for smaller test suites.
I am a bit hesitant to set this for the entire astropy repo just because of this package. But it is good to have this issue open with the solution presented if we decide the ghosting is enough of a nuisance to solve down the road. Thanks!
People might want to use
asdfwithout wanting the plugin, so I see a reason to move it out.
Then they can just not use the plugin in their tests. I.e. don't use the plugin.
But using the plugin or not, the .pyc files still get left behind regardless. Perhaps it is due to entry points? If I delete the use of asdf entry points in astropy and turn off the plugin, the files still get left behind. Hmm.
