rdkit
rdkit copied to clipboard
rdkit
Get pytest to run without incurring in a circular import when RDKit Python modules are installed out-of-tree
Since we moved to pytest I was not able to run pythonSourceTests in my out-of-tree installation because of this error:
$ ctest -I 284,284 -V
[...]
test 284
Start 284: pythonSourceTests
284: Test command: /scratch/toscopa1/.conda/envs/pyds_v1.4L/bin/python3.10 "-m" "pytest"
284: Working Directory: /scratch/toscopa1/src/rdkit/rdkit
284: Test timeout computed to be: 1500
284: ImportError while loading conftest '/scratch/toscopa1/src/rdkit/rdkit/conftest.py'.
284: __init__.py:6: in <module>
284: from . import rdBase
284: E ImportError: cannot import name 'rdBase' from partially initialized module 'rdkit' (most likely due to a circular import) (/scratch/toscopa1/src/rdkit/rdkit/__init__.py)
1/1 Test #284: pythonSourceTests ................***Failed 1.35 sec
The reason is that pytest needs to import the Python binary modules from the out-of-tree installation directory, which is different from the source directory, while the Python tests are only present in the source directory, which also contains some but not all RDKit Python modules.
This causes circular imports, which cannot be solved with any of the pytest command line switches.
One way to solve the problem would possibly be to move the tests to a separate folder separate from the RDKit source tree.
The way I chose, which involves far less changes, is a small monkey-patch to pytest (which is only applied for out-of-tree builds) to instruct it to import things from the right location and avoid circular imports.
After merging this PR, pythonSourceTests runs successfully:
$ ctest -I 284,284 -V
[...]
test 284
Start 284: pythonSourceTests
284: Test command: /scratch/toscopa1/.conda/envs/pyds_v1.4L/bin/python3.10 "/scratch/toscopa1/src/rdkit/Scripts/rdkit_pytest.py"
284: Working Directory: /scratch/toscopa1/src/rdkit/rdkit
284: Test timeout computed to be: 1500
284: ============================= test session starts ============================== 284: platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0 284: rootdir: /scratch/toscopa1/src/rdkit/rdkit
284: configfile: pytest.ini
284: plugins: anyio-3.7.1, cov-4.1.0, tornasync-0.6.0.post2, typeguard-2.13.3, hypothesis-6.92.0, dash-2.14.2
284: collected 101 items
284:
284: Chem/__init__.py . [ 0%]
284: Chem/AllChem.py .... [ 4%]
284: Chem/BRICS.py ... [ 7%]
284: Chem/Descriptors.py ..... [ 12%]
284: Chem/EnumerateHeterocycles.py . [ 13%]
284: Chem/EnumerateStereoisomers.py .. [ 15%]
284: Chem/FragmentMatcher.py . [ 16%]
284: Chem/MACCSkeys.py . [ 17%]
284: Chem/PandasTools.py .. [ 19%]
284: Chem/PropertyMol.py . [ 20%]
284: Chem/QED.py . [ 21%]
284: Chem/Recap.py . [ 22%]
284: Chem/SaltRemover.py .... [ 26%]
284: Chem/TemplateAlign.py . [ 27%]
284: Chem/AtomPairs/Pairs.py ... [ 30%]
284: Chem/AtomPairs/Sheridan.py ... [ 33%]
284: Chem/AtomPairs/Torsions.py .. [ 35%]
284: Chem/AtomPairs/Utils.py ..... [ 40%]
284: Chem/FeatMaps/FeatMapPoint.py ... [ 43%]
284: Chem/Fingerprints/DbFpSupplier.py .. [ 45%]
284: Chem/Fraggle/FraggleSim.py .. [ 47%]
284: Chem/MolKey/MolKey.py .. [ 49%]
284: Chem/Pharm2D/Utils.py .. [ 51%]
284: Chem/Pharm3D/EmbedLib.py ................ [ 67%]
284: Chem/Scaffolds/MurckoScaffold.py .... [ 71%]
284: Chem/SimpleEnum/Enumerator.py .. [ 73%]
284: DataStructs/BitUtils.py . [ 74%]
284: DataStructs/LazySignature.py .. [ 76%]
284: DataStructs/VectCollection.py . [ 77%]
284: ML/Data/DataUtils.py . [ 78%]
284: ML/Data/SplitData.py .. [ 80%]
284: SimDivFilters/SimilarityPickers.py .. [ 82%]
284: VLib/Filter.py . [ 83%]
284: VLib/Node.py ..... [ 88%]
284: VLib/Output.py . [ 89%]
284: VLib/Supply.py . [ 90%]
284: VLib/Transform.py . [ 91%]
284: VLib/NodeLib/DbPickleSupplier.py . [ 92%]
284: VLib/NodeLib/SDSupply.py . [ 93%]
284: VLib/NodeLib/SmartsMolFilter.py . [ 94%]
284: VLib/NodeLib/SmartsRemover.py .. [ 96%]
284: VLib/NodeLib/SmilesDupeFilter.py . [ 97%]
284: VLib/NodeLib/SmilesOutput.py . [ 98%]
284: VLib/NodeLib/SmilesSupply.py . [ 99%]
284: utils/listutils.py . [100%]
284:
284: =============================== warnings summary ===============================
284: ../../../../../home/toscopa1/install/rdkit_github/fix-out-of-tree-pytest/lib/python3.10/site-packages/rdkit/Chem/MolDb/Loader_sa.py:20
284: /home/toscopa1/install/rdkit_github/fix-out-of-tree-pytest/lib/python3.10/site-packages/rdkit/Chem/MolDb/Loader_sa.py:20: MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy
2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variab
le SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
284: decBase = declarative_base()
284:
284: Chem/BRICS.py::rdkit.Chem.BRICS.FindBRICSBonds
284: /home/toscopa1/install/rdkit_github/fix-out-of-tree-pytest/lib/python3.10/site-packages/rdkit/Chem/BRICS.py:293: DeprecationWarning: The *random* parameter to shuffle() has been deprecated
284: since Python 3.9 and will be removed in a subsequent version.
284: random.shuffle(indices, random=random.random)
284:
284: Chem/BRICS.py: 14 warnings
284: /home/toscopa1/install/rdkit_github/fix-out-of-tree-pytest/lib/python3.10/site-packages/rdkit/Chem/BRICS.py:301: DeprecationWarning: The *random* parameter to shuffle() has been deprecated
284: since Python 3.9 and will be removed in a subsequent version.
284: random.shuffle(compats, random=random.random)
284:
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitDbData
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitIndices
284: ML/Data/SplitData.py::rdkit.ML.Data.SplitData.SplitIndices
284: /home/toscopa1/install/rdkit_github/fix-out-of-tree-pytest/lib/python3.10/site-packages/rdkit/ML/Data/SplitData.py:123: DeprecationWarning: The *random* parameter to shuffle() has been deprecated
284: since Python 3.9 and will be removed in a subsequent version.
284: RDRandom.shuffle(perm, random=random.random)
284:
284: -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
284: ====================== 101 passed, 25 warnings in 10.50s =======================
1/1 Test #284: pythonSourceTests ................ Passed 12.12 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 12.15 sec
My attempt to do an out-of-tree build and run the tests works without problems.
Here are the commands I ran:
cmake -D RDK_INSTALL_INTREE=OFF -D RDK_INSTALL_STATIC_LIBS=OFF -DCMAKE_INSTALL_PREFIX=/scratch/install -D CMAKE_INSTALL_RPATH=/scratch/install/lib -D RDK_BUILD_CAIRO_SUPPORT=ON
make -j12
make install
export RDBASE=/scratch/rdkit_master;export PYTHONPATH=/scratch/install/lib/python3.11/site-packages;export LD_LIBRARY_PATH=/scratch/install/lib:$CONDA_PREFIX/lib
ctest
@greglandrum Thanks for checking. I'll try and figure out what triggers that behavior in my build environment.
This PR was marked as stale because it has been open for 60 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}