TileDB-VCF icon indicating copy to clipboard operation
TileDB-VCF copied to clipboard
verified publisher icon TileDB-Inc

Draft: Add initial build wheels CI/CD pipeline

Open dudoslav opened this issue 1 year ago • 3 comments

This PR aims to modify the build system for the VCF project to comply with our packaging and distribution guidelines: https://www.notion.so/Packaging-and-Distribution-96af10190b4f455f9a31180899f47a83?pvs=4

Specifically we want to enable these scenarios:

  1. Simply just build locally without any package. With CMake options you can specify if TileDB should be local or external as well as if Python bindings should be compiled as well.
  2. Correct PyPI wheel, meaning that all shared libraries MUST be packaged inside of the wheel (tiledb, tiledbvcf)
  3. Correct Conda package, meaning that no shared libraries that are provided by Conda should be packaged (included in the install group)

How to achieve this:

  1. Change the CMake logic to have two callable CMakeLists.txt (./libtiledbvcf and ./apis/python).
  2. When calling ./libtiledbvcf users can specify if they wish to also build python bindings by using the -DTILEDBVCF_ENABLE_PYTHON option. When this is set libtiledbvcf AND bindings are build otherwise just libtiledbvcf is build. This is used for PyPI package (python bindings enabled) and for libtiledbvcf conda package (python bindings disabled)
  3. When calling ./apis/python only python bindings are build. This requires pre-installed libtiledbvcf and libtiledb. This scenario is useful for packaging conda tiledbvcf-py bindings.
  4. In order to make sdist work we need to move pyproject.toml out of apis/python to top level of the project.

NOTE: This PR is being tested in my personal fork.

--

[sc-54633]

dudoslav avatar Aug 28 '24 08:08 dudoslav

Correct Conda package, meaning that no shared libraries that are provided by Conda should be packaged (included in the install group)

When calling ./apis/python only python bindings are build. This requires pre-installed libtiledbvcf and libtiledb. This scenario is useful for packaging conda tiledbvcf-py bindings.

Once this is implemented, this will fix Issue https://github.com/TileDB-Inc/TileDB-VCF/issues/717

jdblischak avatar Sep 09 '24 18:09 jdblischak

Right now I was able to produce wheels in this workflow: https://github.com/dudoslav/TileDB-VCF/actions/runs/10827370316/job/30040372000

Next steps:

  1. Clean up changes
  2. Make sure that feedstocks are not negatively affected (a new PR needs to be created)

dudoslav avatar Sep 12 '24 09:09 dudoslav

Latest error that we have face right now is that Windows wheels fail in tests: https://github.com/dudoslav/TileDB-VCF/actions/runs/11348652284/job/31562931109

Windows fatal exception: access violation
  
  Thread 0x000018b4 (most recent call first):
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\distributed\profile.py", line 370 in _watch
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\threading.py", line 1010 in run
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\threading.py", line 1073 in _bootstrap_inner
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\threading.py", line 1030 in _bootstrap
  
  Thread 0x00001824 (most recent call first):
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\selectors.py", line 314 in _select
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\selectors.py", line 323 in select
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\asyncio\base_events.py", line 1949 in _run_once
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\asyncio\base_events.py", line 641 in run_forever
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\asyncio\base_events.py", line 674 in run_until_complete
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\asyncio\runners.py", line 118 in run
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\asyncio\runners.py", line 194 in run
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\distributed\utils.py", line 559 in run_loop
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\distributed\utils.py", line 451 in wrapper
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\threading.py", line 1010 in run
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\threading.py", line 1073 in _bootstrap_inner
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.12.4\tools\Lib\threading.py", line 1030 in _bootstrap
  
  Current thread 0x00001080 (most recent call first):
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\tiledbvcf\dataset.py", line 125 in __init__
    File "D:\a\TileDB-VCF\TileDB-VCF\apis\python\tests\test_dask.py", line 45 in test_ds
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\fixtures.py", line 898 in call_fixture_func
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\fixtures.py", line 1140 in pytest_fixture_setup
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\fixtures.py", line 1091 in execute
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\fixtures.py", line 617 in _get_active_fixturedef
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\fixtures.py", line 532 in getfixturevalue
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\fixtures.py", line 697 in _fillfixtures
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\python.py", line 1630 in setup
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 514 in setup
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 160 in pytest_runtest_setup
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 242 in <lambda>
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 341 in from_call
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 241 in call_and_report
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 126 in runtestprotocol
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\main.py", line 362 in pytest_runtestloop
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\main.py", line 337 in _main
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\main.py", line 283 in wrap_session
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\config\__init__.py", line 175 in main
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Lib\site-packages\_pytest\config\__init__.py", line 201 in console_main
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-abc7nqqz\cp312-win_amd64\venv-test\Scripts\pytest.exe\__main__.py", line 7 in <module>
    File "<frozen runpy>", line 88 in _run_code
    File "<frozen runpy>", line 198 in _run_module_as_main

dudoslav avatar Oct 16 '24 12:10 dudoslav