[Draft] Initial rework into scikit-build-core

[dudoslav]

This PR should prove that we could rework our build system into scikit-build-core. Right now we are able to compile both pybind11 and Cython extensions. On my local tests I was able to pip install --verbose . and then python -c "import tiledb". Right now there is no logic that would download libtiledb.so so further changes are needed. This could be achieved by using Superbuild, or by simply using fetch_content.

[dudoslav]

This logic produces this folder structure (please ignore vector-search there):

(base) taco:tiledb dudoslav$ ls -l
.rw-r--r-- 3.9k dudoslav 17 May 14:39 __init__.py
drwxr-xr-x    - dudoslav 17 May 14:39 __pycache__
.rw-r--r--  469 dudoslav 17 May 14:39 _generated_version.py
.rw-r--r--  16k dudoslav 17 May 14:39 array_schema.py
.rw-r--r--  11k dudoslav 17 May 14:39 attribute.py
.rwxr-xr-x 1.1M dudoslav 17 May 14:39 cc.cpython-310-x86_64-linux-gnu.so
drwxr-xr-x    - dudoslav 27 Mar 18:37 cloud
.rw-r--r-- 1.2k dudoslav 17 May 14:39 CMakeLists.txt
.rw-r--r-- 1.8k dudoslav 17 May 14:39 common.pxi
.rw-r--r-- 3.8k dudoslav 17 May 14:39 consolidation_plan.py
.rw-r--r--  58k dudoslav 17 May 14:39 core.cc
.rw-r--r--  19k dudoslav 17 May 14:39 ctx.py
.rw-r--r--  207 dudoslav 17 May 14:39 data_order.py
.rw-r--r--  36k dudoslav 17 May 14:39 dataframe_.py
.rw-r--r-- 8.0k dudoslav 17 May 14:39 datatypes.py
.rw-r--r-- 2.1k dudoslav 17 May 14:39 debug.cc
.rw-r--r-- 9.5k dudoslav 17 May 14:39 dimension.py
.rw-r--r-- 2.8k dudoslav 17 May 14:39 dimension_label.py
.rw-r--r-- 4.0k dudoslav 17 May 14:39 dimension_label_schema.py
.rw-r--r-- 5.6k dudoslav 17 May 14:39 domain.py
.rw-r--r-- 4.9k dudoslav 17 May 14:39 enumeration.py
.rw-r--r-- 5.5k dudoslav 17 May 14:39 filestore.py
.rw-r--r--  34k dudoslav 17 May 14:39 filter.py
.rw-r--r-- 9.0k dudoslav 17 May 14:39 fragment.cc
.rw-r--r--  22k dudoslav 17 May 14:39 fragment.py
.rw-r--r--  16k dudoslav 17 May 14:39 group.py
.rw-r--r--  13k dudoslav 17 May 14:39 highlevel.py
.rw-r--r--  134 dudoslav 17 May 14:39 indexing.pxd
.rw-r--r--  14k dudoslav 17 May 14:39 indexing.pyx
drwxr-xr-x    - dudoslav 17 May 14:39 lib
.rw-r--r--  18k dudoslav 17 May 14:39 libmetadata.pyx
.rwxr-xr-x 1.6M dudoslav 17 May 14:39 libtiledb.cpython-310-x86_64-linux-gnu.so
.rw-r--r--  32k dudoslav 17 May 14:39 libtiledb.pxd
.rw-r--r-- 155k dudoslav 17 May 14:39 libtiledb.pyx
.rw-r--r--  773 dudoslav 17 May 14:39 main.cc
.rwxr-xr-x 677k dudoslav 17 May 14:39 main.cpython-310-x86_64-linux-gnu.so
.rw-r--r--  27k dudoslav 17 May 14:39 multirange_indexing.py
.rw-r--r--  17k dudoslav 17 May 14:39 npbuffer.cc
.rw-r--r--  330 dudoslav 17 May 14:39 npbuffer.h
.rw-r--r--   99 dudoslav 17 May 14:39 numpyFlags.h
.rw-r--r-- 1.0k dudoslav 17 May 14:39 object.py
.rw-r--r--  148 dudoslav 17 May 14:39 parquet_.py
.rw-r--r--  27k dudoslav 17 May 14:39 py_arrow_io_impl.h
.rw-r--r-- 3.2k dudoslav 17 May 14:39 py_arrowio
.rw-r--r--  789 dudoslav 17 May 14:39 query.py
.rw-r--r-- 9.2k dudoslav 17 May 14:39 query_condition.cc
.rw-r--r--  18k dudoslav 17 May 14:39 query_condition.py
.rw-r--r-- 4.0k dudoslav 17 May 14:39 schema_evolution.cc
.rw-r--r-- 2.4k dudoslav 17 May 14:39 schema_evolution.py
.rw-r--r-- 2.2k dudoslav 17 May 14:39 serialization.cc
.rw-r--r-- 4.0k dudoslav 17 May 14:39 subarray.py
drwxr-xr-x    - dudoslav 17 May 14:39 tests
.rw-r--r--  840 dudoslav 17 May 14:39 util.cc
.rw-r--r-- 1.1k dudoslav 17 May 14:39 util.h
drwxr-xr-x    - dudoslav 14 May 09:03 vector_search
.rw-r--r--  386 dudoslav 17 May 14:39 version_helper.py
.rw-r--r--  17k dudoslav 17 May 14:39 vfs.py

[dudoslav]

I added basic logic for downloading TileDB from release inside CMake so now we need to change CI/CD to not download release archives.

[dudoslav]

Wheel builds pass: https://github.com/TileDB-Inc/TileDB-Py/actions/runs/9157079084/job/25172731354?pr=1971

[dudoslav]

I had to disable build for python 3.9, I assume that that is and issue

[dudoslav]

For some reason python 3.9 fails on:

           assert "Input attribute name is already there" in str(excinfo.value)
  >       assert "tiledb/schema_evolution.cc" in str(excinfo.value)
  E       AssertionError: assert 'tiledb/schema_evolution.cc' in 'ArraySchemaEvolution: Cannot add attribute; Input attribute name is already there (D:\\a\\TileDB-Py\\TileDB-Py\\tiledb\\schema_evolution.cc:44)'
  E        +  where 'ArraySchemaEvolution: Cannot add attribute; Input attribute name is already there (D:\\a\\TileDB-Py\\TileDB-Py\\tiledb\\schema_evolution.cc:44)' = str(TileDBError('ArraySchemaEvolution: Cannot add attribute; Input attribute name is already there (D:\\a\\TileDB-Py\\TileDB-Py\\tiledb\\schema_evolution.cc:44)'))
  E        +    where TileDBError('ArraySchemaEvolution: Cannot add attribute; Input attribute name is already there (D:\\a\\TileDB-Py\\TileDB-Py\\tiledb\\schema_evolution.cc:44)') = <ExceptionInfo TileDBError('ArraySchemaEvolution: Cannot add attribute; Input attribute name is already there (D:\\a\\TileDB-Py\\TileDB-Py\\tiledb\\schema_evolution.cc:44)') tblen=2>.value
  
  D:\a\TileDB-Py\TileDB-Py\tiledb\tests\test_schema_evolution.py:45: AssertionError

[dudoslav]

The failure is not due to python 3.9 but it seems to fail on windows only

[dudoslav]

Build/test issue resolved, it was due to incorrect use of / on Windows

[dudoslav]

I also added sdist tests so that we can be sure that sdist can be build on specified os/python combinations: https://github.com/dudoslav/TileDB-Py/actions/runs/9303158461