tilelang icon indicating copy to clipboard operation
tilelang copied to clipboard

[Release] Unify local build scripts to use `cibuildwheel` and reduce size of sdist

Open oraluben opened this issue 1 month ago • 4 comments

  1. sdist: Update sdist includes, reduce size to meet the 100m limit of pypi;
  2. GA: List artifacts for pull requests with title "[Release]", then we can download built wheel/sdist from artifacts, instead of use qemu on local machine. This also produces macos wheels, example: https://github.com/tile-ai/tilelang/actions/runs/19007774631?pr=1171 @LeiWang1999 ;
  3. sdist: Generates a .git_commit.txt in sdist. pip do not allow a sdist to have different version with the installed package (we cannot have a sdist named tilelang-0.1.6.post2+gitxxxx.tar.gz and install a package named tilelang==0.1.6.post2+cu128.gitxxxx). Since sdist should not contain toolchain info, we need to store its git commit separately. When installed from sdist, the git info will be read from that file.
  4. scripts: delete some file, use cibuildwheel to generate wheels for linux. However this seems not necessary, I hope Github Actions can eliminated most of local builds.

cc @XuehaiPan

Summary by CodeRabbit

  • Chores
    • Removed legacy manifest file and moved to explicit vendored subcomponent packaging in pyproject.
    • Simplified build/distribution: removed multi-arch Docker orchestration, using direct build-tool invocations for local/CI builds.
    • Updated container builder bases and per-architecture CUDA defaults for manylinux builds.
    • Persisted version stamping to a git-pin file (now ignored) and fallback to "gitunknown" when commit info is unavailable.
    • CI: sdist/wheels/artifacts also produced for PRs titled “[Release]”.

oraluben avatar Nov 01 '25 14:11 oraluben

👋 Hi! Thank you for contributing to the TileLang project.

Please remember to run pre-commit run --all-files in the root directory of the project to ensure your changes are properly linted and formatted. This will help ensure your contribution passes the format check.

We appreciate you taking this step! Our team will review your contribution, and we look forward to your awesome work! 🚀

github-actions[bot] avatar Nov 01 '25 14:11 github-actions[bot]

[!WARNING]

Rate limit exceeded

@oraluben has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 7 minutes and 5 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 703e63c15803fd016f321f0ef6e1ef73eb912156 and 54ff485957e25c2c96f97ada83769935b37bb7e1.

📒 Files selected for processing (1)
  • pyproject.toml (5 hunks)

Walkthrough

Deletes MANIFEST.in, tightens sdist/wheel vendoring in pyproject.toml, simplifies multi-arch Docker distribution scripts to single cibuildwheel invocations, updates manylinux Dockerfile and CI build flags, adds git commit caching/persistence in version_provider.py, and ignores _git_commit.txt in .gitignore.

Changes

Cohort / File(s) Summary
Packaging manifests
MANIFEST.in, pyproject.toml, version_provider.py, .gitignore
MANIFEST.in removed; pyproject.toml narrows sdist/wheel vendoring to explicit 3rdparty subpaths (TVM, CUTLASS, Composable Kernel), adds DEFAULT_CUDA_VERSION and arch-aware CUDA defaults, and build/debug flags; version_provider.py adds @lru_cache(maxsize=1), persists git hash to _git_commit.txt and reads it as fallback, and appends gitunknown when missing; .gitignore now ignores _git_commit.txt.
Top-level build orchestration
maint/scripts/docker_build_all.sh
Removed sequential invocation and logging of docker_local_distribute.sh and docker_pypi_distribute.sh.
Local distribution script
maint/scripts/docker_local_distribute.sh
Replaced multi-arch docker buildx orchestration with a single local cibuildwheel invocation (CIBW_BUILD targeted at local arch).
PyPI distribution script
maint/scripts/docker_pypi_distribute.sh
Replaced per-arch docker buildx loop and dist relocation with a single cibuildwheel invocation exposing CIBW_ARCHS, CIBW_BUILD, and NO_VERSION_LABEL; removed per-arch tagging/relocation logic.
Manylinux builder Dockerfile
maint/scripts/pypi.manylinux.Dockerfile
Switched builder bases to quay.io/pypa/manylinux... images, added ARG/ENV CUDA_VERSION and TARGETARCH multi-stage selection, set TZ, updated CUDA repo handling, and install cibuildwheel via pipx (removed venv/uv steps).
GitHub Actions
.github/workflows/dist.yml
Replaced NO_GIT_VERSION with NO_VERSION_LABEL; broadened conditions so SDist, wheel upload, and listing artifacts also run for PRs whose titles contain [Release].

Sequence Diagram(s)

sequenceDiagram
  rect rgb(250,240,230)
    Note over CI: Old flow — build-all invoked per-arch Docker buildx scripts
    CI->>docker_build_all.sh: run
    docker_build_all.sh->>docker_local_distribute.sh: invoke & log
    docker_local_distribute.sh->>docker buildx: setup & per-arch builds
    docker_build_all.sh->>docker_pypi_distribute.sh: invoke & log
    docker_pypi_distribute.sh->>docker buildx: setup & per-arch builds
    docker buildx-->>CI: per-arch artifacts (relocated)
  end

  rect rgb(230,250,240)
    Note over CI: New flow — direct cibuildwheel invocation
    CI->>cibuildwheel: invoke (CIBW_ARCHS/CIBW_BUILD/NO_VERSION_LABEL)
    cibuildwheel->>manylinux container: build wheels (arch-aware)
    manylinux container-->>cibuildwheel: produced wheels
    cibuildwheel-->>CI: artifacts in dist/
  end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Inspect pyproject.toml sdist/wheel include/exclude changes vs. removed MANIFEST.in.
  • Verify atomicity and permissions for _git_commit.txt writes/reads in version_provider.py.
  • Validate manylinux Dockerfile CUDA repo and pipx-based cibuildwheel installation.

Possibly related PRs

  • tile-ai/tilelang#973 — modifies CI distribution/workflow logic overlapping .github/workflows/dist.yml changes.
  • tile-ai/tilelang#1098 — touches packaging and CI manifests (MANIFEST.in, pyproject.toml) similar to these packaging edits.

Suggested reviewers

  • LeiWang1999

Poem

🐇 I nudged the manifest into the ground,

Packed TVM and CUTLASS safe and sound.
From buildx bustle to cibuildwheel’s cheer,
I cached the hash so versions persevere.
Hoppity-hop — the dist folder's near! 🥕

Pre-merge checks and finishing touches

✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the two main changes: unifying build scripts to use cibuildwheel and reducing sdist size, which directly align with the changeset modifications.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Nov 01 '25 14:11 coderabbitai[bot]

is 3rdparty/tvm/src necessary for wheels?

oraluben avatar Nov 01 '25 14:11 oraluben

is 3rdparty/tvm/src necessary for wheels?

yes, this should be included

LeiWang1999 avatar Nov 03 '25 05:11 LeiWang1999