s5cmd Facilitating Python Usage: Proposal for s5cmd Wheels

Firstly, huge thanks for your work on the s5cmd tool and for contributing to the community with such a valuable tool. :pray:

Goal

We're keen on enhancing the accessibility of s5cmd within Python projects. Our aim is to streamline its integration by enabling installation via a simple pip command:

pip install s5cmd

Upon installing the wheel package, we envision an automatic creation of a convenient executable script^4, allowing seamless invocation of the underlying s5cmd executable.

Proposed Next Steps

To foster cohesion and prevent fragmentation, we propose the creation of a dedicated project repository, tentatively named peak/s5cmd-python-distributions. This draws inspiration from analogous projects like scikit-build/cmake-python-distributions^1 and scikit-build/ninja-python-distributions^2.

We're committed to aiding in the setup and maintenance of this infrastructure. With this in place, we can align our project, ImagingDataCommons/idc-index, to become a pure Python wheel, contingent upon the s5cmd project.

The implementation approach mirrors our current practices within our package:

idc_index/s5cmd/init.py: A Python module with a function used as entry point for the launcher script generation.
CMakeLists.txt: CMake configuration for downloading the go executable using URL and SHA specified in s5cmdUrls.cmake.

Furthermore, we propose the implementation of a GitHub workflow, automatically triggered upon each new release of the corresponding go executable. We plan to adapt the bump function from cmake-python-distributions^5 for this purpose.

Alternative Project Repository

We are also considering ImagingDataCommons/s5cmd-python-distributions as an alternative home for this project.

Background Details

@fedorov is leading the https://github.com/ImagingDataCommons initiative
I (@jcfr) along with @henryiii, @thewtex and many others are developing and maintaining the cmake^1, ninja^2 and scikit-build-core^6 Python projects.

Feb 14 '24 21:02 jcfr

After running pip install s5cmd, the following will be possible from Python without having to manually download/extract/install:

import shutil
import subprocess


s5cmdPath = shutil.which("s5cmd")

if s5cmdPath is None:
    raise FileNotFoundError(
        "s5cmd executable not found. Please install s5cmd from https://github.com/peak/s5cmd#installation"
    )

print(f"Found s5cmd executable: {s5cmdPath}")

subprocess.check_call([s5cmdPath, "--help"])

Feb 14 '24 21:02 jcfr

To follow-up on this, a dedicated project has been created at https://github.com/jcfr/s5cmd-python-distributions

Wheels have been published at https://pypi.org/project/s5cmd/

The following platforms are supported by the binary wheels:

OS	Arch
Windows	64-bit 32-bit ARM64
Linux Intel	manylinux 64-bit musllinux 64-bit manylinux 32-bit musllinux 32-bit
Linux ARM	manylinux AArch64 musllinux AArch64
Linux PowerPC	manylinux ppc64le musllinux ppc64le
macOS 10.10+	Intel
macOS 11+	Apple Silicon

Feb 27 '24 21:02 jcfr