automatic wheel builds for linux, osx and windows on travisci

[timkofu]

• removed python 2 support in travis
• wheels built via cibuildwheel
• automatic upload to pypi via twine

[mrjbq7]

This is cool, thanks.

Why did you remove python2 support? Is the wheel build incompatible?

How is $TRAVIS_TAG set? Does it look for a git tag for the current gitref?

It looks like I need to configure Travis with a PyPI token also?

[timkofu]

• No Windows builds for 2.7 (only OSX and Linux), and 2.7 reached EOL last year. What do you reckon?
• It's set automatically TRAVIS_TAG: If the current build is for a git tag, this variable is set to the tag’s name, otherwise it is empty ("").
• Yea, from the sample conf: # Note: TWINE_PASSWORD is set to a PyPI API token in Travis settings.

[mrjbq7]

This is cool, I'm trying to confirm it works locally. Do you know any way to make it not download and sudo install python onto my dev machine? I want to just build with the locally installed python 3.8 that I have?

[timkofu]

Not at the moment no; this line in macos.py is where it's probably being called from (only place in the repo where the sudo keyword is).

[xmatthias]

I love this PR! You should be able to test if this works by pushing to test.pypi.org (obviously a prior version should be available on test.pypi.org) - and then trying to install from test-pypi ...

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

---

Question - Will the wheels then be similar to the inofficial wheels from here - which do include the C dependency as well, so running pip install ta-lib will in the future require no additional (prior) installation step?

[timkofu]

@xmatthias yea, the wheels are effectively statically linked.

[cryptocoinserver]

Yeah ta-lib really needs this. Would save a lot time installing ta-lib. We had to write a whole section about installing ta-lib in our projects docs, because users have problems there. There is another example for travis: https://github.com/joerick/cibuildwheel/blob/master/examples/travis-ci-test-and-deploy.yml

[mrjbq7]

Okay, I'm going to try and get this working today or next few days. It would be nice to reduce the amount of questions I get about installation!

[cryptocoinserver]

---
language: python
python:
  - "3.6"
  - "3.7"
  - "3.8"
  - "3.9"
os: linux
dist: bionic
env:
  global:
    - DEPS_DIR=$HOME/dependencies
    - TWINE_USERNAME=__token__
    # Note: TWINE_PASSWORD is set to a PyPI API token in Travis settings
    - CIBW_SKIP="cp27-* cp35-* pp*"
    # pp leads to build errors.
    - CIBW_BEFORE_BUILD_LINUX="yum install -y gcc && curl -sL http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && tar -zxvf ta-lib-0.4.0-src.tar.gz && rm ta-lib-0.4.0-src.tar.gz && cd ta-lib && ./configure && make && make install"
    - CIBW_BEFORE_BUILD_MACOS="brew install ta-lib"
    - CIBW_BEFORE_BUILD_WINDOWS="curl -sL http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-msvc.zip -o $HOME/ta-lib.zip --create-dirs && 7z x $HOME/ta-lib.zip -o/c/ta-lib && mv /c/ta-lib/ta-lib/* /c/ta-lib/ && rm -rf /c/ta-lib/ta-lib && cd /c/ta-lib/c/make/cdr/win32/msvc && nmake"
  matrix:
    - WITH_TA_LIBRARY=yes
      TA_INCLUDE_PATH=$DEPS_DIR/include
      LD_LIBRARY_PATH=$DEPS_DIR/lib
      TA_LIBRARY_PATH=$DEPS_DIR/lib
cache:
  directories:
    - $DEPS_DIR
        
install:
  - pip install --upgrade pip wheel
  - pip install -r requirements_test.txt
  - if [ $WITH_TA_LIBRARY = "yes" ]; then
      ./tools/build_talib_from_source.bash $DEPS_DIR;
    fi
script:
  - make test

stages:
  - test
  # Only execute deployment stage on tagged commits, and from your repository
  # (e.g. not PRs). Replace with your repo name.
  - name: deploy
    if: tag IS PRESENT AND repo = mrjbq7/ta-lib

jobs:
  include:
    # Deploy source distribution
    - stage: deploy
      name: Deploy source distribution
      install: pip install -r requirements.txt
      script: python3 setup.py sdist --formats=gztar
      after_success: |
        python3 -m pip install twine
        python3 -m twine upload --skip-existing dist/*.tar.gz
    # Deploy on linux
    - stage: deploy
      name: Build and deploy Linux wheels
      services: docker
      install: python3 -m pip install cibuildwheel==1.7.1
      script: python3 -m cibuildwheel --output-dir wheelhouse
      after_success: |
        python3 -m pip install twine
        python3 -m twine upload --skip-existing wheelhouse/*.whl
    # Deploy on mac
    - stage: deploy
      name: Build and deploy macOS wheels
      os: osx
      # PyPy 7.3.2 needs macOS >= 10.14
      osx_image: xcode10.2
      language: shell
      install: python3 -m pip install cibuildwheel==1.7.1
      script: python3 -m cibuildwheel --output-dir wheelhouse
      after_success: |
        python3 -m pip install twine
        python3 -m twine upload --skip-existing wheelhouse/*.whl
    # Deploy on windows
    - stage: deploy
      name: Build and deploy Windows wheels
      os: windows
      language: shell
      before_install:
        - choco install python --version 3.8.0
        - export PATH="/c/Python38:/c/Python38/Scripts:$PATH"
        # make sure it's on PATH as 'python3'
        - ln -s /c/Python38/python.exe /c/Python38/python3.exe
      install: python -m pip install cibuildwheel==1.7.1
      script: python -m cibuildwheel --output-dir wheelhouse
      after_success: |
        python -m pip install twine
        python -m twine upload --skip-existing wheelhouse/*.whl

This should be a good start. The important part is CIBW_BEFORE_BUILD_LINUX, CIBW_BEFORE_BUILD_MACOS and CIBW_BEFORE_BUILD_WINDOWS. Here we have to build ta-lib. I think the paths are not right, the other commands should be. You need to test it on travis.

This part: if: tag IS PRESENT AND repo = mrjbq7/ta-lib makes sure all those deploy stuff is only executed if you make a release on github and prevents travis to deploy on forked versions. You need to add the pypi token to travis environment variables. https://pypi.org/manage/account/token/: In the Travis web UI, go to your project settings and add the environment variable TWINE_PASSWORD, set to your new PyPI API token.

Hope that helps.

[cryptocoinserver]

- CIBW_SKIP="cp27-* cp35-* pp*" This skips the EOL python versions. And PyPy as PyPy lead to some errors during build in my case.

[mrjbq7]

I should really figure out how to test this.

[tylerwmarrs]

@cryptocoinserver I think that this line needs to be ubuntu based as dist is set to bionic. Right now it is a Redhat flavor.

- CIBW_BEFORE_BUILD_LINUX="yum install -y gcc && curl -sL http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && tar -zxvf ta-lib-0.4.0-src.tar.gz && rm ta-lib-0.4.0-src.tar.gz && cd ta-lib && ./configure && make && make install"

@mrjbq7 unfortunately there isn't a nice way to test this without just running it. :cry: I know as I'm a core maintainer of a library having to build wheels for OSx, Windows, and Linux too.

[cryptocoinserver]

@cryptocoinserver Not sure to be honest, but I think cibuildwheel builds not directly in the dist, but in a manylinux docker. Therfore the yum. Its mentioned here: https://cibuildwheel.readthedocs.io/en/stable/faq/#linux-builds-on-docker

I think the only stuff that might need some changes is the path stuff. I'm very confident the rest of the code is right.