buskill-app icon indicating copy to clipboard operation
buskill-app copied to clipboard
verified publisher icon BusKill

Deterministic Builds

Open maltfield opened this issue 5 years ago • 12 comments

This task is for me to confirm that all three of our build scripts produce deterministic artifacts.

Files produced by our build script run via GitHub's CI workflows should have identical checksums to files produced by our build script run on our developer's local machines.

See also:

  • https://reproducible-builds.org/

maltfield avatar Jul 03 '20 02:07 maltfield

I'm starting with this task starting with Linux. I had two issues with my linux build script that was preventing it from being deterministic:

  1. Our AppImage is shipping with a ton of __pycache__/.../*.pyc files that differ from the GitHub build and our local build

  2. The contents of buskill_version.py breaks determinism. I can manually set GITHUB_SHA and GITHUB_REF on local builds (attempted in this commit), but GITHUB_RUN breaks determinism, so it's been removed.

I fixed those in this commit https://github.com/BusKill/buskill-app/commit/0a98242489b5a1b581b1be1651fef6560986c09a

Unfortunately, I found the build from GitHub and my local build on a Debian 10 DispVM produced distinct checksums. I extracted the AppImage and found the dirs identical. I built a new AppImage from one of the build's extracted squashfs-root dir, then I immediately built another thereafter. Indeed, the checkums still differed, and it's extremely unlikely that the contents or its metadata changed in any way between the two runs.

user@disp6736:~/Downloads$ ls
buskill-linux-x86_64.156764000.tar.bz2
user@disp6736:~/Downloads$ tar -xjf buskill-linux-x86_64.156764000.tar.bz2
user@disp6736:~/Downloads$ cd dist/
user@disp6736:~/Downloads/dist$ sha384sum buskill.AppImage
fdd2ede3895567ae709febeef78b28c036eccb6a1e5b3501d8880b9628e5f5acd0416f7eff80f28be73897c82db95f63  buskill.AppImage
user@disp6736:~/Downloads/dist$ ./buskill.AppImage --appimage-extract > /dev/null
user@disp6736:~/Downloads/dist$ wget --quiet --continue --output-document="appimagetool.AppImage" https://github.com/AppImage/AppImageKit/releases/download/12/appimagetool-x86_64.AppImage
user@disp6736:~/Downloads/dist$ chmod +x appimagetool.AppImage
user@disp6736:~/Downloads/dist$ ./appimagetool.AppImage --no-appstream squashfs-root buskill_1.AppImage
appimagetool, continuous build (commit effcebc), build 2084 built on 2019-05-01 21:02:41 UTC
Using architecture x86_64
/home/user/Downloads/dist/squashfs-root should be packaged as buskill_1.AppImage
Generating squashfs...
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on buskill_1.AppImage, block size 131072.
[=============================================================\] 2524/2524 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 31138.46 Kbytes (30.41 Mbytes)
	33.09% of uncompressed filesystem size (94107.66 Kbytes)
Inode table size 21871 bytes (21.36 Kbytes)
	30.34% of uncompressed inode table size (72097 bytes)
Directory table size 21938 bytes (21.42 Kbytes)
	46.19% of uncompressed directory table size (47490 bytes)
Number of duplicate files found 18
Number of inodes 2161
Number of files 2007
Number of fragments 205
Number of symbolic links  19
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 135
Number of ids (unique uids + gids) 1
Number of uids 1
	root (0)
Number of gids 1
	root (0)
Embedding ELF...
Marking the AppImage as executable...
Embedding MD5 digest
Success

Please consider submitting your AppImage to AppImageHub, the crowd-sourced
central directory of available AppImages, by opening a pull request
at https://github.com/AppImage/appimage.github.io
user@disp6736:~/Downloads/dist$ ./appimagetool.AppImage --no-appstream squashfs-root buskill_2.AppImage > /dev/null
appimagetool, continuous build (commit effcebc), build 2084 built on 2019-05-01 21:02:41 UTC
Using architecture x86_64
Generating squashfs...
Embedding ELF...
Marking the AppImage as executable...
Embedding MD5 digest
Success

Please consider submitting your AppImage to AppImageHub, the crowd-sourced
central directory of available AppImages, by opening a pull request
at https://github.com/AppImage/appimage.github.io
user@disp6736:~/Downloads/dist$ sha384sum buskill_1.AppImage 
6156521f583871f485bdc48bb098fbd683a3cf178bc78e4b76bb8f90188b67fd8a8e0b0c9ff5220a1136b3038c42e593  buskill_1.AppImage
user@disp6736:~/Downloads/dist$ sha384sum buskill_2.AppImage 
32c62b6cb6bb91841838af60cf8e2e215ffb9e88a4ab086fe06d1d04885ff9ea76bdf561e72c6da6e9d057e73b4f6908  buskill_2.AppImage
user@disp6736:~/Downloads/dist$

It looks like the issue is related to appimagetools using mksquashfs v4.3 (from 2017). Almost a year ago, v4.4 fixed this issue, but it appears that appimagetool never added-in the updated version of mksquashfs. I asked for an ETA on when appimagetool would expect to release their next stable release including mksquashfs v4.4 here:

  • https://github.com/AppImage/AppImageKit/issues/929#issuecomment-653641526

In the meantime, it looks like some other projects such as the electrum crypto wallet have hacked appimagetool to use squashfskit to make reproducible builds:

  • https://github.com/SomberNight/electrum/blob/ae714772c38410a0169f2c76a14a64a62c0daff0/contrib/build-linux/appimage/build.sh#L212-L225

maltfield avatar Jul 03 '20 18:07 maltfield

Linux deterministic builds are complete. For a diff of the changes (and associated commit messages) needed to achieve this, see:

  • https://github.com/buskill/buskill-app/compare/e29e133dd6f5be17ad7fad6dcb96109affffe188...78ac8ab4a47d4d1d8d63ed90db918d42e08853be

After the changes listed in the comment above, this is what was required:

  1. Downloading & compiling the latest version of mksquashfs (v4.4)
  2. Replacing appimagetool's mksquashfs binary with a dumb script that strips an argument and passes the call on to the new mksquashfs binary
  3. setting the SOURCE_DATE_EPOCH environment variable (for mksquashfs)
  4. changing the atime & mtime of all the files in the AppDir to SOURCE_DATE_EPOCH before creating the AppImage with find and touch

maltfield avatar Jul 04 '20 20:07 maltfield

I hit a wall trying to get pyinstaller to build deterministic binaries with this app in MacOS, so I walked-back to trying to build a simple one-line "hello world" app with pyinstaller that would build deterministically, and I couldn't get it to do so.

I created a repo specific to this task here:

  • https://github.com/maltfield/pyinstaller-deterministic-test

And I filed a bug report with pyinstaller here:

  • https://github.com/pyinstaller/pyinstaller/issues/4972

Per my last comment, it might be that the order of the dependencies written to the binary are out-of-order and pyinstaller just needs to somehow sort them alphabetically to make their builds deterministic.

maltfield avatar Jul 05 '20 21:07 maltfield

I added a windows build CI process to the workflow in my pyinstaller-deterministic-test repo above to see if PyInstaller could successfully produce reproducible builds in Windows of a simple 1-line hello world app. Sadly, it couldn't.

I added this info to a new comment in the bug report I created yesterday with the PyInstaller team, but this actually might require a distinct patch from the MacOS issues found above.

Sadly, I'm not sure there's much I can do to make BusKill's builds deterministic in MacOS/Windows (other than abandoning PyInstaller?) until this is fixed upstream.

maltfield avatar Jul 06 '20 13:07 maltfield

I made a few final changes to the linux build script so our linux builds are reproducible across the github shared runners and my laptop

  • https://github.com/buskill/buskill-app/compare/c25ab88b034076a124bd6858e958fa7e7febbfa3...120db24f2334071404e238123b94e51cf5987dce

Note that, as of now, our first (alpha) release is out!

  • https://github.com/BusKill/buskill-app/releases/tag/v0.1.0

maltfield avatar Jul 31 '20 11:07 maltfield

update: newer versions of PyInstaller appear to have fixed reproducibility or MacOS

  • https://github.com/pyinstaller/pyinstaller/issues/4972#issuecomment-924308972

TODO: see if it works for our buskill MacOS builds, too.

maltfield avatar Sep 21 '21 19:09 maltfield

TOFU 1/3 via VPN in Netherlands

user@disp5319:/tmp/tmp.p7j2Zbimtj$ # first get some info about our internet connection
user@disp5319:/tmp/tmp.p7j2Zbimtj$ ${CURL} -s https://ifconfig.co/country | head -n1
Netherlands
user@disp5319:/tmp/tmp.p7j2Zbimtj$ ${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
user@disp5319:/tmp/tmp.p7j2Zbimtj$ 
user@disp5319:/tmp/tmp.p7j2Zbimtj$ # and today's date
user@disp5319:/tmp/tmp.p7j2Zbimtj$ date -u +"%Y-%m-%d"
2021-12-01
user@disp5319:/tmp/tmp.p7j2Zbimtj$ 
user@disp5319:/tmp/tmp.p7j2Zbimtj$ # get the file
user@disp5319:/tmp/tmp.p7j2Zbimtj$ pip3 download ${PIP_PKGS}
Collecting pyinstaller
  Using cached https://files.pythonhosted.org/packages/00/ca/58dd68fee42490be1c86c9e912fc9ad0bf44c72edd882397ad11c21fbecb/pyinstaller-4.7.tar.gz
  Saved ./pyinstaller-4.7.tar.gz
  Installing build dependencies ... done
Collecting setuptools (from pyinstaller)
  Using cached https://files.pythonhosted.org/packages/6f/ea/e12311dabc63a6a434be25e6011c1513cc95d0cf22e5f13036f75b3ec508/setuptools-59.4.0-py3-none-any.whl
  Saved ./setuptools-59.4.0-py3-none-any.whl
Collecting altgraph (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/84/3f/1a5c9bef54cac9bf41edd6f4aaf61cd52ed578e10ccc607e0278012cb4a5/altgraph-0.17.2-py2.py3-none-any.whl
  Saved ./altgraph-0.17.2-py2.py3-none-any.whl
Collecting pyinstaller-hooks-contrib>=2020.6 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/80/74/4c885df43604c4ae570610e187052f29d806d582e398c2e48b83dad74610/pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl (215kB)
    100% |████████████████████████████████| 225kB 2.4MB/s 
  Saved ./pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
Collecting importlib-metadata (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/c4/1f/e2238896149df09953efcc53bdcc7d23597d6c53e428c30e572eda5ec6eb/importlib_metadata-4.8.2-py3-none-any.whl
  Saved ./importlib_metadata-4.8.2-py3-none-any.whl
Collecting typing-extensions>=3.6.4; python_version < "3.8" (from importlib-metadata->pyinstaller)
  Downloading https://files.pythonhosted.org/packages/05/e4/baf0031e39cf545f0c9edd5b1a2ea12609b7fcba2d58e118b11753d68cf0/typing_extensions-4.0.1-py3-none-any.whl
  Saved ./typing_extensions-4.0.1-py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata->pyinstaller)
  Downloading https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl
  Saved ./zipp-3.6.0-py3-none-any.whl
Successfully downloaded pyinstaller setuptools altgraph pyinstaller-hooks-contrib importlib-metadata typing-extensions zipp
user@disp5319:/tmp/tmp.p7j2Zbimtj$ 
user@disp5319:/tmp/tmp.p7j2Zbimtj$ # checksum
user@disp5319:/tmp/tmp.p7j2Zbimtj$ sha256sum *
743628f2ac6a7c26f5d9223c91ed8ecbba535f506f4b6f558885a8a56a105857  altgraph-0.17.2-py2.py3-none-any.whl
53ccfd5c134223e497627b9815d5030edf77d2ed573922f7a0b8f8bb81a1c100  importlib_metadata-4.8.2-py3-none-any.whl
2c7f4810dc5272ec1b388a7f1ff6b56d38653c1b0c9ac2d9dd54fa06b590e372  pyinstaller-4.7.tar.gz
60a57e4057fa2183bbaa81f10401a27eb7dd701ef8a11b287bb6345b571f94e7  pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
feb5ff19b354cde9efd2344ef6d5e79880ce4be643037641b49508bbb850d060  setuptools-59.4.0-py3-none-any.whl
7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b  typing_extensions-4.0.1-py3-none-any.whl
9fe5ea21568a0a70e50f273397638d39b03353731e6cbbb3fd8502a33fec40bc  zipp-3.6.0-py3-none-any.whl
user@disp5319:/tmp/tmp.p7j2Zbimtj$

maltfield avatar Dec 01 '21 20:12 maltfield

Hmm..I ran this on my actual buskill dev VM, and I got a diff. It's actually a totally distinct file.

In the TOFU above, I get a file named:

2c7f4810dc5272ec1b388a7f1ff6b56d38653c1b0c9ac2d9dd54fa06b590e372  pyinstaller-4.7.tar.gz

But on my buskill VM, I get:

03636feec822de1d23d9753054f0b1229fb23d58723ae796f41b1127fc54f572  pyinstaller-4.7-py3-none-manylinux2014_x86_64.whl

Not that both are running Debian 10 with the same kernel version, but the former is using pip v18.1 and the latter pip v20.1.1.

EDIT: Ah, it looks like I pinned pip to v20.1.1 in 2020-07 as it was a dependency for kivy:

  • https://github.com/BusKill/buskill-app/commit/37c1d0b651faf9e1d551a35e93cf35a60231e84c
  • https://github.com/BusKill/buskill-app/issues/2#issuecomment-655799569

So in order for this 3TOFU to work, I should first make sure I'm using pip v20.1.1. I'll update the TOFU commands to be:

# SETTINGS
PIP_PKGS="pyinstaller"

CURL="/usr/bin/curl"
WGET="/usr/bin/wget --retry-on-host-error --retry-connrefused"
PYTHON="/usr/bin/python3"
PIP="${PYTHON} -m pip"

# prereqs
sudo apt-get -y install python3-pip
python3 -m pip install pip==20.1.1

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

# get the file
pip3 download ${PIP_PKGS}

# checksum
sha256sum *

maltfield avatar Dec 01 '21 20:12 maltfield

Re-doing TOFU (via Netherlands VPN) 1/3 w/ pip v20.1.1

user@disp8698:/tmp/tmp.dSXPTNl8s8$ # first get some info about our internet connection
user@disp8698:/tmp/tmp.dSXPTNl8s8$ ${CURL} -s https://ifconfig.co/country | head -n1
Netherlands
user@disp8698:/tmp/tmp.dSXPTNl8s8$ ${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
user@disp8698:/tmp/tmp.dSXPTNl8s8$ 
user@disp8698:/tmp/tmp.dSXPTNl8s8$ # and today's date
user@disp8698:/tmp/tmp.dSXPTNl8s8$ date -u +"%Y-%m-%d"
2021-12-01
user@disp8698:/tmp/tmp.dSXPTNl8s8$ 
user@disp8698:/tmp/tmp.dSXPTNl8s8$ # get the file
user@disp8698:/tmp/tmp.dSXPTNl8s8$ pip3 download ${PIP_PKGS}
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Collecting pyinstaller
  Downloading pyinstaller-4.7-py3-none-manylinux2014_x86_64.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.9 MB/s 
  Saved ./pyinstaller-4.7-py3-none-manylinux2014_x86_64.whl
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-4.8.2-py3-none-any.whl (17 kB)
  Saved ./importlib_metadata-4.8.2-py3-none-any.whl
Collecting setuptools
  Downloading setuptools-59.4.0-py3-none-any.whl (952 kB)
     |████████████████████████████████| 952 kB 3.8 MB/s 
  Saved ./setuptools-59.4.0-py3-none-any.whl
Collecting pyinstaller-hooks-contrib>=2020.6
  Downloading pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl (215 kB)
     |████████████████████████████████| 215 kB 3.8 MB/s 
  Saved ./pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
Collecting altgraph
  Downloading altgraph-0.17.2-py2.py3-none-any.whl (21 kB)
  Saved ./altgraph-0.17.2-py2.py3-none-any.whl
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Downloading typing_extensions-4.0.1-py3-none-any.whl (22 kB)
  Saved ./typing_extensions-4.0.1-py3-none-any.whl
Collecting zipp>=0.5
  Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
  Saved ./zipp-3.6.0-py3-none-any.whl
Successfully downloaded pyinstaller importlib-metadata setuptools pyinstaller-hooks-contrib altgraph typing-extensions zipp
WARNING: You are using pip version 20.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
user@disp8698:/tmp/tmp.dSXPTNl8s8$ 
user@disp8698:/tmp/tmp.dSXPTNl8s8$ # checksum
user@disp8698:/tmp/tmp.dSXPTNl8s8$ sha256sum *
743628f2ac6a7c26f5d9223c91ed8ecbba535f506f4b6f558885a8a56a105857  altgraph-0.17.2-py2.py3-none-any.whl
53ccfd5c134223e497627b9815d5030edf77d2ed573922f7a0b8f8bb81a1c100  importlib_metadata-4.8.2-py3-none-any.whl
03636feec822de1d23d9753054f0b1229fb23d58723ae796f41b1127fc54f572  pyinstaller-4.7-py3-none-manylinux2014_x86_64.whl
60a57e4057fa2183bbaa81f10401a27eb7dd701ef8a11b287bb6345b571f94e7  pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
feb5ff19b354cde9efd2344ef6d5e79880ce4be643037641b49508bbb850d060  setuptools-59.4.0-py3-none-any.whl
7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b  typing_extensions-4.0.1-py3-none-any.whl
9fe5ea21568a0a70e50f273397638d39b03353731e6cbbb3fd8502a33fec40bc  zipp-3.6.0-py3-none-any.whl
user@disp8698:/tmp/tmp.dSXPTNl8s8$

maltfield avatar Dec 01 '21 21:12 maltfield

Maybe I have to go back to the tarball somehow because the pyinstaller...manylinux.whl file doesn't work for MacOS builds

  • https://github.com/BusKill/buskill-app/runs/4387406517?check_suite_focus=true
2021-12-01T21:28:29.4679930Z + /usr/local/Cellar/python/3.7.8/bin/pip3 install --ignore-installed --upgrade --cache-dir build/deps/ --no-index --find-links file:///Users/runner/work/buskill-app/buskill-app/build/deps/ build/deps/pyinstaller-4.7-py3-none-manylinux2014_x86_64.whl
2021-12-01T21:28:30.0097130Z ERROR: pyinstaller-4.7-py3-none-manylinux2014_x86_64.whl is not a supported wheel on this platform.

I ran this from our MacMini, and confirmed it got the tarball

maltfield@5129 ~ % python3 -m pip --version          
pip 20.1.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
maltfield@5129 ~ % tmpDir=`mktemp -d`
pushd "${tmpDir}"

/var/folders/kx/2fp3kfgj4dj7rx9s5mlb52640000gp/T/tmp.iabUvGw4 ~
maltfield@5129 tmp.iabUvGw4 % python3 -m pip download pyinstaller
Collecting pyinstaller
  Downloading pyinstaller-4.7.tar.gz (2.8 MB)
     |████████████████████████████████| 2.8 MB 13.3 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Saved ./pyinstaller-4.7.tar.gz
Collecting pyinstaller-hooks-contrib>=2020.6
  Downloading pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl (215 kB)
     |████████████████████████████████| 215 kB 12.5 MB/s 
  Saved ./pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
Collecting setuptools
  Using cached setuptools-59.4.0-py3-none-any.whl (952 kB)
  Saved ./setuptools-59.4.0-py3-none-any.whl
Collecting macholib>=1.8; sys_platform == "darwin"
  Using cached macholib-1.15.2-py2.py3-none-any.whl (37 kB)
  Saved ./macholib-1.15.2-py2.py3-none-any.whl
Collecting altgraph
  Using cached altgraph-0.17.2-py2.py3-none-any.whl (21 kB)
  Saved ./altgraph-0.17.2-py2.py3-none-any.whl
Successfully downloaded pyinstaller pyinstaller-hooks-contrib setuptools macholib altgraph
maltfield@5129 tmp.iabUvGw4 %
...
maltfield@5129 tmp.iabUvGw4 % shasum -a 256 *
743628f2ac6a7c26f5d9223c91ed8ecbba535f506f4b6f558885a8a56a105857  altgraph-0.17.2-py2.py3-none-any.whl
885613dd02d3e26dbd2b541eb4cc4ce611b841f827c0958ab98656e478b9e6f6  macholib-1.15.2-py2.py3-none-any.whl
2c7f4810dc5272ec1b388a7f1ff6b56d38653c1b0c9ac2d9dd54fa06b590e372  pyinstaller-4.7.tar.gz
60a57e4057fa2183bbaa81f10401a27eb7dd701ef8a11b287bb6345b571f94e7  pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
feb5ff19b354cde9efd2344ef6d5e79880ce4be643037641b49508bbb850d060  setuptools-59.4.0-py3-none-any.whl
maltfield@5129 tmp.iabUvGw4 %

The hashes of the files here match the first TOFU above -- except there's an additional file for macholib

maltfield avatar Dec 01 '21 21:12 maltfield

It's going to be a PIA to do 3TOFU from my one mac mini server, so I hacked-up a better set of commands. Note the grepping hackery of the PyPI simple API because PIP has no way to say pip download for a given platform.

CURL="/usr/bin/curl"
WGET="/usr/bin/wget --retry-on-host-error --retry-connrefused"

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

# GET THE FILES

# altgraph
download_url=`${CURL} -s https://pypi.org/simple/altgraph/ | grep -oE "https://.*altgraph-0.17.2-py2.py3-none-any.whl#"`
${WGET} ${download_url}

# macholib
download_url=`${CURL} -s https://pypi.org/simple/macholib/ | grep -oE "https://.*macholib-1.15.2-py2.py3-none-any.whl#"`
${WGET} ${download_url}

# pyinstaller
download_url=`${CURL} -s https://pypi.org/simple/pyinstaller/ | grep -oE "https://.*pyinstaller-4.7.tar.gz#"`
${WGET} ${download_url}

# pyinstaller-hooks-contrib
download_url=`${CURL} -s https://pypi.org/simple/pyinstaller-hooks-contrib/ | grep -oE "https://.*pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl#"`
${WGET} ${download_url}

# setuptools
download_url=`${CURL} -s https://pypi.org/simple/setuptools/ | grep -oE "https://.*setuptools-59.4.0-py3-none-any.whl#"`
${WGET} ${download_url}

# checksum
sha256sum *

maltfield avatar Dec 01 '21 22:12 maltfield

So here's the third attempt of TOFU 1/3 (also VPN via Netherlands)

user@disp2324:/tmp/tmp.bzHGsqimss$ # first get some info about our internet connection
user@disp2324:/tmp/tmp.bzHGsqimss$ ${CURL} -s https://ifconfig.co/country | head -n1
Netherlands
user@disp2324:/tmp/tmp.bzHGsqimss$ ${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # and today's date
user@disp2324:/tmp/tmp.bzHGsqimss$ date -u +"%Y-%m-%d"
2021-12-01
user@disp2324:/tmp/tmp.bzHGsqimss$ # GET THE FILES
user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # altgraph
user@disp2324:/tmp/tmp.bzHGsqimss$ download_url=`${CURL} -s https://pypi.org/simple/altgraph/ | grep -oE "https://.*altgraph-0.17.2-py2.py3-none-any.whl#"`
user@disp2324:/tmp/tmp.bzHGsqimss$ ${WGET} ${download_url}
--2021-12-01 23:09:40--  https://files.pythonhosted.org/packages/84/3f/1a5c9bef54cac9bf41edd6f4aaf61cd52ed578e10ccc607e0278012cb4a5/altgraph-0.17.2-py2.py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21200 (21K) [application/octet-stream]
Saving to: ‘altgraph-0.17.2-py2.py3-none-any.whl’

altgraph-0.17.2-py2. 100%[=====================>]  20.70K  --.-KB/s    in 0.001s  

2021-12-01 23:09:40 (29.9 MB/s) - ‘altgraph-0.17.2-py2.py3-none-any.whl’ saved [21200/21200]

user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # macholib
user@disp2324:/tmp/tmp.bzHGsqimss$ download_url=`${CURL} -s https://pypi.org/simple/macholib/ | grep -oE "https://.*macholib-1.15.2-py2.py3-none-any.whl#"`
user@disp2324:/tmp/tmp.bzHGsqimss$ ${WGET} ${download_url}
--2021-12-01 23:09:41--  https://files.pythonhosted.org/packages/59/87/d92bad3903ad3b6e10e949ac1bfbfc660433c2a8ae26849947f23299cb41/macholib-1.15.2-py2.py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37678 (37K) [application/octet-stream]
Saving to: ‘macholib-1.15.2-py2.py3-none-any.whl’

macholib-1.15.2-py2. 100%[=====================>]  36.79K  --.-KB/s    in 0.01s   

2021-12-01 23:09:41 (3.26 MB/s) - ‘macholib-1.15.2-py2.py3-none-any.whl’ saved [37678/37678]

user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # pyinstaller
user@disp2324:/tmp/tmp.bzHGsqimss$ download_url=`${CURL} -s https://pypi.org/simple/pyinstaller/ | grep -oE "https://.*pyinstaller-4.7.tar.gz#"`
user@disp2324:/tmp/tmp.bzHGsqimss$ ${WGET} ${download_url}
--2021-12-01 23:09:41--  https://files.pythonhosted.org/packages/00/ca/58dd68fee42490be1c86c9e912fc9ad0bf44c72edd882397ad11c21fbecb/pyinstaller-4.7.tar.gz
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2783049 (2.7M) [application/x-tar]
Saving to: ‘pyinstaller-4.7.tar.gz’

pyinstaller-4.7.tar. 100%[=====================>]   2.65M  3.00MB/s    in 0.9s    

2021-12-01 23:09:42 (3.00 MB/s) - ‘pyinstaller-4.7.tar.gz’ saved [2783049/2783049]

user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # pyinstaller-hooks-contrib
user@disp2324:/tmp/tmp.bzHGsqimss$ download_url=`${CURL} -s https://pypi.org/simple/pyinstaller-hooks-contrib/ | grep -oE "https://.*pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl#"`
user@disp2324:/tmp/tmp.bzHGsqimss$ ${WGET} ${download_url}
--2021-12-01 23:09:42--  https://files.pythonhosted.org/packages/80/74/4c885df43604c4ae570610e187052f29d806d582e398c2e48b83dad74610/pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 215202 (210K) [application/octet-stream]
Saving to: ‘pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl’

pyinstaller_hooks_co 100%[=====================>] 210.16K  --.-KB/s    in 0.09s   

2021-12-01 23:09:42 (2.39 MB/s) - ‘pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl’ saved [215202/215202]

user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # setuptools
user@disp2324:/tmp/tmp.bzHGsqimss$ download_url=`${CURL} -s https://pypi.org/simple/setuptools/ | grep -oE "https://.*setuptools-59.4.0-py3-none-any.whl#"`
user@disp2324:/tmp/tmp.bzHGsqimss$ ${WGET} ${download_url}
--2021-12-01 23:09:43--  https://files.pythonhosted.org/packages/6f/ea/e12311dabc63a6a434be25e6011c1513cc95d0cf22e5f13036f75b3ec508/setuptools-59.4.0-py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.37.63, 2a04:4e42:9::319
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.37.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 952264 (930K) [application/octet-stream]
Saving to: ‘setuptools-59.4.0-py3-none-any.whl’

setuptools-59.4.0-py 100%[=====================>] 929.95K  2.91MB/s    in 0.3s    

2021-12-01 23:09:43 (2.91 MB/s) - ‘setuptools-59.4.0-py3-none-any.whl’ saved [952264/952264]

user@disp2324:/tmp/tmp.bzHGsqimss$ 
user@disp2324:/tmp/tmp.bzHGsqimss$ # checksum
user@disp2324:/tmp/tmp.bzHGsqimss$ sha256sum *
743628f2ac6a7c26f5d9223c91ed8ecbba535f506f4b6f558885a8a56a105857  altgraph-0.17.2-py2.py3-none-any.whl
885613dd02d3e26dbd2b541eb4cc4ce611b841f827c0958ab98656e478b9e6f6  macholib-1.15.2-py2.py3-none-any.whl
2c7f4810dc5272ec1b388a7f1ff6b56d38653c1b0c9ac2d9dd54fa06b590e372  pyinstaller-4.7.tar.gz
60a57e4057fa2183bbaa81f10401a27eb7dd701ef8a11b287bb6345b571f94e7  pyinstaller_hooks_contrib-2021.4-py2.py3-none-any.whl
feb5ff19b354cde9efd2344ef6d5e79880ce4be643037641b49508bbb850d060  setuptools-59.4.0-py3-none-any.whl
user@disp2324:/tmp/tmp.bzHGsqimss$

maltfield avatar Dec 01 '21 22:12 maltfield