Python3 implementation of fetch-params.sh

[mdr0id]

Follow up to PR #3741.

TLDR; goal is to make scripting consistent (e.g. python calling into python instead of python calling into bash), add support for Windows nodes, update to functioning IPFS node, add IPFS check routine

[mdr0id]

I'm going to check out the cross compile issues I found in pyinstaller. Otherwise the core logic of this appears stable across platforms. Feedback is welcome.

I hope to update some of the logging cosmetics in the next couple days.

[rex4539]

When running pyinstaller --onefile fetch-pyarams.py:

Unable to find "/Library/Python/3.7m/include/pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't
come with proper library files. This usually happens by missing development
package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
  * apt-get install python3-dev
  * apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)

I do have that file but it's installed under /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/Headers/pyconfig.h

[mdr0id]

@rex4539 Updated PR per comments.

Added some clean up to ipfs routine to make it more stable/formal. Ty @benzcash for review

Minus some squashing and minor string cosmetics, hope this is nearing final form. Feedback welcome.

[rex4539]

I tested python3 fetch-pyarams.py --ipfs with ipfs not running.

/Users/rex/zcash/zcutil on branch master rex@MacBook-Pro-2018% python3 fetch-pyarams.py --ipfs
[D 200412 07:25:49 fetch-pyarams:123] Host: Darwin
[D 200412 07:25:49 fetch-pyarams:124] Parameter path: /Users/rex/Library/Application Support/ZcashParams/
Zcash - fetch-params.py 

This script will fetch the Zcash zkSNARK parameters and verify their integrity with sha256sum. 

If they already exist locally, it will verify SHAs and exit. 
 
[W 200412 07:25:49 fetch-pyarams:247] /Users/rex/Library/Application Support/ZcashParams/sapling-spend.params does not exist and will now be downloaded...
DESTINATION:  /Users/rex/Library/Application Support/ZcashParams/sapling-spend.params
REQ:  /ipfs/QmaaA4e7C4QkxrooomzmrjdFgv6WXPGpizj6gcKxdRUnkW
Error: merkledag: not found
Error: lstat /Users/rex/Library/Application Support/ZcashParams/sapling-spend.params: no such file or directory

USAGE
  ipfs add <path>... - Add a file or directory to ipfs.

  ipfs add [--recursive | -r] [--dereference-args] [--stdin-name=<stdin-name>] [--hidden | -H] [--quiet | -q] [--quieter | -Q] [--silent]
           [--progress | -p] [--trickle | -t] [--only-hash | -n] [--wrap-with-directory | -w] [--chunker=<chunker> | -s] [--pin=false] [--raw-leaves]
           [--nocopy] [--fscache] [--cid-version=<cid-version>] [--hash=<hash>] [--inline] [--inline-limit=<inline-limit>] [--] <path>...

  Adds contents of <path> to ipfs. Use -r to add directories (recursively).

  For more information about each command, use:
  'ipfs add <subcmd> --help'

[E 200412 07:25:49 fetch-pyarams:198] Download failed: Multihash on sapling-spend.params does NOT match.
[W 200412 07:25:49 fetch-pyarams:247] /Users/rex/Library/Application Support/ZcashParams/sapling-output.params does not exist and will now be downloaded...
DESTINATION:  /Users/rex/Library/Application Support/ZcashParams/sapling-output.params
REQ:  /ipfs/QmQ8E53Fpp1q1zXvsqscskaQXrgyqfac9b3AqLxFxCubAz
Error: merkledag: not found
Error: lstat /Users/rex/Library/Application Support/ZcashParams/sapling-output.params: no such file or directory

USAGE
  ipfs add <path>... - Add a file or directory to ipfs.

  ipfs add [--recursive | -r] [--dereference-args] [--stdin-name=<stdin-name>] [--hidden | -H] [--quiet | -q] [--quieter | -Q] [--silent]
           [--progress | -p] [--trickle | -t] [--only-hash | -n] [--wrap-with-directory | -w] [--chunker=<chunker> | -s] [--pin=false] [--raw-leaves]
           [--nocopy] [--fscache] [--cid-version=<cid-version>] [--hash=<hash>] [--inline] [--inline-limit=<inline-limit>] [--] <path>...

  Adds contents of <path> to ipfs. Use -r to add directories (recursively).

  For more information about each command, use:
  'ipfs add <subcmd> --help'

[E 200412 07:25:49 fetch-pyarams:198] Download failed: Multihash on sapling-output.params does NOT match.
[W 200412 07:25:49 fetch-pyarams:247] /Users/rex/Library/Application Support/ZcashParams/sprout-groth16.params does not exist and will now be downloaded...
DESTINATION:  /Users/rex/Library/Application Support/ZcashParams/sprout-groth16.params
REQ:  /ipfs/QmWFuKQ1JgwJBrqYgGHDEmGSXR9pbkv51NYC1yNUaWwpMU
Error: merkledag: not found
Error: lstat /Users/rex/Library/Application Support/ZcashParams/sprout-groth16.params: no such file or directory

USAGE
  ipfs add <path>... - Add a file or directory to ipfs.

  ipfs add [--recursive | -r] [--dereference-args] [--stdin-name=<stdin-name>] [--hidden | -H] [--quiet | -q] [--quieter | -Q] [--silent]
           [--progress | -p] [--trickle | -t] [--only-hash | -n] [--wrap-with-directory | -w] [--chunker=<chunker> | -s] [--pin=false] [--raw-leaves]
           [--nocopy] [--fscache] [--cid-version=<cid-version>] [--hash=<hash>] [--inline] [--inline-limit=<inline-limit>] [--] <path>...

  Adds contents of <path> to ipfs. Use -r to add directories (recursively).

  For more information about each command, use:
  'ipfs add <subcmd> --help'

[E 200412 07:25:49 fetch-pyarams:198] Download failed: Multihash on sprout-groth16.params does NOT match.

Instead of these errors, I would have expected a message ipfs is not running. Please start it and try again.

[mdr0id]

FWIW, perhaps we use https://github.com/hsanjuan/ipfs-lite if IPFS daemon is pain point for users.

(UPDATE: Initial tests show that the above tool is slow and might not be useful)

[mdr0id]

@charlieok or @rex4539 if it is possible for a final MacOS test after cleanup it would be greatly appreciated.

I will verify on Windows one more time, but the Linux paths appear to work gracefully.

[mdr0id]

Tested ACK on Windows, after above deadlock issues was found

[rex4539]

[torizoemitchell]

ACK. was able to build and run zcashd after pulling fresh params with fetch-params.py.

[mdr0id]

Just remember what I did not address in this ticket but Str4d and I had spoke about long ago.

pyinstaller --onefile fetch-params.py

This will allow the fetch-params.py to be bundled up for distribution.

[daira]

I would rather not depend on logzero, requests, or tqdm. fetch-params has to Just Work, and the easiest way to reduce the set of things that can go wrong is to not depend on anything outside the standard library. This seems quite feasible:

• logging instead of logzero;
• urllib.request instead of requests;
• one of the solutions that doesn't depend on external libraries here, instead of tqdm.

[mdr0id]

@daira Agree. Do we have any requirements on the status provided to the user during each download? If it is fine with core, I'd like to remove the cosmetics and slim down the packaged binary that pyinstaller creates.

[sellout]

fetch-params.sh has moved a bit since this was created. Also, the plan going forward is to integrate fetch-params with zcashd (rather than a separate program) using the Rust implementation (See #4491).

So I don’t think this PR is relevant any more.