cryptography modules not built due to PEP 517

[jayvdb]

cryptography includes three built libraries _constant_time.abi3.so, _openssl.abi3.so and _padding.abi3.so.

They are not being built by default.

I suspect it is because of the following in the logs

  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): finished with status 'done'

I guess the distutils hackery doesnt work with PEP 517 packages. I notice bcrypt also has the same in the logs and its bcrypt/_bcrypt is missing, not even at top level _bcrypt.

  Building wheel for bcrypt (PEP 517): started
  Building wheel for bcrypt (PEP 517): finished with status 'done'

Worth noting that this means likely requests[security] is currently not secure, and I am seeing a few related errors in the requests test suite, at least one I recognise as indicative of insecure requests (TestRequests.test_https_warnings), but not sure if the two are linked.

Using PIP_USE_PEP517=false globally isnt a magic solution , as with crypography and bcrypt it causes:

ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml

extra_args = "--no-user-cfg"

Doesnt do anything (c.f. its setup.py)

requirement -e git+https://github.com/pyca/cryptography#egg=cryptography didnt work: error: option --home not recognized

[jayvdb]

Deleting the pyproject.toml worked ;-)

[indygreg]

I've seen some weird behavior with pip and PEP 517 enabled projects. I need to spend some time figuring out how that world works and teach PyOxidizer to deal with it.

[jayvdb]

After deleting pyproject.toml, the build fails when linking.

The linker error is

"/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-aeea5fc255482634.rlib" "-Wl,-Bdynamic" "-ldl" "-lm" "-lbrotlidec" "-lbrotlienc" "-lstdc++" "-lpthread" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /path/to/project/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib(ec_lib.o): in function `EC_POINT_get_affine_coordinates_GF2m':
          ec_lib.c:(.text+0x1970): multiple definition of `EC_POINT_get_affine_coordinates_GF2m'; /path/to/project/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib(_openssl.0.o):/tmp/pip-install-8x96hdi7/cryptography/build/temp.linux-x86_64-3.7/_openssl.c:1232: first defined here
          /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /path/to/project/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib(ec_lib.o): in function `EC_POINT_set_affine_coordinates_GF2m':
          ec_lib.c:(.text+0x1790): multiple definition of `EC_POINT_set_affine_coordinates_GF2m'; /path/to/project/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib(_openssl.0.o):/tmp/pip-install-8x96hdi7/cryptography/build/temp.linux-x86_64-3.7/_openssl.c:1229: first defined here
          /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: /path/to/project/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib(ec_oct.o): in function `EC_POINT_set_compressed_coordinates_GF2m':
          ec_oct.c:(.text+0xb0): multiple definition of `EC_POINT_set_compressed_coordinates_GF2m'; /path/to/project/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib(_openssl.0.o):/tmp/pip-install-8x96hdi7/cryptography/build/temp.linux-x86_64-3.7/_openssl.c:1235: first defined here
          collect2: error: ld returned 1 exit status

The source of the problem is likely the customisation in https://github.com/pyca/cryptography/blob/9c759d0/src/_cffi_src/openssl/ec.py#L113

I can workaround it by rebuilding with # define OPENSSL_NO_EC2M removed from my local /usr/include/openssl/opensslconf.h . We need the header files from https://github.com/indygreg/python-build-standalone to be provided so they can be used by Python library builds - maybe only the config headers like opensslconf.h if they can be detached from the other headers and re-used as such.

And with that fix, I have all but one test of pyOpenSSL passing.

__________________________________________________________________________ TestX509StoreContext.test_verify_with_time __________________________________________________________________________

self = <tests.test_crypto.TestX509StoreContext object at 0x7f1d235b66d0>

    def test_verify_with_time(self):
        """
        `verify_certificate` raises error when the verification time is
        set at notAfter.
        """
        store = X509Store()
        store.add_cert(self.root_cert)
        store.add_cert(self.intermediate_cert)
    
        expire_time = self.intermediate_server_cert.get_notAfter()
        expire_datetime = datetime.strptime(
            expire_time.decode('utf-8'), '%Y%m%d%H%M%SZ'
        )
        store.set_time(expire_datetime)
    
        store_ctx = X509StoreContext(store, self.intermediate_server_cert)
        with pytest.raises(X509StoreContextError) as exc:
>           store_ctx.verify_certificate()
E           Failed: DID NOT RAISE <class 'OpenSSL.crypto.X509StoreContextError'>

/path/to/pyOpenSSL-19.0.0/tests/test_crypto.py:3604: Failed

idna passes all tests and test_openssl_memleak is the only interesting failures for cryptography tests, but I see those tests are disabled at https://build.opensuse.org/package/view_file/devel:languages:python/python-cryptography/skip_openssl_memleak_test.patch?expand=1 so I guess the same reason may apply here.

[jayvdb]

Deleting the pyproject.toml worked ;-)

Ideally it should be possible to specify a PyPI package name, or a pip specifier, and still use setup-py-install packaging rule. pip download could be used to fetch the package.

[jayvdb]

This is now quite urgent as attrs sdist includes a pyproject.toml, so lots of projects will start failing.

[b-long]

First off, thanks so much for this project Indygreg. It's fantastic 👍 Thank you.

I'm wondering if there's news on this issue. Is there a workaround? Is there a proposed solution that I can test?