skypilot
skypilot copied to clipboard
[Dependency] `cryptography` package does not support openssl>3.0
cryptography package used in the authentication.py does not support openssl>3.0. It seems the newly created conda environment can have the openssl>3.0 by default and cause the following error for sky launch. related issue
I 08-24 22:05:25 cloud_vm_ray_backend.py:2604] Creating a new cluster: "sky-spot-controller-527bdd84" [1x GCP(n1-highmem-8)].
I 08-24 22:05:25 cloud_vm_ray_backend.py:2604] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 08-24 22:05:25 cloud_vm_ray_backend.py:892] To view detailed progress: tail -n100 -f /home/sky1/sky_logs/sky-2022-08-24-22-05-25-608819/provision.log
I 08-24 22:05:26 authentication.py:211] OS Login is enabled for GCP project prod-635e. Running additional authentication steps.
I 08-24 22:05:26 authentication.py:211] OS Login is enabled for GCP project prod-635e. Running additional authentication steps.
W 08-24 22:05:26 common_utils.py:184] Caught /home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: symbol EC_POINT_get_affine_coordinates_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference. Retrying.
I 08-24 22:05:28 authentication.py:211] OS Login is enabled for GCP project prod-635e. Running additional authentication steps.
I 08-24 22:05:28 authentication.py:211] OS Login is enabled for GCP project prod-635e. Running additional authentication steps.
W 08-24 22:05:28 common_utils.py:184] Caught /home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: symbol EC_POINT_get_affine_coordinates_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference. Retrying.
I 08-24 22:05:31 authentication.py:211] OS Login is enabled for GCP project prod-635e. Running additional authentication steps.
I 08-24 22:05:31 authentication.py:211] OS Login is enabled for GCP project prod-635e. Running additional authentication steps.
W 08-24 22:05:31 common_utils.py:184] Caught /home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: symbol EC_POINT_get_affine_coordinates_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference. Retrying.
Fetching managed spot job statuses...
No managed spot job has been run.
No cached job status table found.
Traceback (most recent call last):
File "/home/sky1/mambaforge/envs/sky-2/bin/sky", line 8, in <module>
sys.exit(cli())
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 147, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/cli.py", line 761, in invoke
return super().invoke(ctx)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 147, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/cli.py", line 761, in invoke
return super().invoke(ctx)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 168, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 168, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/cli.py", line 2317, in spot_launch
retry_until_up=retry_until_up)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/execution.py", line 464, in spot_launch
retry_until_up=True,
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/execution.py", line 190, in _execute
retry_until_up=retry_until_up)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 168, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 147, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/backends/backend.py", line 50, in provision
cluster_name, retry_until_up)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/backends/cloud_vm_ray_backend.py", line 1615, in _provision
task, to_provision_config, dryrun, stream_logs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 168, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/backends/cloud_vm_ray_backend.py", line 1303, in provision_with_retries
cluster_exists=cluster_exists)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/backends/cloud_vm_ray_backend.py", line 915, in _retry_region_zones
dryrun=dryrun)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 168, in _record
return f(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/backends/backend_utils.py", line 744, in write_cluster_config
_add_auth_to_cluster_config(cloud, yaml_path)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/backends/backend_utils.py", line 791, in _add_auth_to_cluster_config
config = auth.setup_gcp_authentication(config)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/utils/common_utils.py", line 182, in method_with_retries
return method(*args, **kwargs)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/authentication.py", line 250, in setup_gcp_authentication
get_or_generate_keys(private_key_path, public_key_path)
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/authentication.py", line 84, in get_or_generate_keys
public_key, private_key = generate_rsa_key_pair()
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/sky/authentication.py", line 40, in generate_rsa_key_pair
key = rsa.generate_private_key(backend=default_backend(),
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/backends/__init__.py", line 8, in default_backend
from cryptography.hazmat.backends.openssl.backend import backend
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 6, in <module>
from cryptography.hazmat.backends.openssl.backend import backend
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 63, in <module>
from cryptography.hazmat.bindings.openssl import binding
File "/home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: /home/sky1/mambaforge/envs/sky-2/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: symbol EC_POINT_get_affine_coordinates_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference
The current workaround is conda install -c conda-forge openssl=1.1