Azure-ci-core has strict dependencies on old 3rd party packages

[sodul]

Describe the bug

When installing Azure cli alongside other 3rd party packages that have been updated recently, the strict requirements of the Azure packages is causing pip to throw errors. We are forced to then use the legacy resolver so that the installation can actually complete.

Errors:

azure-cli 2.24.2 requires PyGithub==1.38, but you'll have pygithub 1.55 which is incompatible.
azure-cli-core 2.24.2 requires cryptography<3.4,>=3.2, but you'll have cryptography 3.4.7 which is incompatible.
azure-cli-core 2.24.2 requires msal<2.0.0,>=1.10.0, but you'll have msal 1.9.0 which is incompatible.

To Reproduce:

pip install --use-deprecated=legacy-resolver azure-cli <other-3rd-party-packages>

We have a long list of requirements so getting the list of third party packages is not practical here. The Azure packages are the ones causing the most conflicts for us.

Expected Behavior

Pip should be able to resolve and install the latest Azure libraries with the latest versions of third party packages.

Environment Summary

macOS-11.4-x86_64-i386-64bit
Python 3.9.5
Installer: PIP

azure-cli 2.24.2 *

[yonzhan]

packaging

[jiasli]

azure-cli 2.24.2 requires PyGithub==1.38, but you'll have pygithub 1.55 which is incompatible.

I can bump this dependency.

azure-cli-core 2.24.2 requires cryptography<3.4,>=3.2, but you'll have cryptography 3.4.7 which is incompatible.

cryptography 3.4 depends on rust, which removes support for a number of platforms (https://github.com/pyca/cryptography/issues/5771). That's why we added cryptography<3.4 dependency.

azure-cli-core 2.24.2 requires msal<2.0.0,>=1.10.0, but you'll have msal 1.9.0 which is incompatible.

msal introduced several new features in 1.10.0 and Azure CLI relies on those features to work properly. Please update your dependency of msal to the latest.

Best solution

Azure CLI is not meant to be installed with other libraries. If you have complex dependencies, please consider

• installing Azure CLI in a separate virtual environment, or
• installing Azure CLI using official methods shown in https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

[sodul]

Azure SDK and CLI are very large so installing them in duplicate in our Docker containers would be a problem as we need both for our CI/CD pipelines. If it were to be much slimmer by removing the prior SDK versions this is something we could consider. There is an open bug to trim the prior SDK versions since they are pretty much unnecessary while wasting a large amount of space.

https://github.com/Azure/azure-sdk-for-python/issues/17801

As for msal the only packages in our tree depending on it are Azure packages, we will upgrade to the latest azure packages and see if the problem is still present.

[sodul]

This is still a problem for us:

ERROR: Cannot install -r requirements.txt (line 10) because these package versions have conflicting dependencies.

The conflict is caused by:
    azure-cli 2.49.0 depends on fabric~=2.4
    The user requested (constraint) fabric==3.1.0

[sodul]

As of azure-cli 2.53.0 we are still unable to upgrade to PyGithub 2.x due to the PyGithub~=1.38 constraint which per a prior comment should have been relaxed in 2021.

With this version the site-packages/azure folder is 1.1 GB so we cannot install the official instructions to install two copies of the SDK in our container: one for the az CLI and one for using in our own python code.

For future reference this is the file that enforce the ancient version: https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/setup.py#L143C6-L143C14

[sodul]

@yonzhan @jsntcy @jiasli This dependency conflict is still present in 2.59.0.

[sodul]

@jiasli is there any expected progress on this issue?

[sodul]

Gentle ping on this old ongoing issue.