azure-functions-python-worker icon indicating copy to clipboard operation
azure-functions-python-worker copied to clipboard

Cryptography module 44.0.0 causing error "ImportError //lib/x86_64-linux-gnu/libc.so.6 /version `GLIBC_2.33' not found"

Open gavin-aguiar opened this issue 8 months ago • 13 comments

Background:

Azure python functions apps running python versions 3.11 and below using cryptography module 44.0.0 fail with error

Exception: Full Exception : Exception while executing function /Functions.api ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException /Result /Failure Exception /ImportError //lib/x86_64-linux-gnu/libc.so.6 /version `GLIBC_2.33' not found (required by /home/site/wwwroot/.python_packages/lib/site-packages/cryptography/hazmat/bindings/_rust.abi3.so). Cannot find module. Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide /https://aka.ms/functions-modulenotfound.........................continued

Root Cause:

Any Python function app, using azure-identity or any cryptography dependent module, built on the newest version of debian based system brings in a requirement of using GLIB_C version >= 2.33. Azure functions for python 3.11 and below are built on Bullseye with GLIB_C 2.31 is installed, and this causes a library-not-found issue.

Mitigation:

Multiple approaches to mitigate:

  • Use Remote Build when deploying Retry performing the deployment using the Remote Build: https://learn.microsoft.com/en-us/azure/azure-functions/functions-deployment-technologies?tabs=windows#remote-build

  • Use older action image for azure pipelines and GitHub Action Azure Pipelines: please change the vmImageName tag from “ubuntu-latest” to “ubuntu-20.04” for your yaml file- please refer to https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software GitHub Actions: please change the runs-on tag from “ubuntu-latest” to “ubuntu-20.04” for your yaml file- please refer to https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

  • For local build , please use Ubuntu 20.04 or any Debian 11 based image when doing a zip-deploy. [to be used only if 1 and 2 cannot be done] If you use azure-identity or any library leveraging azure-identity module

  • Please pin the cryptography library to cryptography==43.0.3. After making this change, redeploy your app. For more details, refer to azure-sdk-for-python/issues/38725

Recommendation / Long-term Solution:

The official recommendation is to migrate to Python 3.12, which is currently in preview. The Python 3.12 image is built on Bookworm, and it brings in a later version of GLIB_C. Thus, the latest versions of cryptography are supported when using Python 3.12.

The ETA for 3.12 GA is the end of March 2025.

gavin-aguiar avatar Mar 12 '25 14:03 gavin-aguiar

For information, option 2 using Azure Pipelines isn't going to be available from April 1st 2025. See https://devblogs.microsoft.com/devops/upcoming-updates-for-azure-pipelines-agents-images/#ubuntu for details.

laurens1984 avatar Mar 19 '25 10:03 laurens1984

Is there an ETA for this? We are already in production with the second mitigation you proposed: using an older action image for Azure Pipelines.

But with Ubuntu-20 expiring on April 1st, we need a resolution soon. Any updates?

gitFire001 avatar Mar 19 '25 13:03 gitFire001

Hi @gitFire001, our recommendation is to migrate to Python 3.12. The latest versions of cryptography are compatible when using Python 3.12, and our GA target is end of this month.

hallvictoria avatar Mar 19 '25 17:03 hallvictoria

@hallvictoria, Could you please confirm if the fix is currently unavailable in Python 3.12 and whether it will be available exclusively in Python 3.12 by March 31st, or if it will also be available in other versions?

gitFire001 avatar Mar 20 '25 05:03 gitFire001

The fix is already available in Python 3.12. Support for python 3.12 is still in preview though, with the GA target for end of this month.

For all other python versions, apps will need to use one of the other mitigation options listed in this issue.

hallvictoria avatar Mar 20 '25 15:03 hallvictoria

update: the sample YAML now uses a container image that is guaranteed to use Debian 11 (bullseye)

@laurens1984 @gitFire001 If you want to stick to 3.11 on Debian 11 (bullseye) in Azure Pipelines you can use the mcr.microsoft.com/devcontainers/python:3.11-bullseye container image with a container job

e.g.

  - job: devContainer
    container: mcr.microsoft.com/devcontainers/python:3.11-bullseye
    displayName: Use mcr.microsoft.com/devcontainers/python:3.11-bullseye container image
    pool:
      name: 'Azure Pipelines'
      vmImage: 'ubuntu-latest'
    steps:
    - task: FuncToolsInstaller@0
      displayName: 'Install Azure Functions Core Tools'
      inputs:
        version: 'latest'
    - bash: |
        lsb_release -a
        echo Azure Function Core Tools $(func --version)
        python --version
      displayName: 'Use Azure Functions Core Tools'

This provides the ability to use a container image of choice in Azure Pipelines. You may need to break up your jobs in YAML.

geekzter avatar Mar 21 '25 07:03 geekzter

Solved installing in requirements.txt:

cryptography==43.0.3

miguelaguirrebwps avatar Mar 27 '25 15:03 miguelaguirrebwps

Given that 3.12 looks like the solution here, are there any plans to actually be able to change language version in a consumption plan function?

According to this it's not supported to change the language version in consumption plan and redeploying is far from ideal.

briandelmsft avatar May 28 '25 01:05 briandelmsft

Hi @briandelmsft, there are currently no plans to support changing the language version without redeployment.

One of the reasons for this is because changing LinuxFx version only changes the python version -- it doesn't perform any rebuilds on the function app. That means that the app's built dependencies in .python_packages are the same for both the old language version and the new one. This can result in errors like No module named '_cffi_backend', where there's a mismatch between the dependencies which were built on an older version and the current version being used.

hallvictoria avatar May 28 '25 15:05 hallvictoria

Hi @briandelmsft, there are currently no plans to support changing the language version without redeployment.

One of the reasons for this is because changing LinuxFx version only changes the python version -- it doesn't perform any rebuilds on the function app. That means that the app's built dependencies in .python_packages are the same for both the old language version and the new one. This can result in errors like No module named '_cffi_backend', where there's a mismatch between the dependencies which were built on an older version and the current version being used.

@hallvictoria but if I was doing using WEBSITE_RUN_FROM_PACKAGE, is there any reason that changing the package to a zip built for 3.12 AND changing the runtime to 3.12 wouldn't work for a consumption plan function? It seems to work fine in all my testing.

briandelmsft avatar May 28 '25 15:05 briandelmsft

I haven't personally tested or reviewed that specific scenario, so I can't say with /complete/ certainty that it would work as anticipated. However, AFAIK as long as the app's built dependencies match the current python version and the zip is correctly formatted, then there shouldn't be any issues. For consumption specifically, I believe you'll need to manually sync triggers and restart the app to ensure the changes are picked up correctly -- assuming the URL hasn't changed, only the contents.

Is this scenario not working in your case?

hallvictoria avatar May 29 '25 15:05 hallvictoria

I wrestled with this for a few days, including Microsoft telling me to entertain upgrading off Consumption Plan to higher paid plans :/

For me, the root cause was discovered after a lot of trial and error:

  • I had built Python function using V3.12 using Azure Build pipeline. So the artifact is 3.12.
  • The Azure Function App in Azure was created using Bicep IaC to target Python V3.10 and never got upgraded to V3.12.
  • Then when releasing using Azure DevOps release pipeline, the Azure Functions Deploy task merrily pushed V3.12 artifact down onto the V3.10 function app, breaking cryptography and causing the function to not appear, without any errors logged.

So the solution is:

IaC Bicep scripts

Ensure your Microsoft.Web/sites properties|siteConfig has the correct versions: linuxFxVersion = PYTHON|3.12 pythonVersion = 3.12

Build Pipeline

Use Python Version|Version Spec = 3.12

Release Pipeline

Azure Functions Deploy task Runtime stack = PYTHON|3.12

To recap:

  • Ensure Python 3.12 is consistant across IaC scripts, build and release pipelines
  • Redeploy your Azure Function site to upgrade it to 3.12.
  • Redeploy your Azure Function code (in 3.12) on top of that.
  • Don't specify cryptography at all in your requirements file (let azure-identity handle that)

Everything should then be happy days!

markbanksaamc avatar Jun 09 '25 05:06 markbanksaamc

Hey everyone! 👋

I've been dealing with this exact issue for a few days, and I wanted to share a solution that has worked reliably for our production deployments.

📋 Quick Context

After adding the cryptography module to our application, we were getting these familiar errors:

  • 0 functions found (Custom)
  • Exceeded language worker restart retry count for runtime:python
  • No HTTP routes mapped
  • ❌ Import errors

While Python 3.12 will likely resolve this long-term, many teams (including ours) are still on Python 3.11 and need an immediate solution.

✅ Our Solution: CLI Remote Build (Not Oryx)

Following Microsoft's official guidance and this issue, the most reliable fix is using CLI remote build instead of Oryx builds. The issue occurs when using the official Azure/functions-action@v1 with Oryx build, but resolves when using the native Azure Functions CLI deployment method.

🔧 Our Specific Problem: Oryx vs CLI Builds in Github Actions.

What breaks (Oryx Build):

- uses: Azure/functions-action@v1  # ❌ Uses Oryx build system
  with:
    scm-do-build-during-deployment: true
    enable-oryx-build: true  # This is what's broken!

What works (CLI Remote Build):

func azure functionapp publish your-app --python  # ✅ This works!

🚀 Community Action Available

I've created and published a GitHub Action that implements the working CLI approach:

Azure Functions Remote Deploy

Why This Works

  • Uses the exact same func azure functionapp publish --python command that works manually
  • Handles all the Azure Functions Core Tools setup automatically
  • Performs CLI remote build instead of problematic Oryx build
  • Drop-in replacement for the broken Azure/functions-action@v1 with Oryx

Usage Example

- name: Deploy Azure Functions
  uses: OpenWit-Technologies/[email protected]
  with:
    app-name: 'your-function-app'
    package: './your-functions-folder'
    python-version: '3.11'

Migration from Broken Official Action

Replace this (Oryx Build - broken):

- uses: Azure/functions-action@v1  # ❌ Oryx build broken since Nov 2024
  with:
    scm-do-build-during-deployment: true
    enable-oryx-build: true

With this (CLI Build - working):

- uses: OpenWit-Technologies/[email protected]  # ✅ Works!

📊 Additional Compatibility Notes

For maximum compatibility, I also recommend:

  1. Runner OS: Use ubuntu-22.04 (since ubuntu-20.04 is being deprecated)
  2. Cryptography Pin: Keep cryptography==43.0.3 in requirements.txt
  3. CLI Remote Build: Primary solution (what the action handles)

🌟 Benefits

  • Immediate fix for existing Python 3.11 projects
  • Production tested in our CI/CD pipelines
  • Community maintained and open source
  • Future compatible - will work with Python 3.12 too

I am sure this affects thousands of developers, so I wanted to make the solution easily accessible. The action is free to use and implements Microsoft's recommended CLI approach in a reusable way.

Hope this helps others who are stuck with this issue! Let me know if you have any questions or suggestions for improvements.


Links:

Feel free to star ⭐ the repository if it helps you - visibility helps other developers find this solution!

CaduceusInc avatar Jun 29 '25 17:06 CaduceusInc

Hi! Here I'm using python 3.12 with self-hosted agent (ubuntu 22.04) and cryptography-45.0.7 but still getting the error :( Any news?

ayelencasamassa avatar Sep 03 '25 23:09 ayelencasamassa

Hey @ayelencasamassa ,

On our side, still have the cryptography module pinned in our requirements, which makes the app build and start up normally.

cryptography==43.0.3

Hope this helps

lmoy28 avatar Sep 04 '25 01:09 lmoy28

The app seems to work fine, looks like the warning from MS will remain just in case. Thanks!

ayelencasamassa avatar Sep 04 '25 17:09 ayelencasamassa

I may be experiencing the same problem because of this package but it was obfuscated inside the httpx package that I recently started using (python 3.12) :(. My local deployments work normally, but when it comes to deploying the function app that we do containerized, we get a runtime error / the function app does not get deployed and I'm guessing this is it. Having trouble gettings logs from the online deployment.

NPap0 avatar Sep 05 '25 12:09 NPap0