Deployment GH action fails while oryx tries to install hdbscan package

[mberneaud]

Bug Report

• At what date and time did you most recently experience the problem? 25.08.22 around 10am GMT+1 The first time I experienced the issue was on 23.08.22 in the morning. Previously, the same build with an identical requirements.txt had succeeded an 16.08.22

• Where did you experience the problem? E.g. Azure Web Apps, Azure Functions, Azure Container Registry, or offline use. Build & Deploy Azure Function using GH actions using the yml file created automatically in the Deployment Center.

• If your repo is publicly available please share its URL: Repo is non-public

• What happened? GH action build finished successfully, but deployment failed at the installation of the hdbscan package in requirements.txt by oryx with exit code 1. See log output below. I suspect this could be a result of a new ubuntu-latest runner-image used by GH actions which was released between my latest successful deployment and my first unsuccesful deployment on 21.08.22. As the workflow yml file references the ubuntu-latest image, changes to the available underlying image could be to blame. I suspect that because different website suggest installing specific python versions or updating compilers as a fix for the same error.

• What did you expect or want to happen? For the GH action runner to install my requirements and afterwards deploy the image built by oryx

• How can we reproduce it? Run an Oryx deployment for a Python Azure Function that requires the hdbscan package using GH actions via the workflow file generated by the Azure Function Deployment Center.

• Do you have log output? Please include between the backticks:

Run Azure/functions-action@v1
Successfully parsed SCM credential from publish-profile format.
Using SCM credential for authentication, GitHub Action will not perform resource validation.
Successfully acquired app settings from function app (with SCM credential)!
Will archive . into /home/runner/work/_temp/temp_web_package_5079435715684786.zip as function app content
Will use Kudu https://<scmsite>/api/zipdeploy to deploy since publish-profile is detected.
Setting SCM_DO_BUILD_DURING_DEPLOYMENT in Kudu container to true
Update using Client.updateAppSettingViaKudu
Response with status code 204
App setting SCM_DO_BUILD_DURING_DEPLOYMENT propagated to Kudu container
Setting ENABLE_ORYX_BUILD in Kudu container to true
Update using Client.updateAppSettingViaKudu
Response with status code 204
App setting ENABLE_ORYX_BUILD propagated to Kudu container
Package deployment using ZIP Deploy initiated.
Updating submodules.
Preparing deployment for commit id 'b0c3f85d-d'.
PreDeployment: context.CleanOutputPath False
PreDeployment: context.OutputPath /home/site/wwwroot
Repository path is /tmp/zipdeploy/extracted
Running oryx build...
Command: oryx build /tmp/zipdeploy/extracted -o /tmp/build/expressbuild --platform python --platform-version 3.9 -i /tmp/8da8678f46fed78 -p packagedir=.python_packages/lib/site-packages
Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
You can report issues at https://github.com/Microsoft/Oryx/issues

Oryx Version: 0.2.20220308.4, Commit: c92fa6a2d6fc[14](https://github.com/loopinggroup/amgdbs-functions/runs/8012336809?check_suite_focus=true#step:3:15)dc9646f80e2bb2e393a5cdc258, ReleaseTagName: 20220308.4

Build Operation ID: |zwlWMd5pkBQ=.759723cb_
Repository Commit : b0c3f85d-dbac-4434-9a9a-0e54d787cc5d

Detecting platforms...
Detected following platforms:
  python: 3.9.7
Version '3.9.7' of platform 'python' is not installed. Generating script to install it...

Using intermediate directory '/tmp/8da8678f46fed78'.

Copying files to the intermediate directory...
Done in 0 sec(s).

Source directory     : /tmp/8da8678f46fed78
Destination directory: /tmp/build/expressbuild


Downloading and extracting 'python' version '3.9.7' to '/tmp/oryx/platforms/python/3.9.7'...
Downloaded in 0 sec(s).
Verifying checksum...
Extracting contents...
performing sha512 checksum for: python...
Done in 3 sec(s).

Python Version: /tmp/oryx/platforms/python/3.9.7/bin/python3.9
Creating directory for command manifest file if it doesnot exist
Removing existing manifest file

Running pip install...
Done in 139 sec(s).
"2022-08-25 09:07:53"|ERROR|ERROR: Could not build wheels for hdbscan which use PEP 517 and cannot be installed directly | Exit code: 1 | Please review your requirements.txt | More information: https://aka.ms/troubleshoot-python
\n/opt/Kudu/Scripts/starter.sh oryx build /tmp/zipdeploy/extracted -o /tmp/build/expressbuild --platform python --platform-version 3.9 -i /tmp/8da8678f46fed78 -p packagedir=.python_packages/lib/site-packages

Generating summary of Oryx build
Deployment Log file does not exist in /tmp/oryx-build.log
The logfile at /tmp/oryx-build.log is empty. Unable to fetch the summary of build
Deployment Failed.
Error: Failed to deploy web package to App Service.
Error: Execution Exception (state: PublishContent) (step: Invocation)
Error:   When request Azure resource at PublishContent, zipDeploy : Failed to use /home/runner/work/_temp/temp_web_package_50794357[15](https://github.com/loopinggroup/amgdbs-functions/runs/8012336809?check_suite_focus=true#step:3:16)68[47](https://github.com/loopinggroup/amgdbs-functions/runs/8012336809?check_suite_focus=true#step:3:48)86.zip as ZipDeploy content
Error:     Package deployment using ZIP Deploy failed. Refer logs for more details.
Error: Deployment Failed!

[qianz2]

Hi @mberneaud , for the contexts you mentioned, the build and deploy was previously successful in Azure Function GH action, but recently failed with same workflow and same requirement.txt file. Would you mind to share the workflow YML file to us? Oryx didn't change much things to the GH image recently, so I suspect it maybe something caused by Azure Function switching its base image?

[mberneaud]

Hey @qianz2 , this could very well be the case. The workflow yml file that failed is essentially the unchanged file as created by setting up the workflow file via the deployment center dialogue on the function app.

Here is the full YML file.

# Docs for the Azure Web Apps Deploy action: https://github.com/azure/functions-action
# More GitHub Actions for Azure: https://github.com/Azure/actions
# More info on Python, GitHub Actions, and Azure Functions: https://aka.ms/python-webapps-actions

name: Build and deploy Python project to Azure Function App - amgfuncapp

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  AZURE_FUNCTIONAPP_PACKAGE_PATH: '.' # set this to the path to your web app project, defaults to the repository root
  PYTHON_VERSION: '3.9' # set this to the python version to use (supports 3.6, 3.7, 3.8)

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Setup Python version
        uses: actions/setup-python@v1
        with:
          python-version: ${{ env.PYTHON_VERSION }}

      - name: Create and start virtual environment
        run: |
          python -m venv venv
          source venv/bin/activate
      - name: Install dependencies
        run: pip install -r requirements.txt

      # Optional: Add step to run tests here

      - name: Upload artifact for deployment job
        uses: actions/upload-artifact@v2
        with:
          name: python-app
          path: |
            . 
            !venv/
  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'dev'
      url: ${{ steps.deploy-to-function.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v2
        with:
          name: python-app
          path: .

      - name: 'Deploy to Azure Functions'
        uses: Azure/functions-action@v1
        id: deploy-to-function
        with:
          app-name: 'amgfuncapp'
          slot-name: 'dev'
          package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_1AF0A8840A194775BCFC0E923751CF8C }}
          scm-do-build-during-deployment: true
          enable-oryx-build: true

After I submitted the issue, I tried a few things to look deeper into what might be causing the issue. I think it's not the underlying runner-image, as the first installation of the requirements.txt in the first job of the yml file works without issues and it's using the same runner-image as the subsequent deploy job. So the issue might be, as you suggested, in the Azure Function base image.

Edit: Misspelled username

[bala1802]

Hello @mberneaud and @qianz2 I am facing the similar issue, is this resolved?

[mberneaud]

@bala1802 No, I have never heard back since my latest message here. In the meantime, I have converted my Function App from a code-based app to an app that uses a custom Docker image. For that I had to delete my function app and the storage account and recreate everything following the steps for using a custom Linux image for running the function app.

Once deployed, the workflows are similar and having a Github Action build the image works just fine with the default settings and files created from within the Function App deployment center.

Now everything works, but the issue with Oryx remains.

EDIT: I see that you have forked BERTopic on your page. My issues also arose from using BERTopic within my function app, which required the HDBScan install. As I said, the only way to fix it was to migrate the function app to use a custom Linux image instead. There's a good guide on the Azure Function docs on how to create it.

[pauld-msft]

Hello @bala1802, would you be able to share the logs from your failure, similar to the logs shared in the original message? Also, do you have a public repo that we would be able to use to reproduce the issue? Thanks

[simonjj]

We're closing this issue due to its age. If this is still relevant and impacting you please open a new issue at https://github.com/projectkudu/kudu and link back to this issue for reference.