Add GitHub Actions and Azure DevOps Pipelines

[lluppesms]

trafficstars

Purpose

This change introduces a set of GitHub Actions and Azure DevOps Pipelines to the project, letting users opt to use automated pipelines to deploy instead of azd.

Some users do not have ability to create/assign roles and permissions, so the roles have been moved to a separate bicep file and an optional parameter lets the user decide to assign those roles or not.

There are individual pipelines for deploying the bicep, website, and search index, and main pipeline that mirrors the azd up and does everything.

There is a readme.md in the .github/workflows and .azdo/pipelines folder that describe the pipelines and the variables that need to be set up for each of the scenarios.

Does this introduce a breaking change?

[ ] Yes
[X] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[X] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

• Get the code

• Set up the pipelines and variables in GitHub or Azure DevOps

• Run the pipelines

• Test the code

What to Check

Verify that the following are valid

• ...

Other Information

[jimmylevell]

I have implemented your proposed Azure Pipeline. In my case I needed to change the python version to 3.10, as it is defined my the Azure web service (https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/infra/main.bicep#L101).

If interest I can share the adapted pipeline here.

[lluppesms]

@jimmylevell - I thought this pipeline already had Python 3.10... if you want to share your changes, feel free to send me a PR. If you think that's something that may differ by environment, I'd opt to make it an optional parameter with a default.

[jimmylevell]

@lluppesms - please apologize, currently I am a little bit overwhelmed with other tasks and therefore cannot contribute with a PR. I have updated my fork with my working Azure pipeline. ps. thank you your great work, it helped my to implemented the CI/CD at my company.

[lluppesms]

@jimmylevell - Glad to hear it helped! I've had a few of my customers use these pipelines now.

[pamelafox]

It looks like the pipelines for ADO do not use the Azure Developer CLI (azd), is that right? Is there a reason not to use it? It seems like it would be simpler, but maybe I don't understand ADO (I have only used GHA). Thanks!

[lluppesms]

One of our customers did not want to use AZD. They had lots of other pipelines that use Bicep and YML and wanted to integrate with that. They had existing Resource Groups and subscriptions with tight security policies, VNETs, etc. They can't create RG's on the fly, or create service principals, and they couldn't assign security roles themselves (which this script does), etc.

I think AZD is great for simple stuff, but a little more difficult to expand/modify. Don't get me wrong - I like AZD and use it on lots of demos. But the more complex stuff with security concerts and multiple environment and multiple subscriptions and multiple components - I'm usually back to doing custom YML and Bicep. AZD has gotten much better recently with the ability to customize it. When it first came out, it was pretty difficult to use if you didn't do it exactly as original author projected.

As for GHA - I can pretty much replicate what I have in ADO in GHA also - there are equivalent actions, just haven't done it for this project yet.

Happy to chat or talk more if you have questions!

Lyle

---

From: Pamela Fox @.> Sent: Thursday, July 20, 2023 8:53 AM To: Azure-Samples/azure-search-openai-demo @.> Cc: Lyle Luppes @.>; Mention @.> Subject: Re: [Azure-Samples/azure-search-openai-demo] Add GitHub Actions and Azure DevOps Pipelines (PR #198)

It looks like the pipelines for ADO do not use the Azure Developer CLI (azd), is that right? Is there a reason not to use it? It seems like it would be simpler, but maybe I don't understand ADO (I have only used GHA). Thanks!

— Reply to this email directly, view it on GitHubhttps://github.com/Azure-Samples/azure-search-openai-demo/pull/198#issuecomment-1643970566, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2HP6JMI7YY56BFXIJN57ULXREZ5LANCNFSM6AAAAAAX7RP2JA. You are receiving this because you were mentioned.Message ID: @.***>

[pamelafox]

@lluppesms Okay, that makes sense. Thanks for the explanation! I'm not sure what the maintainers team thinks about merging all these pipelines, as I'm a little concerned about our collective ability to maintain their complexity. (I haven't used ADO myself) We should definitely keep this PR open for anyone to find, however, and I've been porting folks at it looking for ADO. I'm also looking into ADO pipelines for azd, as some have requested that.

[lluppesms]

Sounds good. LMK if you need anything or help setting up the ADO azd pipeline - that should be pretty straightforward.

Lyle

---

From: Pamela Fox @.> Sent: Thursday, July 20, 2023 11:21 AM To: Azure-Samples/azure-search-openai-demo @.> Cc: Lyle Luppes @.>; Mention @.> Subject: Re: [Azure-Samples/azure-search-openai-demo] Add GitHub Actions and Azure DevOps Pipelines (PR #198)

@lluppesmshttps://github.com/lluppesms Okay, that makes sense. Thanks for the explanation! I'm not sure what the maintainers team thinks about merging all these pipelines, as I'm a little concerned about our collective ability to maintain their complexity. (I haven't used ADO myself) We should definitely keep this PR open for anyone to find, however, and I've been porting folks at it looking for ADO. I'm also looking into ADO pipelines for azd, as some have requested that.

— Reply to this email directly, view it on GitHubhttps://github.com/Azure-Samples/azure-search-openai-demo/pull/198#issuecomment-1644221542, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2HP6JPRJRO3Y5ASILPLYB3XRFLKNANCNFSM6AAAAAAX7RP2JA. You are receiving this because you were mentioned.Message ID: @.***>

[pamelafox]

@lluppesms Ah, I realize the azd TODO samples already have such a pipeline, could you try adding https://github.com/Azure-Samples/todo-python-mongo/blob/main/.azdo/pipelines/azure-dev.yml in a branch and seeing if it works for you on ADO? That'd be very helpful.

[vhvb1989]

@pamelafox @lluppesms, here's some about pipelines and azd (azdo/github) which might be useful: https://github.com/Azure/azure-dev/blob/main/cli/azd/docs/manual-pipeline-config.md

It can help you to set up the Azdo pipeline in cases where you can't use azd pipeline config --provider azdo

But, in case you own the Azure DevOps, then you can just use azd to create the pipeline. azd can either pick an existing Azdo project/repo or create a new one. All you need is a the OrgName and a PAT (see: https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/make-azd-compatible?pivots=azd-create#configure-a-devops-pipeline)

I'm happy to help if you have some questions.

[namratam30]

I am getting below error even after successful deployment- :( Application Error If you are the application administrator, you can access the [diagnostic resources]).

When i search the logs i get below error-

Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages.	Error	Recommended Action	Further Analysis/Steps(Experimental)	2023-08-22T13:50:53.542617022Z Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages.

2023-08-22T13:50:53.542617022Z Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages.

[pamelafox]

Sometimes azd up reports a successful deploy, but App Service deployment wasn't actually successful. That message looks like an unsuccessful deploy.

Can you check Deployment Center > Logs? See if the most recent status is "Success" or "Failed".

[namratam30]

oryx_packages Here is the full error log-

Application Errors that may have caused the container to crash were detected

	ErrorRecommended ActionFurther Analysis/Steps(Experimental)2023-08-22T13:50:53.542617022Z Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages.The application maybe facing a generic error which is causing your app to fail proper startup.Click here to access your logs and review them by opening the files with the words default_docker.log.Or,Click here to access the LogStreamThis error message indicates that the virtual environment directory and package directory could not be found in the web app. To resolve this issue, you can try to create the virtual environment directory and package directory manually in the web app. You can refer to the following link for more information: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#create-a-docker-container-image-for-your-web-app.2023-08-22T13:50:55.370836023Z import openai2023-08-22T13:50:55.370839622Z ModuleNotFoundError: No module named 'openai'2023-08-22T13:50:55.380479647Z [2023-08-22 13:50:55 +0000] [69] [INFO] Worker exiting (pid: 69)The application maybe missing a Python package which is causing your app to fail proper startup.Refer this article to deploy your app properly.Or,Click here to access the LogStreamThis error message indicates that the module 'openai' is not installed in the environment. To resolve this issue, you should add the missing package in the requirements.txt file and redeploy the application. For more information, please refer to the following link: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#add-a-requirementstxt-file.	Error	Recommended Action	Further Analysis/Steps(Experimental)	2023-08-22T13:50:53.542617022Z Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages.	The application maybe facing a generic error which is causing your app to fail proper startup.Click here to access your logs and review them by opening the files with the words default_docker.log.Or,Click here to access the LogStream	This error message indicates that the virtual environment directory and package directory could not be found in the web app. To resolve this issue, you can try to create the virtual environment directory and package directory manually in the web app. You can refer to the following link for more information: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#create-a-docker-container-image-for-your-web-app.	2023-08-22T13:50:55.370836023Z import openai2023-08-22T13:50:55.370839622Z ModuleNotFoundError: No module named 'openai'2023-08-22T13:50:55.380479647Z [2023-08-22 13:50:55 +0000] [69] [INFO] Worker exiting (pid: 69)	The application maybe missing a Python package which is causing your app to fail proper startup.Refer this article to deploy your app properly.Or,Click here to access the LogStream	This error message indicates that the module 'openai' is not installed in the environment. To resolve this issue, you should add the missing package in the requirements.txt file and redeploy the application. For more information, please refer to the following link: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#add-a-requirementstxt-file.

2023-08-22T13:50:53.542617022Z Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/oryx_packages. | The application maybe facing a generic error which is causing your app to fail proper startup.Click here to access your logs and review them by opening the files with the words default_docker.log.Or,Click here to access the LogStream | This error message indicates that the virtual environment directory and package directory could not be found in the web app. To resolve this issue, you can try to create the virtual environment directory and package directory manually in the web app. You can refer to the following link for more information: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#create-a-docker-container-image-for-your-web-app. 2023-08-22T13:50:55.370836023Z import openai2023-08-22T13:50:55.370839622Z ModuleNotFoundError: No module named 'openai'2023-08-22T13:50:55.380479647Z [2023-08-22 13:50:55 +0000] [69] [INFO] Worker exiting (pid: 69) | The application maybe missing a Python package which is causing your app to fail proper startup.Refer this article to deploy your app properly.Or,Click here to access the LogStream | This error message indicates that the module 'openai' is not installed in the environment. To resolve this issue, you should add the missing package in the requirements.txt file and redeploy the application. For more information, please refer to the following link: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#add-a-requirementstxt-file.

Application Errors that may have caused the container to crash were detected Error Recommended Action Further Analysis/Steps(Experimental) 2023-08-22T13:50:53.542617022Z Writing output script to '/opt/startup/startup.sh'2023-08-22T13:50:53.603100847Z WARNING: Could not find virtual environment directory /home/site/wwwroot/antenv.2023-08-22T13:50:53.604091439Z WARNING: Could not find package directory /home/site/wwwroot/__oryx_packages__. The application maybe facing a generic error which is causing your app to fail proper startup. Click [here](https://backend-111-dev.scm.azurewebsites.net/api/logs/docker) to access your logs and review them by opening the files with the words default_docker.log.Or, Click [here](https://backend-111-dev.scm.azurewebsites.net/api/logstream) to access the LogStream This error message indicates that the virtual environment directory and package directory could not be found in the web app. To resolve this issue, you can try to create the virtual environment directory and package directory manually in the web app. You can refer to the following link for more information: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#create-a-docker-container-image-for-your-web-app. 2023-08-22T13:50:55.370836023Z import openai2023-08-22T13:50:55.370839622Z ModuleNotFoundError: No module named 'openai'2023-08-22T13:50:55.380479647Z [2023-08-22 13:50:55 +0000] [69] [INFO] Worker exiting (pid: 69) The application maybe missing a Python package which is causing your app to fail proper startup. Refer [this](https://docs.microsoft.com/en-us/azure/app-service/quickstart-python?tabs=bash&pivots=python-framework-flask#deploy-the-sample) article to deploy your app properly.Or, Click [here](https://backend-111-dev.scm.azurewebsites.net/api/logstream) to access the LogStream This error message indicates that the module 'openai' is not installed in the environment. To resolve this issue, you should add the missing package in the requirements.txt file and redeploy the application. For more information, please refer to the following link: https://docs.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#add-a-requirementstxt-file.

[pamelafox]

It says it could't find "openai" which means it did not install the dependencies from requirements.txt, which it should have done as part of the deployment step. Can you check the Deployment Center Logs as I showed in the screenshot above, and share what you see there?

[github-actions[bot]]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed.