azure-search-openai-demo icon indicating copy to clipboard operation
azure-search-openai-demo copied to clipboard

Azure Cognitive Search - Index not deployed. Why?

Open timzhan opened this issue 3 years ago • 15 comments

Hi, I followed the installation guide and run "azd up" and deployed the resources on Azure. However, I got below error. It seems the Indexer was not deployed. I verified from my Azure Portal and couldn't find the indexer.

Could you help?

Thanks! Tim

error2

timzhan avatar Mar 15 '23 07:03 timzhan

I got the same error right now. Could you please let me know if you've managed to find the solution of this problem? :D

Thanks! Mihai

mihaimunteanu28 avatar Mar 15 '23 09:03 mihaimunteanu28

Same problem too!

pietheinstrengholt avatar Mar 15 '23 12:03 pietheinstrengholt

Getting the same error message

sindre-bouvet avatar Mar 15 '23 12:03 sindre-bouvet

We've seen this problem when there's a permissions error during azd up, the script doesn't run at first and even if you fix it later, you've skipped the step to load the data. Curious if for each of you that mentioned this, you saw an error during the first run or if there's a different failure.

The script that creates the index, prepares the data, and loads it to blob storage and search, is in scripts/prepdocs.py, though running it through prepdocs.ps1 gets you all the parameters wired up based on your environment.

If the original run had failed while assigning permissions, you'll want to add the roles of blob contributor and search contributor to your user for storage/search services respectively before you run this. You can do that in the Azure portal, go to the blob/search service and use the "access control" section on the left.

pablocastro avatar Mar 15 '23 14:03 pablocastro

Yes, I'm seeing this error message when running "azd up"

Ensuring search index gptkbindex exists
Traceback (most recent call last):
  File "/home/sindre/projects/openai/./scripts/prepdocs.py", line 208, in <module>
    create_search_index()
  File "/home/sindre/projects/openai/./scripts/prepdocs.py", line 146, in create_search_index
    if args.index not in index_client.list_index_names():
  File "/home/sindre/anaconda3/lib/python3.9/site-packages/azure/core/paging.py", line 132, in __next__
    return next(self._page_iterator)
  File "/home/sindre/anaconda3/lib/python3.9/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/sindre/anaconda3/lib/python3.9/site-packages/azure/search/documents/indexes/_generated/operations/_indexes_operations.py", line 520, in get_next
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: () Authorization failed.
Code:
Message: Authorization failed.

But because seemingly all of the resources was available in Azure I thought nothing more of it.

How would we go about solving this authorization issue? For my situation at least, I've logged in using the "azd login", though with device_key because running in WSL for Windows (so issues with browser not opening).

Also, how can we fix this in the same project? Would "azd up", after fixing the authorization issue solve everything, or should the current project be deleted, and a new one created?

sindre-bouvet avatar Mar 15 '23 15:03 sindre-bouvet

Same problem

macalb69 avatar Mar 15 '23 16:03 macalb69

We've seen this problem when there's a permissions error during azd up, the script doesn't run at first and even if you fix it later, you've skipped the step to load the data. Curious if for each of you that mentioned this, you saw an error during the first run or if there's a different failure.

The script that creates the index, prepares the data, and loads it to blob storage and search, is in scripts/prepdocs.py, though running it through prepdocs.ps1 gets you all the parameters wired up based on your environment.

If the original run had failed while assigning permissions, you'll want to add the roles of blob contributor and search contributor to your user for storage/search services respectively before you run this. You can do that in the Azure portal, go to the blob/search service and use the "access control" section on the left.

Still identical problem as reported by @sindre-bouvet above, identical log. Steps I made sure to take:

  • I added the Owner roles in addition to the suggested (Storage Blob Data Contributor and Search Index Data Contributor) to both the Storage container and Search service respectively.
  • Made sure to run azd login beforehand and made sure that the correct tenant and account were used.
  • Included a --searchkey and --storagekey to the environment variables, but also hardcoded to the ps1 file, with and without single quote on the parameter.

This is not after a full deploy but just running prepdocs.ps1; after being certain that the provisioning was succesful. Totally unrelated I believe, also installed the Powershell modules Az.Accounts and AzureCli, totally unnecessary in my opinion but did just in case.

Have not tried to delete (undeploy everything) and deploy again but really don't have the time right now...

andres-swax avatar Mar 15 '23 18:03 andres-swax

Similar error, already added permissions and index is not being created:

Ensuring search index gptkbindex exists Traceback (most recent call last): File "C:\Demos\Azure-search-openai-demo\scripts\prepdocs.py", line 208, in create_search_index() File "C:\Demos\Azure-search-openai-demo\scripts\prepdocs.py", line 146, in create_search_index if args.index not in index_client.list_index_names(): File "C:\Users\miguelangels\Anaconda3\lib\site-packages\azure\core\paging.py", line 132, in next return next(self._page_iterator) File "C:\Users\miguelangels\Anaconda3\lib\site-packages\azure\core\paging.py", line 76, in next self._response = self._get_next(self.continuation_token) File "C:\Users\miguelangels\Anaconda3\lib\site-packages\azure\search\documents\indexes_generated\operations_indexes_operations.py", line 518, in get_next map_error(status_code=response.status_code, response=response, error_map=error_map) File "C:\Users\miguelangels\Anaconda3\lib\site-packages\azure\core\exceptions.py", line 109, in map_error raise error azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Unauthorized'

mikesoes avatar Mar 15 '23 23:03 mikesoes

Similar Error; Even after trying to execute script manually passing arguments

Followed Steps

azd logout

azd login

azd reset

executed python script with variable flags python ./prepdocs.py --storagename --servicesearch --storagekey --index --searchkey --container

It's working finally. Just make sure to run the python script with all the above variables and the relative path of /data/*

vrajroutu avatar Mar 16 '23 00:03 vrajroutu

We've seen this problem when there's a permissions error during azd up, the script doesn't run at first and even if you fix it later, you've skipped the step to load the data. Curious if for each of you that mentioned this, you saw an error during the first run or if there's a different failure. The script that creates the index, prepares the data, and loads it to blob storage and search, is in scripts/prepdocs.py, though running it through prepdocs.ps1 gets you all the parameters wired up based on your environment. If the original run had failed while assigning permissions, you'll want to add the roles of blob contributor and search contributor to your user for storage/search services respectively before you run this. You can do that in the Azure portal, go to the blob/search service and use the "access control" section on the left.

Still identical problem as reported by @sindre-bouvet above, identical log. Steps I made sure to take:

  • I added the Owner roles in addition to the suggested (Storage Blob Data Contributor and Search Index Data Contributor) to both the Storage container and Search service respectively.
  • Made sure to run azd login beforehand and made sure that the correct tenant and account were used.
  • Included a --searchkey and --storagekey to the environment variables, but also hardcoded to the ps1 file, with and without single quote on the parameter.

This is not after a full deploy but just running prepdocs.ps1; after being certain that the provisioning was succesful. Totally unrelated I believe, also installed the Powershell modules Az.Accounts and AzureCli, totally unnecessary in my opinion but did just in case.

Have not tried to delete (undeploy everything) and deploy again but really don't have the time right now...

I managed to follow @swarchitex steps and it looks like it works perfectly on my side.

Thanks, Mihai

mihaimunteanu28 avatar Mar 16 '23 07:03 mihaimunteanu28

@swarchitex , @mihaimunteanu28 , could you please provide me with an example of how you set the --searchkey and --storagekey to the environment variables, and hardcode it in the ps1 file?

sindre-bouvet avatar Mar 16 '23 12:03 sindre-bouvet

I got it working now thanks to @swarchitex example. I'll try to be a bit more specific if anybody runs into the same confusion as myself:

  • Follow the instructions in the README file
  • After running "azd up -t azure-search-openai-demo" and most probably get the same error messages mentioned earlier:
  • Access the resource group in the Azure portal.
  • Navigate to the Storage Account resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Storage Blob Data Contributor and Storage Blob Data Owner --> Members --> Add your user --> Create
  • Navigate to the Search Service resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Search Index Data Contributor and Search Index Data Reader (not sure if necessary)

Then for the keys:

  • searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key
  • storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key

These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey "<Primary Admin Key>" --storagekey "<Key>" after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script.

I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script.

After that is done, run "azd up", and everything should be working.

sindre-bouvet avatar Mar 16 '23 14:03 sindre-bouvet

@mihaimunteanu28 @sindre-bouvet Good morning and thank you for reviewing.

Following pablocastro's suggestion, yesterday I deleted the whole environment both in Azure and locally, going as far as deleting fork and local directories.

The index was created successfully indeed, without resorting to use the keys.

I do remember the very first time I performed the installation I did it via an online codespace straight from the main fork, and just by clicking the deploy button. Thats the one that requested credentials created the azure folder (with all the environment variables).

As far as I know, I never saw an error during first deployment but could very well happened. I suspect the issue was local (even though the environment variables' values were all correct).

So, index creation has been solved and thank you for reviewing my question.

andres-swax avatar Mar 16 '23 14:03 andres-swax

is there a screenshot of how to add the correct index indexer semantic etc. so to avoid running all this again? also if we add new pdfs do we need to run again the script?

gourgoul avatar Mar 17 '23 08:03 gourgoul

We've seen this problem when there's a permissions error during azd up, the script doesn't run at first and even if you fix it later, you've skipped the step to load the data. Curious if for each of you that mentioned this, you saw an error during the first run or if there's a different failure.

The script that creates the index, prepares the data, and loads it to blob storage and search, is in scripts/prepdocs.py, though running it through prepdocs.ps1 gets you all the parameters wired up based on your environment.

If the original run had failed while assigning permissions, you'll want to add the roles of blob contributor and search contributor to your user for storage/search services respectively before you run this. You can do that in the Azure portal, go to the blob/search service and use the "access control" section on the left.

Thanks for the reply. Yes. I added the permissions to both search and storage and it seems the problem was resolved.

2023-03-18_120657

timzhan avatar Mar 18 '23 04:03 timzhan

We've seen this problem when there's a permissions error during azd up, the script doesn't run at first and even if you fix it later, you've skipped the step to load the data. Curious if for each of you that mentioned this, you saw an error during the first run or if there's a different failure.

The script that creates the index, prepares the data, and loads it to blob storage and search, is in scripts/prepdocs.py, though running it through prepdocs.ps1 gets you all the parameters wired up based on your environment.

If the original run had failed while assigning permissions, you'll want to add the roles of blob contributor and search contributor to your user for storage/search services respectively before you run this. You can do that in the Azure portal, go to the blob/search service and use the "access control" section on the left.

In regards to predocs.py you mentioned. I just executed the azd up command. Noticed the following error while script was running:

Running "prepdocs.py"
Ensuring search index gptkbindex exists
Traceback (most recent call last):
  File "E:\dataDocuments\projects\scripts\prepdocs.py", line 299, in <module>
    create_search_index()
  File "E:\dataDocuments\projects\scripts\prepdocs.py", line 237, in create_search_index
    if args.index not in index_client.list_index_names():
  File "E:\dataDocuments\projects\scripts\.venv\lib\site-packages\azure\core\paging.py", line 132, in __next__
    return next(self._page_iterator)
  File "E:\dataDocuments\projects\scripts\.venv\lib\site-packages\azure\core\paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "E:\dataDocuments\projects\scripts\.venv\lib\site-packages\azure\search\documents\indexes\_generated\operations\_indexes_operations.py", line 520, in get_next
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: () Authorization failed.
Code:
Message: Authorization failed.

After script ran, I hit the endpoint and received the same error as above:

image

Looking at the Traceback message, there appears to be an issue with create_search_index() function definition (line 233-256 in prepdocs.py)

For some reason, prepdocs.py doesn't like the search_creds variable and can't create the SearchIndexClient(). I imported AzureCliCredential and changed to search_creds = AzureCliCredential(). Had to do the same thing with formrecognizer_creds = AzureCliCredential().

I also added the --storagekey and --searchkey to the prepocs.ps1.

Janky but worked. I couldn't get the other solutions to work (adding my service principal to the various RBAC roles listed above).

image

SeaDude avatar Mar 24 '23 05:03 SeaDude

I have seen this Authorization failed. for cases where the account used for azd login is the owner of the Subscription

For some reason, the expected roles are not created. Here's a workaround for this case: https://github.com/Azure-Samples/azure-search-openai-demo/issues/4#issuecomment-1483176774

Note: The workaround does not require creating storage-keys to set the auth with keys.

vhvb1989 avatar Mar 24 '23 17:03 vhvb1989

I followed the fixes about with --storagekey and --searchkey added to prepdocs.ps1, but get a little different error: Running "prepdocs.py" Ensuring search index gptkbindex exists Traceback (most recent call last): File "C:\Dev\AzSearchOAI-Demo\scripts\prepdocs.py", line 210, in create_search_index() File "C:\Dev\AzSearchOAI-Demo\scripts\prepdocs.py", line 148, in create_search_index if args.index not in index_client.list_index_names():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Dev\AzSearchOAI-Demo\scripts.venv\Lib\site-packages\azure\core\paging.py", line 132, in next return next(self._page_iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Dev\AzSearchOAI-Demo\scripts.venv\Lib\site-packages\azure\search\documents\indexes_generated\operations_indexes_operations.py", line 520, in get_next raise HttpResponseError(response=response, model=error) azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Forbidden'

Any idea? Thanks!

geniusdanny avatar Mar 29 '23 16:03 geniusdanny

I had the same issue.

Import AzureCliCredential to prepdocs.py then change all _creds variables to AzureCliCredential().

I think I also made myself Owner RBAC role to the Search and Storage resources.

Janky but worked for a POC.

Maybe it has to do with how I installed azd? I used the terminal in VS Code. Not sure if it installs as Admin or not. Or maybe it's something to do with the Bicep files?

Either way, I've stood the demo up 2x using this method.

SeaDude avatar Mar 30 '23 00:03 SeaDude

I got it working now thanks to @swarchitex example. I'll try to be a bit more specific if anybody runs into the same confusion as myself:

  • Follow the instructions in the README file
  • After running "azd up -t azure-search-openai-demo" and most probably get the same error messages mentioned earlier:
  • Access the resource group in the Azure portal.
  • Navigate to the Storage Account resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Storage Blob Data Contributor and Storage Blob Data Owner --> Members --> Add your user --> Create
  • Navigate to the Search Service resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Search Index Data Contributor and Search Index Data Reader (not sure if necessary)

Then for the keys:

  • searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key
  • storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key

These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey "" --storagekey "" after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script.

I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script.

After that is done, run "azd up", and everything should be working.

To make it works, I had to specify another argument in prepdocs.ps1 command --formrecognizerkey taking it the key from: Form Recognizer resource. Keys and Endpoint on the left side. -> Copy the KEY 1 key.

The line 35 in prepdocs.ps1 should look like this one: Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prepdocs.py $cwd/data/* --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --searchkey mySourCeSerViceSuperSecretKeyWithoutQuotes --storagekey myStoRageServIceSuperSecretKeyWithoutQuotes --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --formrecognizerkey myForMReCogniZerServiCeSuperSecretKeyWithoutQuotes --tenantid $env:AZURE_TENANT_ID -v" -Wait -NoNewWindow

HtH

zaffaste avatar Mar 31 '23 10:03 zaffaste

@zaffaste - thanks! Worked only suggestion is to put the keys into your .env file with the other settings for env so they are not accidentally exposed if you check your code into a repo.

kevintupper avatar Mar 31 '23 12:03 kevintupper

I got it working now thanks to @swarchitex example. I'll try to be a bit more specific if anybody runs into the same confusion as myself:

  • Follow the instructions in the README file
  • After running "azd up -t azure-search-openai-demo" and most probably get the same error messages mentioned earlier:
  • Access the resource group in the Azure portal.
  • Navigate to the Storage Account resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Storage Blob Data Contributor and Storage Blob Data Owner --> Members --> Add your user --> Create
  • Navigate to the Search Service resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Search Index Data Contributor and Search Index Data Reader (not sure if necessary)

Then for the keys:

  • searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key
  • storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key

These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey "" --storagekey "" after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script.

I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script.

After that is done, run "azd up", and everything should be working.

Step1.Get the keys:

searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key formrecognizerkey:Again nagivate to the Form Recognizer resource --> Keys and Endpoint (left hand side) --> Copy the Key

Step2.Edit the script: These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey {searchkey} --storagekey {storagekey} --formrecognizerkey {formrecognizerkey} after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script. like this: Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prepdocs.py $cwd/data/* --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --searchkey xxxxxxxxxxx --storagekey xxxxxxxxxxxx --formrecognizerkey xxxxxxxxxxx --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --tenantid $env:AZURE_TENANT_ID -v" -Wait -NoNewWindow

Step3.Run the script: I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script.

done.

zorc avatar Apr 05 '23 16:04 zorc

I got it working now thanks to @swarchitex example. I'll try to be a bit more specific if anybody runs into the same confusion as myself:

  • Follow the instructions in the README file
  • After running "azd up -t azure-search-openai-demo" and most probably get the same error messages mentioned earlier:
  • Access the resource group in the Azure portal.
  • Navigate to the Storage Account resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Storage Blob Data Contributor and Storage Blob Data Owner --> Members --> Add your user --> Create
  • Navigate to the Search Service resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Search Index Data Contributor and Search Index Data Reader (not sure if necessary)

Then for the keys:

  • searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key
  • storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key

These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey "" --storagekey "" after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script. I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script. After that is done, run "azd up", and everything should be working.

Step1.Get the keys:

searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key formrecognizerkey:Again nagivate to the Form Recognizer resource --> Keys and Endpoint (left hand side) --> Copy the Key

Step2.Edit the script: These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey {searchkey} --storagekey {storagekey} --formrecognizerkey {formrecognizerkey} after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script. like this: Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prepdocs.py $cwd/data/* --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --searchkey xxxxxxxxxxx --storagekey xxxxxxxxxxxx --formrecognizerkey xxxxxxxxxxx --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --tenantid $env:AZURE_TENANT_ID -v" -Wait -NoNewWindow

Step3.Run the script: I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script.

done. I got this error message just before deploying service backend >>

Running "prepdocs.py" ModuleNotFoundError: No module named 'cffi_backend' thread '' panicked at 'Python API call failed', C:\Users\runneradmin.cargo\registry\src\index.crates.io-6f17d22bba15001f\pyo3-0.15.2\src\err\mod.rs:582:5 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace Traceback (most recent call last): File "E:\Project\GPT-Test\azure-search\azure-search\scripts\prepdocs.py", line 9, in from azure.identity import AzureDeveloperCliCredential File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_init.py", line 10, in from .credentials import ( File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_credentials_init.py", line 5, in from .authorization_code import AuthorizationCodeCredential File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_credentials\authorization_code.py", line 9, in from ..internal.aad_client import AadClient File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_internal_init.py", line 5, in from .aad_client import AadClient File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_internal\aad_client.py", line 11, in from .aad_client_base import AadClientBase File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_internal\aad_client_base.py", line 23, in from .aadclient_certificate import AadClientCertificate File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\azure\identity_internal\aadclient_certificate.py", line 7, in from cryptography import x509 File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\cryptography\x509_init_.py", line 6, in from cryptography.x509 import certificate_transparency File "E:\Project\GPT-Test\azure-search\azure-search\scripts.venv\Lib\site-packages\cryptography\x509\certificate_transparency.py", line 10, in from cryptography.hazmat.bindings._rust import x509 as rust_x509 pyo3_runtime.PanicException: Python API call failed

gbo3000 avatar Apr 06 '23 11:04 gbo3000

We've seen this problem when there's a permissions error during azd up, the script doesn't run at first and even if you fix it later, you've skipped the step to load the data. Curious if for each of you that mentioned this, you saw an error during the first run or if there's a different failure.

The script that creates the index, prepares the data, and loads it to blob storage and search, is in scripts/prepdocs.py, though running it through prepdocs.ps1 gets you all the parameters wired up based on your environment.

If the original run had failed while assigning permissions, you'll want to add the roles of blob contributor and search contributor to your user for storage/search services respectively before you run this. You can do that in the Azure portal, go to the blob/search service and use the "access control" section on the left.

works for me. Thanks

ghost avatar Apr 07 '23 12:04 ghost

It seems to work now with fresh installation from github. Last time I installed by downloading the zip file which did not work.

gbo3000 avatar Apr 08 '23 13:04 gbo3000

As mentioned by @vhvb1989, I was getting Authorization failed. error using the account that is owner of the subscription. Instead of creating a new user as referenced in the workaround, I just used the keys instead. My Start-Process command now looks as follows:

$storagekey = "YOUR_STORAGE_ACCOUNT_KEY";
$searchkey = "YOUR_COGNITIVE_SEARCH_KEY";
$formkey = "YOUR_FORM_RECOGNIZER_KEY";
Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prepdocs.py `"$cwd/data/*`" --storageaccount $env:AZURE_STORAGE_ACCOUNT  --container $env:AZURE_STORAGE_CONTAINER --storagekey '`"$storagekey`"' --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --searchkey $searchkey --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --formrecognizerkey $formkey --tenantid $env:AZURE_TENANT_ID -v" -Wait -NoNewWindow

itsamk avatar Apr 08 '23 23:04 itsamk

I have seen this Authorization failed. for cases where the account used for azd login is the owner of the Subscription

For some reason, the expected roles are not created. Here's a workaround for this case: #4 (comment)

Note: The workaround does not require creating storage-keys to set the auth with keys.

@vhvb1989 - thank you. What is the guidance when the customer is not Owner of the subscription but Contributor? Do we have instructions for deploying manually to bypass the authorization issues?

chulahlou avatar Apr 12 '23 20:04 chulahlou

I followed the fixes about with --storagekey and --searchkey added to prepdocs.ps1, but get a little different error: Running "prepdocs.py" Ensuring search index gptkbindex exists Traceback (most recent call last): File "C:\Dev\AzSearchOAI-Demo\scripts\prepdocs.py", line 210, in create_search_index() File "C:\Dev\AzSearchOAI-Demo\scripts\prepdocs.py", line 148, in create_search_index if args.index not in index_client.list_index_names(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Dev\AzSearchOAI-Demo\scripts.venv\Lib\site-packages\azure\core\paging.py", line 132, in next return next(self._page_iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Dev\AzSearchOAI-Demo\scripts.venv\Lib\site-packages\azure\search\documents\indexes_generated\operations_indexes_operations.py", line 520, in get_next raise HttpResponseError(response=response, model=error) azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Forbidden'

Any idea? Thanks!

@geniusdanny I got the same error. Did you resolve this and can you share your fix?

chulahlou avatar Apr 19 '23 15:04 chulahlou

I got it working now thanks to @swarchitex example. I'll try to be a bit more specific if anybody runs into the same confusion as myself:

  • Follow the instructions in the README file
  • After running "azd up -t azure-search-openai-demo" and most probably get the same error messages mentioned earlier:
  • Access the resource group in the Azure portal.
  • Navigate to the Storage Account resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Storage Blob Data Contributor and Storage Blob Data Owner --> Members --> Add your user --> Create
  • Navigate to the Search Service resource. Access Control (IAM) on the left side. +Add --> Add role assignment --> Role --> Search Index Data Contributor and Search Index Data Reader (not sure if necessary)

Then for the keys:

  • searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key
  • storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key

These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey "" --storagekey "" after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script. I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script. After that is done, run "azd up", and everything should be working.

Step1.Get the keys:

searchkey: Again nagivate to the Search Service resource --> Keys (left hand side) --> Copy the Primary Admin Key storagekey: Again nagivate to the Storage Account resource --> Access Keys (left hand side) --> Copy the Key formrecognizerkey:Again nagivate to the Form Recognizer resource --> Keys and Endpoint (left hand side) --> Copy the Key

Step2.Edit the script: These keys should be added as arguments in the /scripts/prepdocs.ps1 file. Open it in nano, vim or VS Code and add: --searchkey {searchkey} --storagekey {storagekey} --formrecognizerkey {formrecognizerkey} after the "--index $env:AZURE_SEARCH_INDEX", in the last line of the script. like this: Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prepdocs.py $cwd/data/* --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --searchkey xxxxxxxxxxx --storagekey xxxxxxxxxxxx --formrecognizerkey xxxxxxxxxxx --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --tenantid $env:AZURE_TENANT_ID -v" -Wait -NoNewWindow

Step3.Run the script: I run everything in git bash, but for this part I switched to powershell: run "pwsh" in git bash. Enter "./scripts/predocs.ps1" to run the script.

done.

I did that, but now I get another error:

prepdocs.py: error: argument --storageaccount: expected one argument

marceloricciardi avatar Jun 19 '23 22:06 marceloricciardi

Ok so this is an old thread but I had the same issue. What helped for me was to comment out the --searchkey and --storagekey lines in prepdocs.ps1. The missing index was finally created.

DuboisABB avatar Jul 03 '24 21:07 DuboisABB