azure-search-openai-demo
azure-search-openai-demo copied to clipboard
Azure-Samples
--removeall not working for datalake ingested data
How do we remove data when local data folder is not used?
prepdocs.ps1 args list for reference:
$argumentList = "./scripts/prepdocs.py --subscriptionid $env:AZURE_SUBSCRIPTION_ID " + "--datalakepath $env:AZURE_ADLS_GEN2_FILESYSTEM_PATH " +
" --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER --storageresourcegroup $env:AZURE_STORAGE_RESOURCE_GROUP " + "--searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX " +
"$searchAnalyzerNameArg $searchSecretNameArg " + "--openaihost"$env:OPENAI_HOST" --openaimodelname "$env:AZURE_OPENAI_EMB_MODEL_NAME" " +
"--openaiservice "$env:AZURE_OPENAI_SERVICE" --openaideployment "$env:AZURE_OPENAI_EMB_DEPLOYMENT" " + "--openaikey"$env:OPENAI_API_KEY" --openaiorg "$env:OPENAI_ORGANIZATION" " +
"--documentintelligenceservice $env:AZURE_DOCUMENTINTELLIGENCE_SERVICE " + "$searchImagesArg $visionEndpointArg $visionKeyArg $visionSecretNameArg " +
"$adlsGen2StorageAccountArg $adlsGen2FilesystemArg $adlsGen2FilesystemPathArg " + "$tenantArg $aclArg " +
"$disableVectorsArg $localPdfParserArg $localHtmlParserArg " + "$keyVaultName " +
"$integratedVectorizationArg --removeall"
Do you get an error? Or it just doesn't remove? Could you put some print() statements in remove_blob to see how far it's getting?
running ./scripts/prepdocs.sh --removeall from today's azd init -t azure-search-openai-demo version, deployed, does not remove all from the index. @pamelafox. I think there's a bug in the py fie as it accumulates the args passed.
my env: macos.current, py 3.9, azd version 1.9.2 (commit c58b02f71710960aba28f81f3698e64cfdda9f96)
@evogelpohl That isn't actually expected to work, it doesnt pass on the args. You have to actually modify the prepdocs.sh file itself and add it to the command at the bottom. However I do think it would be better if you could do what you said. Pull requests welcome, if you have the time!