azure-search-openai-demo
azure-search-openai-demo copied to clipboard
Azure-Samples
Cost estimates
This issue is for a: (mark with an x)
- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [X ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
Is there anyone who could estimate what this would cost to run in production? Before we dive into the technical aspects of getting this to work, we wanted to know if it is even financially feasible for what we want to do. I know there will be a large range but maybe a site with 100 unique documents and 100 queries per day and another with up to 10,000 documents and 10,000 queries per day.
With the deployment as-is, the Azure cognitive search cost seems to dwarf the OpenAI service cost by 10x. It doesn't help that the example as-is sends queries to cognitive search for every query.
Wild guess based on my usage so far but a standard Cognitive search is about 300$CAD/month and the OpenAI service is an extra 10% of that. Pretty confident it should cover your 100 unique docs and 100 queries per day.
https://azure.microsoft.com/en-ca/pricing/details/search/#pricing
...says that the standard plan covers up to 250K semantic queries a month and 25GB worth documents so I'd guess it could also cover your upper case?
With the deployment as-is, the Azure cognitive search cost seems to dwarf the OpenAI service cost by 10x. It doesn't help that the example as-is sends queries to cognitive search for every query.
Wild guess based on my usage so far but a standard Cognitive search is about 300$CAD/month and the OpenAI service is an extra 10% of that. Pretty confident it should cover your 100 unique docs and 100 queries per day.
https://azure.microsoft.com/en-ca/pricing/details/search/#pricing
...says that the standard plan covers up to 250K semantic queries a month and 25GB worth documents so I'd guess it could also cover your upper case?
Thanks for that. Does this include the server costs too?
If you use Azure deployment, it uses the application service for the app deployment. The cost of that is 20x times less than the search... so it's not the main driver of cost. again, I just deployed a trivial example and i'm reading it up based on the current computed costs.
One tool to help watch cost of a deployed environment: https://github.com/mivano/azure-cost-cli
I can give you an anecdotal answer here. I deployed this project as-is. I had a "Visual Studio Professional" subscription with $50 in free credits for Azure services. I deployed the app with close to 200 sample PDFs, totaling about 8700 pages. Doing a "chat with your documents" setup using OpenAI's API and its embedding model always resulted in costs below $1. Running this demo on Azure (configured as-is), racked up about $200(!). The main cost came from the ingest/embedding process because it charges by the page. Only myself and one other developer briefly played with the app before shutting it down the next morning when I saw the costs. Even after shutting down the web server it kept racking up costs for the other (idle) services. I had to delete the whole resource group to stop it from generating more costs. Azure support promised to refund me (which I appreciate). The refund has been reflected in an invoice, but it hasn't showed up on my credit card yet. We will see.
Here are some screenshots:
As you can see, the bulk comes from Azure Form Recognizer, which is used in the ingest/embedding process. Surely this project can somehow be configured to use cheaper resources. I imagine that it would also be possible to not use Azure Form Recognizer for the ingest process and save a considerable amount. But if you try to deploy and run this demo as-is with more than a couple of PDFs: better stock up on some Top Ramen, because it will cost you.
Hi all, I'm working on putting more details in the cost estimation in this PR: https://github.com/Azure-Samples/azure-search-openai-demo/pull/537 It includes a link to the Azure pricing estimation calculator. That link puts ACS as the more costly part, but it will depend on how many documents you analyze and how many users issue OpenAI queries. The README also contains tips for reducing costs.
The cost estimation is now in the README, so I'm closing this issue. Comment or open a new issue if more details are needed.
I can give you an anecdotal answer here. I deployed this project as-is. I had a "Visual Studio Professional" subscription with $50 in free credits for Azure services. I deployed the app with close to 200 sample PDFs, totaling about 8700 pages. Doing a "chat with your documents" setup using OpenAI's API and its embedding model always resulted in costs below $1. Running this demo on Azure (configured as-is), racked up about $200(!). The main cost came from the ingest/embedding process because it charges by the page. Only myself and one other developer briefly played with the app before shutting it down the next morning when I saw the costs. Even after shutting down the web server it kept racking up costs for the other (idle) services. I had to delete the whole resource group to stop it from generating more costs. Azure support promised to refund me (which I appreciate). The refund has been reflected in an invoice, but it hasn't showed up on my credit card yet. We will see.
Here are some screenshots:
As you can see, the bulk comes from Azure Form Recognizer, which is used in the ingest/embedding process. Surely this project can somehow be configured to use cheaper resources. I imagine that it would also be possible to not use Azure Form Recognizer for the ingest process and save a considerable amount. But if you try to deploy and run this demo as-is with more than a couple of PDFs: better stock up on some Top Ramen, because it will cost you.
I'm also quite shocked with the cost of the "Azure Applied AI Services" (which I assume is the form recognizer as well). I have simple pdfs too, with nothing but plain text. Help?
If the form recognizer costs are too high for your budget, there's also a local PDF parser parser you can try. To use it, modify the final command in prepdocs.sh or prepdocs.ps1 to pass --localpdfparser to the prepdocs.py script.
Also, make sure you aren't repeatedly calling prepdocs.py for the same docs. Once you've successfully indexed them, you should disable the prepdocs script either deleting the postprovision hooks from azure.yaml.
Great documentation!
I still have a question in terms of the pricing, especially for the Azure openai service.
When I check metrics and pull the consumption chart, I notice there are prompt vs inference, and count vs sum.
When I ask 1 question, I get 3 prompt and 3 inference. It was quite confusing as I believe from the chat approach there are only 2 ChatCompletion calls. I used hybrid so I guess the embedding counts as 1 for both prompt and inference?
Also the sum of tokens in the metrics is kind of weird, the prompt and the inference are very close. Since there are token limit for output, it just doesn't make sense. Perhaps my understanding for the metrics is incorrect...
Where do you get that chart from? Is that in Azure OpenAI Studio? I'll need to find whatever team is responsible and ask them. Thanks!
Where do you get that chart from? Is that in Azure OpenAI Studio? I'll need to find whatever team is responsible and ask them. Thanks!
It is in the Azure OpenAI dashboard, in the Azure Portal before you click into the studio. The studio itself doesn't have any management tool as I observe.
Thank you for following up! I believe the cost is calculated correct based on the consumption, just confusing with the metric reports.