chat-with-your-data-solution-accelerator icon indicating copy to clipboard operation
chat-with-your-data-solution-accelerator copied to clipboard

Remove duplicate upload_file logic from Admin Ingest page.

Open superhindupur opened this issue 9 months ago • 2 comments

Purpose

While looking at setting up tests for the admin app pages under #736, I realized that we can refactor the pages to have a bare minimum of business logic in them, with the logic coming from the backend utilities package.

In this PR, I've removed the dupliocate upload_file to blob storage functionality and used the functionality from the AzureBlobStorageHelper.py file instead. I've also added a missing unit test to cover this functionality, achieving the overall goal of testing the admin app pages better.

Required by #736

Does this introduce a breaking change?

[ ] Yes
[x ] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[x ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

Deploy the admin app and test whether the following are working:

  • Ingestion of files
  • Ingestion of URLs
  • I've tested the above with and without integrated vectorization, and for txt and pdf files.

Other Information

superhindupur avatar Apr 29 '24 14:04 superhindupur

Coverage

Coverage Report •
FileStmtsMissCoverMissing
code/backend/batch
   BatchPushResults.py31196%53
   BatchStartProcessing.py190100% 
code/backend/batch/utilities/common
   SourceDocument.py59493%33, 37, 41, 130
code/backend/batch/utilities/helpers
   AzureBlobStorageClient.py803062%22–24, 32, 52, 55–56, 61, 65, 90–91, 93, 97, 153, 156, 175, 178, 180, 188–192, 215, 219–223, 225
   ConfigHelper.py1120100% 
code/backend/pages
   01_Ingest_Data.py88880%1–12, 14–16, 18, 24, 31, 34–35, 38–47, 51–53, 56–59, 61, 64, 67–73, 76–79, 82–89, 92–94, 96, 99–102, 105, 110–112, 114–115, 117–118, 121–122, 126, 129–130, 136–139, 146–147, 152, 158–159
TOTAL213180062% 

Tests Skipped Failures Errors Time
143 0 :zzz: 0 :x: 0 :fire: 8.537s :stopwatch:

github-actions[bot] avatar Apr 29 '24 14:04 github-actions[bot]

Was this tested with Integrated Vectorization enabled for URLs?

komalg1 avatar Apr 29 '24 15:04 komalg1