vectorflow Remove duplicate code

Methods such as update_batch_and_job_status are duplicated across the code to prevent different modules from being dependent on each other.

We need to 1) find and document all the repetitive code 2) move it into a shared or utils location 3) make sure the duplication is removed from the existing code

Oct 27 '23 04:10 dgarnitz

Hi Dgarnitz,

I just wanted to confirm before i get ahead of my self, the update_batch_and_job_status is located in app.py, worker.py, and vbd_upload_worker.py

Is the goal here to move these duplicate functions to the shared/utils.py file, and import the necessary dependencies?

I have actioned according to the above in PR #83

Cheers, Aleks

Oct 27 '23 11:10 Aleksandir

Hey thanks for your interest in this task. Yes, the goal is to remove that duplicate method from those locations and move it into the shared location. shared/utils.py is good for now. We can refactor that file once it grows a bit more.

As of today, we merged a new big PR and there is now an extractor.py where this method also lives.

Oct 28 '23 01:10 dgarnitz

@Aleksandir I see you opened a PR for it - great work! Really appreciate it!!

Make sure you rebase the new code that just merged. Its quite a big merge, you will need to tear down the docker contains, and re-run all the docker compose stuff. Also make sure to pull the minio docker image.

Once you do that, please retest to confirm stuff still works and tag me on the PR.

Oct 28 '23 01:10 dgarnitz