unstract
unstract copied to clipboard
feat: Avoid storing file history for API based wfs
What
- Set and passed
use_file_historyfor WF execution in case of API based source - Included changes for v2 migration as well
- Allows File / DB based runs to still cache the file history (can be reviewed later), API based runs are not stored
- Set API deployment timeout's min value and default to 0 instead of -1
Why
- To avoid caching file history for API based WF runs
Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
- No, added params with sane defaults
Related Issues or PRs
- #745
Dependencies Versions
Notes on Testing
- Attempted running an API based WF locally
Screenshots
Checklist
I have read and understood the Contribution Guidelines.
Code duplication is due to the v2 migration changes also being present in this PR
LGTM. Just to confirm, we're only saving the result in the database when the API is called with specific parameters, right?
@muhammad-ali-e in case of workflow execution, we'll never save the results from API related source / destination. Only for ETL / task (file/ DB related source / destination) will make use of the file history table to mark processed files (not results)
| filepath | function | $$\textcolor{#23d18b}{\tt{passed}}$$ | SUBTOTAL |
|---|---|---|---|
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_logs}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_cleanup}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_client\_init}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_get\_image}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ | $$\textcolor{#23d18b}{\tt{test\_run\_container}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ | $$\textcolor{#23d18b}{\tt{1}}$$ |
| $$\textcolor{#23d18b}{\tt{TOTAL}}$$ | $$\textcolor{#23d18b}{\tt{9}}$$ | $$\textcolor{#23d18b}{\tt{9}}$$ |
