DALI
DALI copied to clipboard
Add warnings against resetting pipeline before end of epoch and test parallel ES with fw iterator
Other (e.g. Documentation, Tests, Configuration)
Category:
Description:
- If someone uses fw iterators and passes
size
that is incorrect (i.e. different than what follows from external source raising StopIteration or one of the pipelines raises while the other do not) it may result in premature resetting of the pipelines, which further can lead to different issues: from empty epochs to corrupted data.
Adding the warnings in hope that it will be helpful for users or us handling gh issues to have those warnings in logs. Adding the PES tests to iterators, as the existing tests focused on a pipeline's run API which is not the one used by fw iterators.
Additional information:
Affected modules and functionalities:
- Python pipeline
- Python fw iterator (plugins)
Key points relevant for the review:
Tests:
- [ ] Existing tests apply
- [x] New tests added
- [x] Python tests
- [ ] GTests
- [ ] Benchmark
- [ ] Other
- [ ] N/A
Checklist
Documentation
- [x] Existing documentation applies
- [ ] Documentation updated
- [ ] Docstring
- [ ] Doxygen
- [ ] RST
- [ ] Jupyter
- [ ] Other
- [ ] N/A
DALI team only
Requirements
- [ ] Implements new requirements
- [ ] Affects existing requirements
- [x] N/A
REQ IDs: N/A
JIRA TASK: DALI-2864
!build
CI MESSAGE: [5209598]: BUILD STARTED
CI MESSAGE: [5209598]: BUILD PASSED
CI MESSAGE: [8224548]: BUILD FAILED