DALI icon indicating copy to clipboard operation
DALI copied to clipboard

Add warnings against resetting pipeline before end of epoch and test parallel ES with fw iterator

Open stiepan opened this issue 2 years ago • 4 comments

Other (e.g. Documentation, Tests, Configuration)

Category:

Description:

  • If someone uses fw iterators and passes size that is incorrect (i.e. different than what follows from external source raising StopIteration or one of the pipelines raises while the other do not) it may result in premature resetting of the pipelines, which further can lead to different issues: from empty epochs to corrupted data.

Adding the warnings in hope that it will be helpful for users or us handling gh issues to have those warnings in logs. Adding the PES tests to iterators, as the existing tests focused on a pipeline's run API which is not the one used by fw iterators.

Additional information:

Affected modules and functionalities:

  • Python pipeline
  • Python fw iterator (plugins)

Key points relevant for the review:

Tests:

  • [ ] Existing tests apply
  • [x] New tests added
    • [x] Python tests
    • [ ] GTests
    • [ ] Benchmark
    • [ ] Other
  • [ ] N/A

Checklist

Documentation

  • [x] Existing documentation applies
  • [ ] Documentation updated
    • [ ] Docstring
    • [ ] Doxygen
    • [ ] RST
    • [ ] Jupyter
    • [ ] Other
  • [ ] N/A

DALI team only

Requirements

  • [ ] Implements new requirements
  • [ ] Affects existing requirements
  • [x] N/A

REQ IDs: N/A

JIRA TASK: DALI-2864

stiepan avatar Jun 29 '22 09:06 stiepan

!build

stiepan avatar Jun 29 '22 10:06 stiepan

CI MESSAGE: [5209598]: BUILD STARTED

dali-automaton avatar Jun 29 '22 11:06 dali-automaton

CI MESSAGE: [5209598]: BUILD PASSED

dali-automaton avatar Jun 29 '22 12:06 dali-automaton

CI MESSAGE: [8224548]: BUILD FAILED

dali-automaton avatar May 09 '23 06:05 dali-automaton