data icon indicating copy to clipboard operation
data copied to clipboard
verified publisher icon pytorch

Improve AsyncIODataPipe

Open ejguan opened this issue 2 years ago • 0 comments

🚀 The feature

A few improvements can be achieved for AsyncIODataPipe:

  • Make it working properly with nested_async
  • Constantly fetching rather than fetching batch per batch with closing(nested_async.prefetch_sequence(datapipe, self.max_prefetch)) as seq:
  • Align the API with threadpool_map without the need to specify batch.

Motivation, pitch

Improve performance, reliability, functionality.

Alternatives

No response

Additional context

No response

ejguan avatar May 10 '23 16:05 ejguan