data icon indicating copy to clipboard operation
data copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

[DataLoader] Add __len__ to DataLoader2

Open NivekT opened this issue 2 years ago • 5 comments

Stack from ghstack:

  • -> #728
  • #746

Adding __len__ to DataLoader2.

See inline comments. We should discuss the details and if this makes sense.

Fixes #549

Differential Revision: D38999743

NivekT avatar Aug 11 '22 22:08 NivekT

Offline discussion:

  • Initialize when __len__ is called. First, check if RS is initialized, if not, initialize RS, then call self.ReadingService.get_datapipe_length(delegating to RS).
  • Need to change interface of RS to support these operations.

NivekT avatar Aug 12 '22 19:08 NivekT

Please re-request review when discussed changes applied.

VitalyFedyunin avatar Aug 15 '22 15:08 VitalyFedyunin

@nivekt has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

NivekT avatar Aug 24 '22 22:08 NivekT

@nivekt has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

NivekT avatar Aug 25 '22 17:08 NivekT

@nivekt has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

NivekT avatar Aug 25 '22 20:08 NivekT

@nivekt has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

NivekT avatar Oct 18 '22 22:10 NivekT

@nivekt has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

NivekT avatar Oct 18 '22 23:10 NivekT

@NivekT Is there any update? I think it's very helpful for using Trainers like pytorch-lightning or huggingface Transformers.

SeanLiaoy avatar Apr 21 '23 18:04 SeanLiaoy

Hi @NivekT!

Thank you for your pull request.

We require contributors to sign our Contributor License Agreement, and yours needs attention.

You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

facebook-github-bot avatar Jun 09 '23 07:06 facebook-github-bot

Closing. Feel free to re-open if someone else would like to work on this.

NivekT avatar Jun 11 '23 04:06 NivekT