data icon indicating copy to clipboard operation
data copied to clipboard

[DataLoader2] Skip wrapping if serialization wrapper attached & deepcopy DataPipe in load_state_dict

Open ejguan opened this issue 1 year ago • 1 comments

Per title

  • We don't need to attach serialization wrapper if the last DataPipe has been _DataPipeSerializationWrapper
  • When we load_state_dict, we still need a copy of self.datapipe as it has been done in __init__ function. We should either do deepcoy to self._datapipe_before_reading_service_adapt or skip deepcopy to self._datapipe_before_reading_service_adapt at those two places.

ejguan avatar Oct 14 '22 21:10 ejguan

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot avatar Oct 14 '22 21:10 facebook-github-bot

LGTM! Question: do we expect any case where the datapipe being passed in already has a wrapper?

It's more like a future proof. (Technically, if users manually deserialize the DataPipe state, the returned DataPipe is wrapped by the wrapper.

ejguan avatar Oct 17 '22 19:10 ejguan