data
data copied to clipboard
[DataLoader2] Skip wrapping if serialization wrapper attached & deepcopy DataPipe in load_state_dict
Per title
- We don't need to attach serialization wrapper if the last DataPipe has been
_DataPipeSerializationWrapper
- When we
load_state_dict
, we still need a copy ofself.datapipe
as it has been done in__init__
function. We should either do deepcoy toself._datapipe_before_reading_service_adapt
or skip deepcopy toself._datapipe_before_reading_service_adapt
at those two places.
@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
LGTM! Question: do we expect any case where the
datapipe
being passed in already has a wrapper?
It's more like a future proof. (Technically, if users manually deserialize
the DataPipe state, the returned DataPipe is wrapped by the wrapper.