best practice for `snapshot_every_n_steps`

[ShoufaChen]

Hello,

Thank you for your awesome implementation of StatefulDataloader.

I have a question about snapshot_every_n_steps. It seems there is not much detailed explanation about this argument.

• Will frequent snapshots (i.e., snapshot_every_n_steps=1) cause a data loading burden?
• What is the best practice for setting this value? Is it related to checkpointing frequency?

cc @andrewkho

[andrewkho]

Hi @ShoufaChen thanks for the issue, we should update the documentation to explain this better.

To answer your questions: it depends mainly on the size and composition of your state. If you're storing eg an int representing index or file-offset, then it shouldn't be an issue. If your state is very large, involving eg buffers of data for shuffling, then the overhead of creating a checkpoint and passing it through multiprocessing queue may slow down training, this variable lets you decrease the frequency of checkpointing. If eg you know you're checkpointing every 1000 steps, you can set this value to 1000.

[yzhangcs]

@andrewkho Hi, wondering if this arg will affects the final state loading? For example, if snapshot_every_n_steps=4, what will happen if I want to load_state_dict or save state_dict at the 5th step?

Does that me I should always need to make the steps for ckpting divisible by snapshot_every_n_steps?

Thank you.

[andrewkho]

@yzhangcs you can still request a checkpoint/state_dict at any time, the dataloader will load the last snapshot and "fast forward" the required steps to get to the correct point. In your example, dataloader would load the state from step 4, then throw away one batch before continuing at step 5. Hope this helps!

@divyanshk lets double check if our docs can be improved

[yzhangcs]

@andrewkho thank you for the nice response! That is a very clever tradeoff

[yzhangcs]

@andrewkho It would be better if adding explanations of “fast forward” into docs for anyone curious about it