ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

Published 20 hours ago • verified publisher icon hpcaitech

[autoparallel] GPT2MLP runtime test

Open YuliangLiu0306 opened this issue 2 years ago • 0 comments

What does this PR do

  1. This PR construct a test to check the result of MLP with the auto parallel strategy equals to the original result or not.
  2. Update runtime preparation passes: a. add strategies_constructor as a optional argument to get the no strategy nodes list. b. Change the get_attr parameter shard workflow to fix a bug which makes parameters retain None as grad.

Note: In future PR, we need to fix the rng state issue for modules with randomness, such as nn.Dropout.

YuliangLiu0306 avatar Dec 09 '22 11:12 YuliangLiu0306