Jiarui Fang（方佳瑞）
Jiarui Fang（方佳瑞）
Yes, it is on the roadmap of the v0.2.0 version. In that version, we will have a new interface for ZeRO.
The DeepSpeed benchmark script https://github.com/feifeibear/DeepSpeedZeRO3Benchmark The PatrickStar https://github.com/Tencent/PatrickStar/blob/master/examples/run_transformers.sh The benchmarking is very easy. ``` export SUFFIX="colossal_compare" env GPU_NUM=8 MODEL_TYPE="GPT" MODEL_NAME=GPT3_10B BS=2 CPU_EBD=0 AMM=1 MSC=1 CACHE=1 SP=0 CS=288 HYB=1 TILING=0 ACT_OFFLOAD=0...
I have uploaded the logs of DeepSpeed and PatirckStar to Baidu WangPan... Note that for DeepSpeed, the SamplesPerSec is not equal to 'Throughput'. You have to calculate it by batch/elapse....
@1SAA did you fix the issue?
Could please get an example to illustrate the not supported cases! @1SAA
You can add a setter in class Linear. Make sure your setting is valid in terms if the tensor shape. colossalai/nn/layer/colossalai_layer/linear.py ``` @property def bias(self, value): self.layer.bias = value ```
The bias shape is different. Users do not know how did you split the bias?
> > > > Should I create 3 new issues for these and start working on them? Sure, you can post the issues separately and descript your solution to them....
@xbasly Did you solve you problem? Can you guys provide a reproducible script for us?
Hello, @SMesForoush , Thanks for your suggestion. I am looking for your PR to the main branch.