yinwei

`numa_set_interleave_mask` problem

2

Hi,xiaowei, I'm trying to run pagerank on scale16_s.bin, which from [here](https://github.com/bibrakc/graph-datasets/tree/master/data) from issue #6 , but I get a segment fault, it looks like following. I think the numa_set_interleave_mask() function...

I just can not run the code with correct data format.

SP support overlap

2

### PR types Others ### PR changes Others ### Description Others

add delay_scale_loss for stage1

4

### PR types Others ### PR changes Others ### Description Others

add run_hybrid_parallel.sh

2

### PR types Others ### PR changes Others ### Description 这个PR是为了我实验跑测试方便

MP2-PP2 hack shared layer to non-sharded layer to Step Alignment

2

### PR types Others ### PR changes Others ### Description MP2-PP2与MP2无法做到逐位对齐，原因是MP使用了共享Embedding层，共享层需要使用梯度累加，MP的梯度累加是使用加法来实现的，而PP由于跨卡，因此共享层的梯度同步是使用通信来完成的，具体的操作是Allreduce。这个PR将共享层hack成非共享层，来实验MP2-PP2与MP2的逐位对齐，目的是为了验证Allreduce和加法实现的梯度累加本身就存在diff，因此无法做到完全逐位对齐。一定要注意要注释掉self.tie_weights()这一行代码，这一行手动绑定了两个层作为共享层

sharding v1 overlap

2

### PR Category Others ### PR Types Others ### Description card-13678

Stage1 V2 allgather overlap forward

3

### PR Category Others ### PR Types Others ### Description Others

V1 overlap without pybind

2

### PR Category Others ### PR Types Others ### Description Others

Support Sharding Overlap

3

### PR types Performance optimization ### PR changes Models ### Description 1.支持sharding overlap