ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard
verified publisher icon hpcaitech

LAMB is not suited for tensor parallel

Open FrankLeeeee opened this issue 4 years ago • 3 comments

The current LAMB optimizer implementation does not support tensor parallel as it needs to compute norm of the whole matrix. It is not compatible with tensor parallel as the tensor is split.

FrankLeeeee avatar Nov 12 '21 07:11 FrankLeeeee

This issue is stale because it has been open for 14 days with no activity.

github-actions[bot] avatar Jan 20 '22 00:01 github-actions[bot]

@ver217 , i think we got to fix this bug these two days

FrankLeeeee avatar Jan 20 '22 05:01 FrankLeeeee

This issue is stale because it has been open for 14 days with no activity.

github-actions[bot] avatar Feb 04 '22 00:02 github-actions[bot]

We have updated a lot. This issue was closed due to inactivity. Thanks.

binmakeswell avatar Apr 13 '23 03:04 binmakeswell