Hao Zhang

icon indicating a website link https://cseweb.ucsd.edu/~haozhang/

icon location Asst. prof. at UCSD working on LLMs.

Results 174 comments of Hao Zhang

Integration

> I have addressed the comments. Do you have any further suggestions? Will review tomorrow and see whether more changes are needed.

Some questions about AutoSync

@wjykl22 Thanks for your interest! We're currently working on cleaning up the dataset and will make it available in the Neurips week (i.e., next week). For the AutoSync code, the...

Add PowerSGD compressor for AllReduce based strategy

@ZeyaWang : are you working on this one now?

Add PowerSGDCompressor #27

@Ezra-H CI still fails, please check & fix

[FEATURE] OPT-175B service authentication and new priority queue

@aurickq

RFC for alpa api authentication

@rjiang-ptm thanks! @merrymercy @zhuohan123 please comment

[RUNTIME] TPU Backend Support

Is TPU+pipeshard still in our scope? @merrymercy @ZYHowell

[FEATURE] Plan to improve the overall cross mesh resharding speed

See this paper by Alpa team: https://arxiv.org/pdf/2211.05322.pdf Related PR: #773

Running OPT 175B with different hardware configurations

@sachit-menon @xhluca Hi, We recently integrate OPT-175B with the Berkeley-backed system [Alpa](https://github.com/alpa-projects/alpa). I guess you can try Alpa which exactly allows you to use more heterogeneous GPU setups to train/serve...

Have you considered Cerebras-GPT-13B

We have done some study and our temporal conclusion is that the backbone model Cerebras-GPT has much lower quality than the llama, due to pertaining dataset size, and data filtering...