DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Expose DS-inference and ZeRO-inference configs to user

Open jeffra opened this issue 3 years ago • 2 comments
trafficstars

After #25 is complete we want to expose all DS-inference configs (https://deepspeed.readthedocs.io/en/latest/inference-init.html#deepspeed.init_inference) and ZeRO inference configs in the MII config dictionary.

jeffra avatar May 17 '22 18:05 jeffra

ZeRO inference currently takes the general deepspeed config json path. Do we want to expose DS-inference config in the deepspeed config file as well?

cli99 avatar May 17 '22 22:05 cli99

Good point @cli99, I forgot that the zero-inference PR adds ds_config as a MII deploy argument. Let's leave it like that for now, but maybe we can merge some of these configs in the near future.

jeffra avatar May 17 '22 22:05 jeffra