ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

Published 20 hours ago • verified publisher icon hpcaitech

[FEATURE]: serving multiple models

Open dlzou opened this issue 2 years ago • 4 comments

Describe the feature

Currently, does Colossal-AI have support or ongoing work for deploying multiple models concurrently, possibly using the Ray framework?

For context, I’m doing a course/research project related to multi-model serving, inspired by the AlpaServe paper. My professor referred me to Colossal-AI, and I would be interested in incorporating it in my project.

dlzou avatar Apr 07 '23 02:04 dlzou

this will make it great

ucas010 avatar Apr 07 '23 09:04 ucas010

Interesting feature. Would you like to attempt to make an implementation roadmap?

JThh avatar Apr 08 '23 07:04 JThh

As I'm still learning how to use Colossal-AI, perhaps someone with more experience can lay out a general roadmap for this.

My project is more limited in scope, and the usage I'm interested in likely deviates from the broader use case. In particular, I'm looking to colocate multiple models on a set of devices like figure 1 in the AlpaServe paper shows.

dlzou avatar Apr 10 '23 18:04 dlzou

Hi @dlzou Yes, we are considering it, https://github.com/orgs/hpcaitech/projects/17/views/1

binmakeswell avatar Apr 17 '23 06:04 binmakeswell