FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Published 20 hours ago • verified publisher icon lm-sys

Awesome Jobs guys - Got it work in Docker it's fast on a my 3060 and even faster on my 3080

Open CTGS-Innovations opened this issue 2 years ago • 3 comments

I would like to know how fast, I could locate and setting to see token/s any change this is an option or change info the logger settings ?

Awesome job tons of fun.

CTGS-Innovations avatar Apr 12 '23 00:04 CTGS-Innovations

You can write a simple throughput calculator here https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/inference.py#L98 and estimate the throughput (e.g., words/s) right? Contributions are welcome.

zhisbug avatar Apr 12 '23 01:04 zhisbug

cool, some updates will be inbound soon.

CTGS-Innovations avatar Apr 12 '23 06:04 CTGS-Innovations

How did you do this? How did you set up the Docker? Can you guide me?

billsecond avatar Apr 15 '23 17:04 billsecond

How did you do this? How did you set up the Docker? Can you guide me?

it's a bit of a manual process right now, working on a simple script possible one click. keep checking back with me and if you can run a install.sh I would hope this would get you 99% of the way there.

CTGS-Innovations avatar Apr 17 '23 05:04 CTGS-Innovations

It seems the problem of this issue has resolved -- closing.

zhisbug avatar Apr 21 '23 03:04 zhisbug