jetstream-pytorch icon indicating copy to clipboard operation
jetstream-pytorch copied to clipboard

Add per request sampling support.

Open wang2yn84 opened this issue 1 year ago • 0 comments

Supports sampling from request. When user set sampling_algorithm to '', each request can send the sampler config which contains algorithm, temperature, topk, nucleus to enable different sampling strategy. We don't have a good way to support random user provided sampling function yet due to the limitation of Jit compilation.

There will be a coming PR to enable it from JetStream side for the e2e workflow to work.

wang2yn84 avatar Oct 21 '24 20:10 wang2yn84