petals
petals copied to clipboard
Report average queue size in tokens (per last 10 min) for routing
This would help to take the server load into account while planning a route for inference and fine-tuning.
This would help to take the server load into account while planning a route for inference and fine-tuning.