Keshav Santhanam issues

Results 9 issues of


                                            Keshav Santhanam

Rename lease variables

The lease variables `duration` and `max_duration` are too ambiguous - rename these

Add heartbeats to allow scheduler to remotely kill failed jobs

Clean up imports for scheduler.py

Make Scheduler and Profiler subclasses of SchedulerMechanism

The `Scheduler` and `Profiler` classes currently share a lot of code - we can factor this out into a common superclass (e.g. `SchedulerMechanism`)

Do not do initial retrieval if pids are passed in

Variable length generation does not support padding tokens

Variable length generation requires `seq_idx` and `cu_seqlens` to span the full input sequence length but in some cases I would want to include padding tokens (e.g. for maintaining static shapes...

Variable length generation does not match single-request-at-a-time generation

The outputs produced by variable-length generation (i.e., passing `seq_idx` and `cu_seqlens`) do not match the outputs produced by sequentially generating a single request at a time. I have included a...

Use DynamicInferenceCoordinator for text generation server

Updates the text generation server to use the `DynamicInferenceCoordinator`.