Evict requests if the client has disconnected

[aniketmaurya]

🚀 Feature

If a client sends a request and is disconnected, LitServe still processes the request which is a waste of computation. We need to evict the request if the client is disconnected.

Motivation

Pitch

Alternatives

Additional context

[bhimrazy]

Hi @aniketmaurya, I've been exploring potential implementations for this feature and came across the req.is_disconnected() method, which seems useful for checking if the client has disconnected. I will try to draft a PR within the next few days to address this issue.

Looking forward to your thoughts on it! 😊

[aniketmaurya]

sounds great @bhimrazy!! the only thing to take care of would be that we send the request to another process so request might lose the ability to check if it's disconnected (haven't tested personally). But definitely go ahead and give it a try.

Another idea could be is to check using socket connection but we can try that later.

[bhimrazy]

Thank you, @aniketmaurya, for the insights, and apologies for the delay in drafting the PR. I've tried implementing an idea by checking for the uid. However, I'm a bit unsure if this will impact performance. It's still a work in progress, but I would appreciate any feedback.

[aniketmaurya]

Commented on the PR @bhimrazy! let's move the conversation there.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.