multi-model-server icon indicating copy to clipboard operation
multi-model-server copied to clipboard

Published 20 hours ago • verified publisher icon awslabs

config MMS to process the newest request only

Open carter54 opened this issue 4 years ago • 3 comments

Hello~ Is it possible to config MMS, let it only process the newest request, and stop all the previous requests if they have not finished?

carter54 avatar Mar 12 '20 02:03 carter54

@carter54 : If you want to stop model workers from processing the requests, you can set the response timeout to a lower value. This will stop the processing of any request which takes longer than the "response timeout".

The request queue in MMS ensures that older messages are processed first (FIFO). There is currently no mechanism to flush this queue. So, if you are looking to delete messages from request-queue, I don't think this is possible today.

vdantu avatar Mar 12 '20 03:03 vdantu

@vdantu I see, thanks for the reply. If I have to delete unfinished request in the queue, do you have any recommendation? Thx!

carter54 avatar Mar 13 '20 01:03 carter54

I can't think of any good way to do this as of now.

vdantu avatar Mar 23 '20 06:03 vdantu