HumanSignal
Webhook not deleted when ML backend deleted
Hello! I have encountered a bug when linking custom ML backends with Label Studio. When explicitly deleting an ML backend, the corresponding webhook for the backend is not also deleted.
For example, if I set up a custom YoloV5 ML backend at 10.40.52.3:9090 then a webhook is created for 10.40.52.3:9090.
# Start the Label Studio ML Backend with the YoloV5 backend
label-studio-ml init yolov5 --script ./yolov5_autolabeling_backend.py
label-studio-ml start ./yolov5 -p 9090 --log-level DEBUG
But if I delete that custom YoloV5 ML backend at 10.40.52.3:9090 from the UI, the webhook persists.
CTRL+C for the previous label-studio-ml start process
If, at some point in the future, I attempt to link a new YoloV5 ML backend at 10.40.52.3:9091 from the UI, there will be two webhooks.
When the specified events handled by the webhooks occur (for example, an ANNOTATION_UPDATED event), Label Studio attempts to send the notification to both URLs. However, if one of the URLs doesn't exist - like how there is no longer a YoloV5 ML backend at 10.40.52.3:9090, the process will fail silently. Not only will the notification not reach the non-existent backend, it won't reach any of the backends.
See the attached logs from the 10.40.52.3:9091 label studio backend and the label studio container, where this exact situation played out. I attempted to update an annotation in a project with its custom YoloV5 ML backend at 10.40.52.3:9091 - and this backend never received the ANNOTATION_UPDATED event. The container shows that attempting to send notification of the ANNOTATION_UPDATED event to 10.40.52.3:9090 failed - and that finally led me to look at the webhooks page in the UI.
A few potential solutions:
- Delete webhooks after explicit deletion of their corresponding models
- Notify users with a pop-up upon model deletion that they may have persistent webhooks
