Auto label contribution

[marcfielding1]

Hey Guys,

So I'm from startup in the UK who are using Via pretty extensively to label our data, one of things we've build is some additional code that gives the following in via:

1. User loads images
2. After labelling the first ten Via pings a backend we built in python that trains a model(starting with coco weights) for one epoch(configurable)
3. On subsequent image loads(when you step to the next) Via asynchronously asks the backend for inference which produces a mask or bounding box.
4. That bounding box is then added to the current image and can be tinkered with as required.

The processes can be repeated every X images, so for example 20,30,40 - it works well for us because we tend work stepping through video frames so the variance in the images isn't a lot, this means it becomes pretty handy, pretty fast.

One thing we've yet to work on is the Masks produced have a LOT of points compared to hand drawn ones so we intend on smoothing them out at some point.

I didn't really want to fork Via but I'm also not sure if you'd want to pull all that functionality into the main repo. Is this type of feature of interest?

[marcfielding1]

Saying that I suppose we could add it as a module/plugin - we also need to refactor our code since all it essentially is, count up X images, offload the annotations and the images to an API endpoint and then make a call each time you load an image in view in the model is available.

We have a small HTML span which shows the model status IE X images till next train and then green if it's using a model orange if it's training or not got enough images.

The actual implementation we're using is Matterport's MaskRCNN - I guess we could demonstrate this and then if the community liked it allow other people to replicate the setup for different implementations.

It also is useful because if you say have COCO pre loaded, for example coco detects a tin can as a "bottle" now that loaded into via you simply take the region and add whatever classId's you'd like but you've not had to draw a bounding box or a mask. In some cases we can actually autolabel 1000 images just off 20 manually labelled ones in via.

OR

We just provide the plugin that allows the user to specify an endpoint that will return Via annotations for the current image - that's actually a better idea.

[thelinuxmaniac]

Hi Marc, Thank you for offering to share your extension of the VIA software. If find the workflow that you have described very interesting and it should be useful to our other users as well.

We just provide the plugin that allows the user to specify an endpoint that will return Via annotations for the current image - that's actually a better idea.

Yes, I agree this is a better way to incorporate your ideas into VIA. You are addressing a more generic work flow that exists in a large number of computer vision projects.

POST /dump
  payload: a VIA project which contains some (say 10 ) manually annotated images
  response: unique MODEL-ID which denotes the model trained using provided manual annotations

POST /annotate?model=MODEL-ID
  payload: an image
  response: if the model training has not completed, the response will be {"status":"not-ready"}. If the model training has completed, the response will be {"status":"done", "regions":[...]}

For example, for your workflow, you would load a VIA project with 1000 images and define the HTTP API endpoints for "/dump" and "/annotate". When you manually annotate the first image and move to the next image, the VIA application sends the full VIA project to the backend, since the backend wants 10 manually annotated images for training an epoch, it will ignore this and the next 9 such requests. When the 10th request arrives, the model training proceeds and returns the model-id for a training model. When VIA receives model-id, it sends the subsequent images to the "/test" endpoint to receive automatic annotations. These automatic annotations are shown to the user.

The idea is to keep VIA application ignorant and include all the intelligence (based on a user's requirement) in the backend. The VIA application will perform only the following two tasks (without knowing anything about backend models): a) DUMP: The VIA application dumps all its manual annotations to a predefined HTTP endpoint (e.g. "/dump") after certain criterion is fulfilled (e.g. 10 new images have been manually annotated)

b) ANNOTATE: The VIA application fetches automatic annotation from a predefined HTTP endpoint (e.g. POST /annotate" after certain criterion is fulfilled (e.g. GET /annotate?model=MODEL-ID returns success implying that certain model is ready to serve)

The DUMP and ANNOTATE behaviour is very generic for common annotation tasks. For example, if I want to use pre-trained face detection to automatically detect all human faces and the manual annotator is required only to make final adjustments, they can only use the ANNOTATE behaviour for a backend that is configured to detect human faces in posted images (without requiring any training). For workflows that require training, they can use both DUMP and ANNOTATE behaviours of the VIA application.

Would you like to go ahead and implement this in VIA and send me a pull request?

Note: This github repository is a mirror of our gitlab repository at https://gitlab.com/vgg/via/. In future, please post your queries at the gitlab issues page at https://gitlab.com/vgg/via/issues

[thelinuxmaniac]

If possible, please use this gitlab thread: https://gitlab.com/vgg/via/issues/240