Idea: Try an implementation with Googles ML Kit

[thegnuu]

Due to some issues which are hard to resolve in a cross-platform manner, it might make sense to create a PoC with ML Kit.

https://developers.google.com/ml-kit/vision/barcode-scanning

• This should enhance 1D barcode detection (#105)
• Allow a client-side implementation of a frame with the help of cornerPoints (#10)
• Would simplify the development since the same library can be used on iOS and Android

It might be absolutely possible that after the PoC we find out that it is not working better, but at it least it is worth trying since it should not be too much work to create a test plugin :)

[JanMisker]

@thegnuu Looks interesting! The samples provided in the MLKit docs are already pretty close to what the plugin should provide, so indeed should be pretty doable to implement a PoC.

[thegnuu]

Indeed yes, I already played around with the example app and the results are really promising so far.

It will take a few hours to create a PoC though, at least for me since I am not as fluent in Swift/Java as in other languages, but I should find some time in the next two weeks to tinker around with it 👍🏼

[JanMisker]

It will take a few hours to create a PoC though, at least for me since I am not as fluent in Swift/Java as in other languages, but I should find some time in the next two weeks to tinker around with it 👍🏼

I'm quite comfortable with Swift; if you can do an Android PoC implementation I could use that as a guide to make the iOS one.

[thegnuu]

Sounds great! I spent some time today to read some examples how all of this has to be done...

Unfortunately I was not able to start working on it, since I was not really sure where to start... I think the easiest way will be to create a completely new plugin to work on this one and move it to this repo once most of the work is done...

This seems to be a good and simple example to start the android implementation.

[phal0r]

Pretty cool idea. Wouldn't this also allow to detect multiple codes at the same time and send and update the position of the detected codes? In combination with the overlayed webview, this would allow to augment the barcodes with related content within the webview.

[thegnuu]

@phal0r Exactly yes! This is one of the reasons why I think this is a good idea. It would allow us to detect any barcode within the camera frame and report back all of those with the position where they are detected.

[phal0r]

@thegnuu yeah, so what does it need to move this forward?

[thegnuu]

@phal0r Right now, everything ;)

I played around with the demo projects of ML Kit and thought about implementing a simple PoC first, but I realised that once the PoC is implemented most of the critical parts are already done...

As I already told, I am not often developing native code. @JanMisker offered help on the iOS/Swift part of the implementation once the Android part is done, so I will try to focus on the Android part. If you @phal0r have some Android/Java/Kotlin experience and could offer some help, that would totally speed up the process. I have a lot of other projects running at the moment and it might take me some weeks to get this done (especially because I am not really fast at coding this stuff at the moment)...

I can create a branch this week and publish the (in my opinion) required interfaces changes for the new implementation. This is the only thing I was currently able to prepare.

[phal0r]

@thegnuu yeah, let me know, when the interfaces are available. This will certainly help :)

[thegnuu]

Of course, I will do this this week and let you know!

I think we can reduce the current functions and remove the "single scan" ability, since this can be easily implemented by each developer itself.... also things like hideBackground and showBackground do not really make sense, since they are called by the plugin anyways.

[thegnuu]

@phal0r & @JanMisker I just pushed the branch ml-kit for this changes...

The only thing I updated right now is the typescript definitions part: https://github.com/capacitor-community/barcode-scanner/blob/ml-kit/src/definitions.ts

What do you think about this changes?

[phal0r]

@thegnuu yeah, makes sense to me.

So in addition my general considerations:

• the js side is responsible for deduplication of scanned codes
• the native sides provides scanned codes as a stream, that will contain the same code multiple times (i.e. with the new coordinates due to phone movement)
• the native sides controls, how often the scanned codes are send to the js side (i.e. for repositioning or newly detected codes)
• the js sides needs to decide, when to stop detection

Regarding these considerations, the interface makes a lot of sense, so I guess, you would agree to my thoughts, correct?

[thegnuu]

@phal0r exactly my thoughts yes. In my opinion we should keep the native side quite simple and let the individual implementation handle parts like deduplication and when the scanning is paused, stopped aso.

ML Kit will generate a lot of events, so a small throttling on native side might be required, but I think we will see this once we have a simple test environment...

[danielberebel]

Hi, I found other package on development (apparently abandoned) for capacitor ml kit barcode scanner. https://github.com/franckysolo/capacitor-mlkit-barcodescanner

And other for cordova. https://github.com/Ratsoc/cordova-plugin-firebase-ml-kit-barcode-scanner

I hope it can help you . And although I have no experience with native development I want to offer my help for anything.

[thegnuu]

@danielberebel sorry for my delayed response and thank you for pointing to that examples, this will surely help to create at least the Android part of the plugin.

If you would like to help you can try to start the rewrite of the Android plugin - I won't find time until next week to get started, if there is already some work done I can pick up the work from there :)

[phal0r]

@thegnuu Is there a branch somewhere already?

[thegnuu]

@phal0r Kind of: https://github.com/capacitor-community/barcode-scanner/tree/ml-kit

Currently only the new interfaces are committed, I am currently working on it and trying to get at least a single code scanned on Android.... But it's not going forward really fast since I have to look up a lot of stuff ;)

[Taufeeq-Mowzer]

Hi, I'm not experienced at all but I see that this has been done with flutter and was wondering if you could use their Java/Ios code to help form your bindings. *Github repo of the most used barcode scanner implementation *Github repo of the flutter barcode scanner using firebase ML Kit

[thegnuu]

I was finally able to get ML Kit working (I am currently only working on the iOS version) and so far the results are really promising. I need to do some cleanup but in a week or so I should be ready to provide a iOS test build.

Would anyone out there be willing to support the Android development once the iOS version is done as an example?

[smachi]

It would be awesome to have an MLKit implementation for this plugin as the current version is not working well for us (it isn't able to scan some barcodes...)

[ferranJS]

Would anyone out there be willing to support the Android development once the iOS version is done as an example?

Hi I managed to make a quick little iOS app (permissions are not given yet so had to make a workaround) and the results are very good so I'm looking forward to starting with coding the Android version. If anyone wants to join me in the implementation, I'm open to get some help! as I'm not very experienced in Android development.

[1andonlyziz]

@fewNeurons I'm currently working on android implementation the results are quite promising , I need to do some code clean up and then I'll Create a PR for it.

My next milestone is drawing rectangles around the barcodes that was scanned and detected on the camera View

[thegnuu]

@1andonlyziz sounds great! Were you able to make any progress or do you need any help somewhere? :)

[1andonlyziz]

@thegnuu yeah, I created a PR on ml-kit branch, please review and take a look, I can make a demo if you want ^_*

the only thing that is missing is drawing graphic objects on the barcode view when something is scanned with startScanning function

[thegnuu]

@1andonlyziz awesome! Thank you very much for your work! I will have a look at it later today!

I don't think that this feature is necessary to begin with to be honest, surely we can think about it later. :)

[thegnuu]

Hi everyone!

Thanks to the great work of @1andonlyziz the ml-kit branch should now work on android as well! So far, the results are really promising!

There are still a few TODOs and some cleanups that need to be done, but I should be able to create a dev-release in the upcoming week or so!

[PetervLeeuwen]

Hey @thegnuu , when do you think the ml-kit branch will be merged to master and available in the official npm repo? I would love to start using this feature! 😄

[thegnuu]

@PetervLeeuwen I would love to release it as soon as possible as well, unfortunately, I am way too busy at the moment to provide a proper ETA, I really hope I can finish the open work by the end of this month, but I cannot promise anything. I will do my best though.

[JanMisker]

Hi @thegnuu I'm more than happy to help out. I am testing the branch and it seems to work quite well already. I do see a couple of small things, should I just make issues for each separate? And submit PRs if I have them? Or should we coordinate more? Not sure how much double work there would be... For now I see:

• [ ] No support for setting the camera direction on Android
• [ ] Camera and scanning 'hangs' when switching on the "torch" light on iOS.

[thegnuu]

v5.0.0-beta.1 is available now :) This is an initial pre-release of the ml-kit implementation and should help anyone who is willing to test the new implementation with an easier setup ;)