1D barcode performance

[thegnuu]

Currently, the performance when scanning 1D barcodes like code 39 or ean 13 is not great. Since there are multiple issues regarding this subject I closed the older issues and reopened this one to keep track of it.

Old issues: #77 #104

Current ideas:

• Use IntegratorView on android (PR #72)

[thegnuu]

According to my tests reading 1D barcodes (EAN 13) on iOS is working pretty good if the barcode is positioned properly in the center of the rectOfInterest which defaults to the center of the image captured.

https://developer.apple.com/library/archive/technotes/tn2325/_index.html https://developer.apple.com/documentation/avfoundation/avcapturemetadataoutput/1616291-rectofinterest

Further testing on android is required, but I think adding some sort of optional overlay based on this rect of interest would really help in this case (#10). Maybe even allowing to change those settings might make sense...

@sephallen Can you verify that scanning your code 39 barcodes works in the center of the screen?

[sephallen]

Scanning a code 39 barcode on Android is pretty fast (if, as you mention, the barcode is held in the middle of the viewfinder - I would love for a way to move the rectOfInterest by the way!).

As for iOS, no matter where the barcode is positioned, it takes me around 15 seconds to successfully scan a small code 39 barcode sticker - the same sticker on Ionic 3 using other plugins which utilise AVFoundation are almost instant.

[thegnuu]

Well, at least this allows us to assume that there is an issue at our side of the implementation which should be fixable ;)

There are a lot of possible settings using AVFoundation - can you point me to a plugin which works for you? It should be easier for me to find something with a better working implementation :)

[sephallen]

Sadly, as I mentioned in my initial issue, I believe this to be an Ionic 5+ issue (perhaps even on Ionic 4, though I have not tested this).

If you were to try this plugin (https://github.com/phonegap/phonegap-plugin-barcodescanner) on both Ionic 3 and 5+ on iOS, you should notice a huge difference in scanning speed (for code 39 barcodes) between the two Ionic versions.

[thegnuu]

Hmm this is weird... Since ionic 4 capacitor is used as default as far as I know, prior to v4 it was cordova if I'm not wrong.

I don't think that the issue has something to do with ionic itself, since I am using capacitor completely without ionic.

If there would be an issue between the communication between the native capacitor stuff and the js implementation, the delay should apply to all codes as well...

I will play around with some settings if i find time - maybe I can figure something out :)

[sephallen]

That is interesting! So are you seeing the same delay in scanning code 39 barcodes on iOS with Capacitor without Ionic? The reason why Ionic looked like the likely culprit was because initially I was using Ionic 5 without Capacitor (Cordova only). One of the reasons why I added Capacitor to that project was to switch to this barcode scanner in the hopes that it may improve code 39 scanning performance.

Could this be something as simple as Ionic 3 targets a lower iOS version by default and those lower iOS target versions happen to be more performant in scanning code 39 barcodes? I will experiment a bit and let you know if I find anything.

[sephallen]

The Ionic 3 project which does not have code 39 issues targets iOS 11 and my Ionic 6 project targets iOS 12. I was not able to target iOS 11 with the Ionic 6 project due to a Swift dependency, but I was able to increase the target to iOS 13 and it did manage to scan a code 39 barcode much faster. I did need to position the barcode in the center of the viewfinder though. To sanity check, I built out to iOS 12 again to compare and I found that if I move the phone up and down slightly with the code 39 barcode in the middle of the viewfinder, it does scan almost as fast as targeting iOS 13 - perhaps your initial thought regarding rectOfInterest is playing a large part of the issue here.

Would it be possible to make the rectOfInterest cover be the entire size of the viewfinder? Or would this be very resource intensive?

[thegnuu]

According to the definition the default rectOfInterest is (0.0, 0.0, 1.0, 1.0) which should be the entire capture size of the capture session, but based on the documentation 1D codes are only scanned in the center. I guess scanning 1D codes require more resources, but since only the center is used I don't know if it will make any difference.

I need to test if creating a smaller rectOfInterest while using a higher resolution (which is not recommended by Apple, but using anything other lower than high will result in "no codes are scannable" anyways) might help....

A proper rectOfInterest, resolution and maybe even only check for the codes really needed might indeed help....

This link is only a reminder for me, there might be some additional info: https://www.appcoda.com/simple-barcode-reader-app-swift/

[thegnuu]

That is interesting! So are you seeing the same delay in scanning code 39 barcodes on iOS with Capacitor without Ionic? The reason why Ionic looked like the likely culprit was because initially I was using Ionic 5 without Capacitor (Cordova only). One of the reasons why I added Capacitor to that project was to switch to this barcode scanner in the hopes that it may improve code 39 scanning performance.

Could this be something as simple as Ionic 3 targets a lower iOS version by default and those lower iOS target versions happen to be more performant in scanning code 39 barcodes? I will experiment a bit and let you know if I find anything.

Yes, I can see the same delay on 15 with a capacitor/angular combination without ionic... Changes in AVFoundation might be an issue on higher iOS versions, but it does not really make sense IMHO that newer and faster devices running iOS 15 have more issues than older devices running iOS 11 or whatever.

[sephallen]

Thanks for looking into this, please let me know if you need me to test anything!

[thegnuu]

@sephallen I created the issue #107, I was playing around with different settings but I was not really able to get better results. On my device (iPhone 15 Pro Max, iOS 15) the detection works relatively fast, but only if the code is positioned exactly at a given line.

So my idea was to make a PoC with Google ML Kit to see how this works on Android and iOS, this would allow us to fix other issues as well, or at least provide a solution for them. I think it is worth looking into it, because I think a properly trained ML model can provide better result than the current implementation....

[thegnuu]

@sephallen would it be possible for you to quickly clone and install https://github.com/googlesamples/mlkit/tree/master/ios/quickstarts/vision on one of your test devices to see how this works?

The outputs on my device are pretty amazing to be honest :)

[sephallen]

@thegnuu the results are very impressive for larger barcodes - especially with the live view! Sadly, I was not able to scan a code 39 barcodes though :(

I have attached a couple of photos of the code 39 barcodes I am working with to give you a sense of the size.

[thegnuu]

Well, this thing is really small, I guess for sizes like this we need a higher resolution (the example uses AVCaptureSession.Preset.medium) resulting in a higher latency, hight should be enough though. Need to test it out, but it might work. Can you tell me the text within that code? If I create a code 39 with the written serial I get a more detailed output somehow....

If i use full ASCII the code is even longer:

[sephallen]

Very small! The barcode only displays the last 6 characters, so in the example above it contains: e0013e.

[thegnuu]

Changing the session capture preset to high allowed me to scan the code... I printed your code with a width of 2.2cm.

There are some false positives, so I guess handling small codes like this one would require some app side checks, for example to only trigger a detection when the same code is sent x times, but it should be possible with ML Kit.

I did not change the outline rendering so the green rect is positioned wrong, but it works :)

To be honest I don't see a way to get codes with this size to work with the current implementation...

[sephallen]

Nice work! And yes, validating in the app would be the way to go.

[sephallen]

@thegnuu apologies for chasing up, but has there been any more progress on this?

[thegnuu]

Not for now, I am currently preparing the new interfaces for the plugin, I will create a branch this afternoon. Once this is done and we are happy with it we will start the implementation of ML Kit natively for Android and iOS. It hardly depends on the help I will get from others, if I have to do all the work myself it will at least take a month or so, since I have a lot of other projects going on. But a few people offered help on this one, so I think we might get results faster 👍🏼

[sephallen]

@thegnuu That sounds promising! When you commit the new branch, I will pull it down and start using it within my app - if there is anything I can fix, I will attempt to do so too and create some PRs.

My client has also enquired if there might be a way to sponsor you / this plugin in the hopes of speeding up development - is this possible?

[thegnuu]

@sephallen sorry, somehow I missed that...

Since it is a community plugin and I'm just helping to maintain it I am not able to setup some sort of sponsoring. I don't think it makes sense anyways, since it would be pretty hard to get the funds to the right person.

At least for me personally it is not a question of money anyhow, the time aspect is more problematic.

If my current plans go well I will start with the Android implementation around next week. I think the main thing you can do to help is to offer some of your time with testing and developing :)

[sephallen]

@thegnuu sounds like a plan, I'm looking forward to it ;)

[thegnuu]

@sephallen I just wanted to update you that I have a working iOS ML Kit version on my device which is able to scan the code from the photo directly from my macbook screen. Sometimes there are false values, but it is hard to say if it will work if you have direct access to the code.

It will take me some time to finish the details, but in a week or two you should be able to test it (at least on iOS). I am not really an Android developer so I am not sure how long it will take me to mirror the implementation to android and create v3.0.0 of the plugin.

[thegnuu]

I decided to start with iOS since I used Swift more often in the past months and the implementation looked simpler in my opinion to create a PoC :)

[sephallen]

@thegnuu that's great news! Thank you for the update :)

[thegnuu]

@sephallen would it be possible for you to have a look at the ml-kit branch and test it with that one?

[sephallen]

@thegnuu I've had a bit of a play with it and it appears to work. I needed to do a lot of refactoring based on the new typings but it isn't entirely clear how I am supposed to achieve a continuous scan as the docs do not appear to have been updated on that branch. Do you have an example on how I should do this?

[thegnuu]

@sephallen Yeah you are right, still WIP and I need to completely refactor the readme. But since there might be changes I thought it will make sense to do this once the rest is finished :)

I dropped support for a single scan action, since you can do this in a callback without issues. I will provide a proper upgrade-guide once I am done. The startScan and startScanning was confusing for some users and it was a result of a PR which added continuous support that was previously missed.

Therefore there is only the scan method which takes a callback. I did some work on iOS and somehow broke it yesterday, I receive the logs from the js bridge but the callback is only called once somehow.

  public async start(): Promise<void> {
    this.bgColor = document.body.style.getPropertyValue(
      '--ion-background-color'
    );
    // await BarcodeScanner.checkPermissions();
    await BarcodeScanner.requestPermissions({ permissions: ['camera'] });
    await BarcodeScanner.start({ formats: [BarcodeFormat.QR_CODE] }, (code) => {
      console.log('debug', code);
    });
    document.body.style.setProperty('--ion-background-color', 'transparent');
  }

This is my ionic/angular test code at the moment... I am currently working primarily on iOS on this stupid callback issue and on some orientation issues to calculate the proper cornerPoints of the barcode. For some reasons mlkit is not recognising an orientation change at the moment. But if you start the scanning in portrait and keep it like this the cornerPoints should be okay.

[sephallen]

@thegnuu what you're saying makes a lot of sense. Thank you for the example code, it does appear to work on Android at an acceptable speed, albeit still a little slow. It also appears that, like the previous version, only the very center of the viewfinder is capable of reading the barcode - is there any way to adjust this?

[thegnuu]

@sephallen I was not testing the android part fully at the moment, I just made the PR work with the new interfaces. What I found out until now is that it seems like the autofocus tries to focus to the element in the center of the viewport and this results of course in blurry images if you move the code to the edge of the screen, at least it works like this on my Pixel 6.

But the results are still not that bad, even if the image is blurry... Currently there is a piece of code which prevents multiple calls from the same code, if e remove this the updates are quite frequent event if the barcode part is not focused.

I have no big knowledge about this ecosystem and the usage of cameraX, but I will try to find a solution. And my Android Studio is somewhat broken which does not help as well ;)

I have to rewrite the permission stuff anyways since I changed it on iOS to allow getting an image from the library in a future update. I will align this to the usage in @capacitor/camera to fix a few issues.