Android
Android copied to clipboard
CatimaLoyalty
Card duplication detection during import doesn't always work
In #2337, @KlausD-GH mentioned getting duplicates of all cards when importing the same Catima export twice. I've been able to confirm this with an export they sent on the latest release.
I have not yet had the time to research what exactly causes this, but my guess would be that it is related to images (as this is less commonly tested part of deduplication).
Low severity (as duplicates aren't data loss, but a bit annoying) and low commonness (as most users won't import the same import twice). But fixing this would still be a good thing to do.
(Note to self: this export can be found back in a mail sent on February 13, 2025)
Should it be an idea to add an uuid to every card so that import-export duplicates can be managed more easily?
I don't think that will really help this case.
The issue here is specifically: the code incorrectly fails to be able to tell that a local card and a card in the backup file are the same when they are. Simply ignoring any card in the backup with the same ID would just bring back bug #512 in full glory.
A more complex UUID (like 6a0649a2-3089-4208-8544-a1126adff8cb or whatever) would help differentiate cards if they're created on different devices, to help in syncing cases like #791, but I don't think it will help with this specific issue.
I am a new user to Catima and was surprised that importing a just created export with images results in duplicates, even though https://github.com/CatimaLoyalty/Android/pull/1422 states that it should not.
Here is a simple test export with one card and an image: catima.zip
Just import it twice and it results in the same card being there twice. Tested with Catima version 2.34.5 (installed via Google Play Store).
Thanks for providing an affected export publicly so others could work on this issue too :)
Does anyone already verified if the problem is the image compression done during the different steps of the process? I noticed that a compression is done while exporting (in the bitmapToByteArray) and also while saving (in saveCardImage) and these recompressions may change the checksum. At the end, the image in the zip is neither the original one nor the one in the app after the import. I don't have an android environment available atm so I can't check if I'm right.
I think we concluded the image compression is most likely the issue. I at least can't trigger this bug in an export without images, yet the small test case with image shared before triggers it reliably
Hello @TheLastProject , I would like to be assigned to this issue. During testing, I discovered that while comparing images, imported bitmaps are always null. So function not return that imported card as duplicate. Problem is that images are loaded at a later stage, after solving duplicates.
I would therefore suggest discarding image comparison and detecting duplicates based on other information. Then, update the images from the new import. Or its possible to add some popup to let user decide.
Nice find @matuspetro
I would therefore suggest discarding image comparison and detecting duplicates based on other information.
Well, what other information would allow you to know what's a duplicate? And what is the correct, new version?
Then, update the images from the new import.
It seems dangerous to assume the data in the import is always newer, we want to prevent data loss.
Or its possible to add some popup to let user decide.
I'd rather not complicate the import logic as it is, as it's extremely deprecated: https://github.com/CatimaLoyalty/Android/issues/513. If we have a working background import, that would be nice yeah.
So for now it's probably best to see if we can load the images first before comparing.
@TheLastProject I fixed the problem with img comparing, now it's working fine, but I find another one. So I want to discuss it with you.
Main logic problem is here, duplicity is comparing just with card with same ID. Example: User create card, fill info and export it. Then change the card (for example, name). If user try to import old that export, systems will check duplicity, find existing card by ID and here it is... existing card has new name so it is not duplicity. So system create new card. Theoretically this is good.. But now, when user try to import the same import again (which is rare...), it will still import that card, because with first card it has different name and with second card it has different ID.
Do you understand what I mean? You can try it with no img. It is the same issue. I was thinking about solution, and it is possible to check duplicity with every card ignoring ID. What do you think?