jidoujisho
jidoujisho copied to clipboard
lrorpilla
Crash when importing JMDict
Hi, thanks for your work on Yuuna, though I've encountered a problem.
On v2.0, the app crashed when I tried to import any versions of JMDict, while other dictionaries works fine. It wasn't a problem on v1.1.12, only on the newest release. I've also included a screencast below.
Edit: Seems like the app also crashed when I'm trying to import any dictionary with large entries, though v1.1.12 imports without any issue.
These JMDICTs I got from TheMoeWay's Resources page.
Additional info: Device: Kindle Fire HD8 Plus (10th Gen), with 3GB of RAM. Android version: 9 Jidoujisho version: jidoujisho_yuuna_2_0_0-armeabi-v7a_hotfix1
https://user-images.githubusercontent.com/48660041/188803227-ff39fd55-9cfa-423b-89ec-793c2f09377c.mp4
Hi, I'll have a look and see what's going on. For now, the JMdict you can download from the Yomichan site should work fine.
The JMdict from shoui's collection seemed to import fine after I renamed the file to something else without special characters, but failed without.
How much free space do you have on the tablet? The database demands a lot of storage space for indexing.
I have documented what I've done based on your suggestions:
- Tried out the JMDICT from Yomichan's site. Crash.
- Renaming the .zip file to both with/without special characters. Crash.
- Double-check the storage, there is>5GB of free space on both Internal and SD Card.
- Quit all running apps except Jidoujisho Yuuna to see if it's a RAM issue. Still crash at importing.
- Tried importing using other dictionaries, mostly crash if they have a lot of entries.
And all the while v1.1.12 still imports fine without a hitch.
2.0.1 is out, though I doubt it would make a difference. Chisa and Yuuna use entirely different database solutions, so really I can't track the difference between the two because what goes on with the two dictionary import operations is completely different.
The current build I am using for the database (Isar) is not the most updated, and there have been massive improvements to put operations since, but I have to wait till they go stable and I can migrate.
I'll investigate if it is probably something else. What would be really useful, though it is much to ask, if you can somehow get the logs for the crash through Android Debug Bridge.
I've included the logs from running the two commands adb logcat -b crash, and adb logcat where I attempted to recreate the crash.
adb logcat -b crash 202209070733.txt
logcat 202209070736.txt
As you said, the problem is still there on v2.0.1. At least for the time being I can still rely on the Kenkyusha/Shinwaei dict(which imported fine) until the cause of the import bug is identified. Apart from that, Yuuna has been a solid update, with no other issues so far.
To address this issue, I have upgraded the backing database solution and improved the performance of the dictionary search speed. This is a preview version, so your mileage may vary.
You will need to wipe your data or uninstall the app first in order to use this version.
The preview can be found here.
To address this issue, I have upgraded the backing database solution and improved the performance of the dictionary search speed. This is a preview version, so your mileage may vary.
You will need to wipe your data or uninstall the app first in order to use this version.
The preview can be found here.
Hi again, thanks for still sticking with this issue.
I've done a clean install of v2.1.0. Unfortunately, the crash to home screen is still there when I'm trying to import dictionaries with large entries, no difference from v2.0.1.
If small dictionaries import fine, maybe it's a matter of all the work having to be done at once. Can you try and see if importing many small dictionaries causes an issue?
If you can import a lot of small dictionaries with around the same amount of entries as one big dictionary that doesn't work, that tells me that the issue is the size of the work to be done and maybe it'll work out if I split up the import into many small steps.
I'm not sure what you meant, but I did try to import 5 dictionaries with a small number of entries all at once: the import process went through fine without issue. By the way, it was the same with v2.0.1.
After that, I tried importing a 300k entries pitch accent dict (around the 2 min mark). Sure enough, Jidoujisho crashed, even though the .zip was less than 3MB.
Here's the screencast: https://user-images.githubusercontent.com/48660041/191710870-9d76ee99-0943-4409-8318-87a353c842ff.mp4
What I meant is that if I stage the import of a big dictionary into maybe ten small steps, maybe the import will work.
If you can say, import 50K entries ten times, but not 500K all at once, then that tells me maybe something I can do is divide the work and perform it ten times, instead of stressing your device to perform something all at once.
So, if your device is able to play well with say, 300K entries imported in small amounts, then I think we have a potential solution.
@mcdmcdmcdmcd I'm going to share a build that splits the database imports into 10K objects maximum at a time, multiple times. Can you please see if it performs any differently on your device?
I've updated the preview builds in the same link here.
@mcdmcdmcdmcd I'm going to share a build that splits the database imports into 10K objects maximum at a time, multiple times. Can you please see if it performs any differently on your device?
I've updated the preview builds in the same link here.
Tested your new v2.1.0 build: the import speed was noticeably slower, but I am now able to get all dicts to work, including ones with large number of entries. Seems like this is the solution going forward!
So in the end, is it a device-specific bug, or it is related to Yuuna's new database solution?
Cheers!
It's probably an issue on low to mid-end devices in regards to putting a large amount of objects in the database all at once in a single operation.
The solution was to split up the put operations into multiple batches, basically split entries into multiple lists of entries, and perform smaller, multiple put operations. This problem didn't exist in Chisa because that used a different database solution (ObjectBox) and that probably handled it differently. I'll probably report this issue into the Isar repo but given the solution is pretty straightforward, I wonder if it'll be regarded as an issue of that package.
Really glad to hear it's working, I'm going to include this in the final 2.1.0 release.
Ah that's good to know!
Anyways thanks for finding out the problem and pushing a fix. You can probably close this issue now.