kiwix-android
kiwix-android copied to clipboard
kiwix
[REGRESSION] Not able to upload 512MB+ custom apps anymore
So far we have been able to upload up to 2GB custom apps using APK extensions. Unfortunately this is not possible anymore (for new apps at least). We need to upload bundles and the ZIM file(s) should be handle as an install time Asset. Unfortunately this asset can only be 512MB big.
We need to find a solution to continue to be able to propose custom apps with ZIM big (over 512MB) files.
Few questions:
- Can we confirm that the limit for one asset is 512MB?
- How many assets could be uploaded for one app?
- Do we have an overall app size limit?
- I see no other way to split the ZIM files, do you see an other possible approach?
@kelson42 I was thinking dividing zim into multiple parts and put them in the assets. Is that technically possible and is it a good idea?
@kelson42
Can we confirm that the limit for one asset is 512MB?
The total download size limit for all install-time asset packs combined is 1 GB.
How many assets could be uploaded for one app?
As many as you want as long as it is less than 1GB
Do we have an overall app size limit?
Yes, The documetaion say it's 150Mb. But while searching for solution some say they can allow a little bit more than that.
I think if there were no size limit, we could achieve this by dividing Zim into multiple parts and storing them into multiple assets. Later after the download, we can merge them into one Zim file. I think this solution would work. But before that we need to overcome the Play Store issues.
Can we confirm that the limit for one asset is 512MB? The total download size limit for all install-time asset packs combined is 1 GB.
So the limit for one asset is?
The download size limit for each fast-follow and on-demand asset pack is 512 MB. The total download size limit for all install-time asset packs combined is 1 GB.
@kelson42 They didn't mention specifically for one install-time asset pack is. https://developer.android.com/guide/playcore/asset-delivery
I faced slightly similar issues with the build process (not the delivery avenue) for the Windows WikiMed custom app. My initial solution was to split the WikiMed ZIM into several 100MB chunks (.zimaa, .zimab, etc.), but when we introduced full-text search, that was no longer viable (libzim doesn't support full-text search on arbitrarily split ZIM archives).
The ultimate solution for me was to download the ZIM at the point of need. Wouldn't it be possible for the Android custom app to do that straight after install, on first launch? Obviously, it's an issue if the user doesn't launch the app straight after install, but effectively the install itself would be a pretty light and quick package (without ZIM), which would then auto-download the required custom ZIM into app's local storage and launch it. Description of app would need to state this clearly so that users are aware it will happen, and that they should launch the app to finish installing.
@Jaifroid You should use zimsplit... and hope the fulltext index is not bigger than 100MB.
Like written, handling ZIM at install time is a requirement.
@Jaifroid You should use zimsplit.
Does that produce split ZIMs that keep the Xapian index all in one chunk (if less than chunk size)? Thanks for tip! Will bear in mind if I need to use split ZIMs in future. For now it all works smoothly.
Like written, handling ZIM at install time is a requirement.
Yes, but if I understood correctly, you've got a limit on the size that can be handled by installer (install-time asset limit). Download after install (not as part of install) would be a different process, and would be handled by the app on first launch, so not subject to any install-time asset limit. IMHO, for the end-user the result would be almost the same, so long as they know to open app straight after install. I'm thinking of a similar process for delivering an optional sample ZIM with the PWA.
Like written, handling ZIM at install time is a requirement.
Yes, but if I understood correctly, you've got a limit on the size that can be handled by installer (install-time asset limit). Download after install (not as part of install) would be a different process, and would be handled by the app on first launch, so not subject to any install-time asset limit. IMHO, for the end-user the result would be almost the same, so long as they know to open app straight after install. I'm thinking of a similar process for delivering an optional sample ZIM with the PWA.
Yes, but there is no reason to talk and assess this now at it is unecessary. Looks like we can do all what we need at install time.
Seems clear we should use ZIM in chunks in the asset bundle, downloaded at install time.
The real problems seem to be:
- The overall limit at 1GB (seems vrry little)
- The libzim limitation which does not allow (yet) to open split ZIM files using filehandles.
@MohitMaliFtechiz @gouri Would you be able to confirm please? Do you see other challenges? Do you have an other approach in mind?
The overall limit at 1GB (seems very little)
MDWiki in English is currently 1.7GB. Hence suggestion of post-install solution.
@kelson42 @Jaifroid I researched a lot! Other developers use their code to download their file but unfourtenly we can't do that here. There is no other way to go up to 1.5GB. We can add install-time which is 1 GB and fast-follow 512MB. We can also add on-demand 512MB, but this is a tricky part. In combine both install-time and fast-follow we can download the assets before the app open button by the Play Store. To my knowledge almost most apps under 1.5GB except WikiSpecies.
Seems clear we should use ZIM in chunks in the asset bundle, downloaded at install time.
Yes!, I think it would work! We need to first test this by dividing a small Zim file into multiple assets.
The overall limit at 1GB (seems very little) MDWiki in English is currently 1.7GB. Hence suggestion of post-install solution.
Yes, this is the only ZIM having this problem, but before changing requirements, we should consider reducing the size of the ZIM.
The smallest version with images is the app-maxi here: https://master.download.kiwix.org/zim/.hidden/custom_apps/mdwiki_en_all-app_maxi_2023-10.zim` which is 1.4GB or 1,506,922,496 bytes (despite the listing saying it is 1GB). This is the version without ft index. I use it for the WikiMed UWP app, but I use the larger version (1.7GB with ft index) for the Electron app.
How we manage to get smaller ZIM file is an other topic. But the overall feeling is that custom apps over 1GB is anyway not recommended.... so maybe this restriction is a good thing.
This ticket is how to get over the 512MB limit, if we could handle this, this would fix that ticket. Even if this would mean having a new restriction on maximal size.
My proposal is to split the zim file into 2 and put it into 2 assets by the ci. While the installation process, we can combine 2 assets together and create one zim flle. That way, we can have under 512 mb limit.
@gouri-panda You could use the mdwiki "mini" ZIM (https://master.download.kiwix.org/zim/.hidden/custom_apps/mdwiki_en_all-app_mini_2023-10.zim) which is 511MB (no images), so a good test case for splitting into two ~256MB chunks.
@MohitMaliFtechiz The most obvious problem we have is that we are not able - for now - to load split ZIM files via fd. I have open a ticket here to request the feature at the libkiwix level.. But I have one question to you: would you be able to secure that if you upload to ZIM chunks that they are one just beside each other (at the byte level) in the bundle? That way we could just open the next fd, just at the offset+1 after the end of the first chunk!
@MohitMaliFtechiz @mgautierfr I don't think the .aab file format can garanty that two blobs/chunks are behind each other. See https://developer.android.com/guide/app-bundle/app-bundle-format
I believe we have to go with solution #1
@kelson42 We are able to read the zim chunks from the asset folder in aab, I have tried it at https://github.com/kiwix/kiwix-android/pull/3526.
@MohitMaliFtechiz explain how you can open many chunks considering that you can only open one fd via the libzim?!
@kelson42, I am following the comment of @mgautierfr https://github.com/kiwix/libkiwix/issues/1014#issuecomment-1797944710
@kelson42, I am following the comment of @mgautierfr https://github.com/kiwix/libkiwix/issues/1014#issuecomment-1797944710
OK, I get it now... but this was not an invitation to try to do so. We will implement a function to open many chunks via many fd in the libzim.... or/and via inputstream...