here-sdk-examples
here-sdk-examples copied to clipboard
heremaps
Android Navigate SDK: MapDownloader
Hey, I'm trying to utilize MapDownloader for my app offline mode.
During my testing I found some notes that would be worth mentioning, hopefully, you will be able to help me resolve those.
- Downloading regions.
Most of the time our app will use the whole US area downloaded for offline use - this is quite a lot of data ~8.5 Gb. Downloading it to internal storage takes around 35-40 mins - and this is quite fast, no issues with it.
But once I start using the external storage, like an SD card the download speed drops a lot. I notices that while downloading 51 US states to an SD card, the first 10-15 states downloads with a similar speed as they would on internal storage. But the more states are downloaded to the SD card, the slower download speed gets. I was trying to find out a reason and logged all the download progress.
At some point, once there are multiple states downloaded, and you keep downloading the DownloadRegionsStatusListener might stop producing onProgress callback. With normal conditions onProgress callback is called several times per second, to indicate the progress change. But it might randomly stop for a few minutes and then continues producing the updates. It happens for random states at a random percentage and the more states you have downloaded, the bigger pause you will get. I even had download pauses for 10 minutes and then it just resumes.
Interesting tho, that while onProgress pauses progressing, I don't receive any other callbacks, like
onPause(MapLoaderError error) or onResume() - the connection is stable and the download speed is over 300Mb. It just resumes the download on its own after a few minutes.
I thought it might be because of a slow or broken SD card, but I tried it with different ones - the issue is the same.
This issue increases the download speed on external storage drastically. It takes around 2 hours to download the US area, while as I said on internal storage it only takes around 35 mins. Do you have any idea why it is happening with external storage and if there is a way to fix it?
- Deleting regions.
The other issue happens to me as well on external storage only. Once you have the whole US downloaded and tries to delete some states, say California or Alaska, the DeletedRegionsCallback just won't be called or at least it takes super long to perform the delete. Even while you have only California downloaded and try to delete it, it takes 2-3 minutes to complete - it is more time than to download it.
There should be a better and more optimized way to perform file delete.
Deleting all offline maps via clearPersistentMapStorage is working fine tho. It is much faster.
Looking forward to your response and recommendations.
SDK version:
heresdk-navigate-android-4.12.2.0.9350
I find the same issue is true for the HereWeGo app, its download/delete offline maps speed is way slower with external storage.
Thanks, we have created a ticket on our end (#14289) to investigate this further. Are you sure that you are using the same network conditions for the comparison?
Yeah, pretty sure it is not a network issue. As I said I was only experiencing this issue while using external storage, once I switch to internal storage the download speed is back to normal and has no weird pauses. I've tried downloading maps on both the OfflineMaps sample project and my own fresh project to make sure it reproduces there as well - and it does.
Please let me know if you need any more details - I would be happy to help. If you don't mind, please keep me posted on how the fix is going, or at least if you were able to reproduce it on your end - we are really counting on the fix any time soon!
The issue reported here is listed in HERE mobile SDK 4.12.4 as a known issue - hopefully, the fix is coming soon!
HERE support responded that issue #1 Downloading region is addressed and going to be released in 4.12.7 - which is really good!
But we are also quite interested in the fix of the following issue. @datasun do you have any idea when it could be fixed?
#14606 and #14607 have been triaged, but are not done yet. We cannot provide a clear due date yet.
We have improved the behavior for 4.12.9.0, especially on Samsung devices download on external SD cards should work now as expected.
Awesome news @datasun, looking forward to trying it out, I will let you know how it goes.
@datasun I've been trying 4.12.9.0 and didn't notice any changes related to offline maps downloading / updating. The download speed on SD card is still slow (similar to previous versions). Checking the release notes of 4.12.9.0 the speed issue is still listed in the known issues.
What's also weird is that under Depreciation changes it specified that MapUpdater performMapUpdate and checkMapUpdate are depricated, and we should now use the updateCatalog and retrieveCatalogsUpdateInfo - but there are no such functions in 4.12.9.0
Do you have any ideas related to it? Probably the fix is in the following release version?
Yes, it seems we fixed only a part of the problem. The related bug tickets are still open on our side and will be worked on.
Regarding the deprecation changes, it seems something went wrong for the last release, but today we released 4.12.10.0 and the example app shows how to use the new methods.
The remaining issue was marked as resolved for 4.13.1.0 release, ticket for reference: #14607.
Thanks for letting me know @datasun. Will check it out once it is released.
@HERE-SDK-Support-Team This issue was reported to be fixed many times now (4.13.0, 4.13.1, 4.13.2), but still wasn't resolved. Can you please elaborate on it and provide a realistic date for the fix release?
You are referring to this resolved issue in our release notes:
"Calling sdkNativeEngine.dispose() now always shuts down ongoing MapDownloader requests.". Indeed it was falsely reported as fixed in 4.13.0 - however, it was fixed in 4.13.1.0 - it was not reported again in 4.13.2.
The ticket #14607 is also closed.
If you can still reproduce this bug, please leave a reply. Thank you.
No, I was referring to slow download / update and delete regions speed on external storage like SD cards.
This issue is still listed as a known issue and is not fixed yet in the latest release notes of Navigate SDK 4.13.2
This ticket wasn't referring to a sdkNativeEngine.dispose() at all. I think you are confused with the other ticket I creates a while ago. It was referring to the disposal issue. It was fixed and I haven't seen issues with it now.
https://github.com/heremaps/here-sdk-examples/issues/170
But what this one is all about - is a really slow speed of downloading, updating, or deleting offline map packages on SD cards. This ticket should be reopened since I can still reproduce it.
Thanks, @romandanylyk. Indeed the comment from Dec 15, 2022 was mentioning a wrong ticket number.
The ticket for the external SD card issue is #14289, but it was never reported as fixed in our release notes. And it is still open on our side.
The issue was blocked by another issue for a longer time. Unfortunately, we can still give no due date, but we have contacted the team working on the task. If we get more feedback on the details, we'll paste it here. Hope this helps.
@HERE-SDK-Support-Team please let me know how it goes, at least you guys started working on it. My team and I were waiting for it to be fixed like forever now. Any information, even minor might be helpful.
@romandanylyk Yes, we will provide an update as soon as we see progress on the ticket. Thanks for being patient and sorry for the inconvenience. This seems to be an issue that has many device-specifics, ie. Samsung vs. other vendors, so it seems to be not that easy to find a solution.
Good, let me know if you need any help testing it even on alpha builds. Our team has multiple different tablets to run it against. I found this issue reproducing even without an SD card, but just setting an offline maps path to the public directories, like "Documents" or "Downloads". So the speed is affected as soon as the cache path is outside of the app directory.
@romandanylyk We would like to narrow down the issues a bit more. At first, can you list the details about the used SD cars? Especially, on newer SD cards with good write/read speed we would not expect these issues. If possible, maybe you can give a measure on the speeds under labor situations, ie. WiFi, would be also helpful, then we can try to compare this to our measures.
And lastly, when you use internal storage, what path have you set and what countries have you downloaded with which measures?
Sorry, that we cannot get a better analysis at this point of time. At least, there have been some internal improvements for the latest releases, but it seems they did not had the desired effect.
PS: Since this issue was already moved to our customer service team and we can connect directly with you, we can close the issue here. Tickets are in place and this issue does not seem to relate directly to example apps. So, closing for now. We'll share updates directly with you. Hope this helps. If there are any concerns, please leave a reply. Thanks!
Hi, I feel like we might need to consolidate together to resolve the issue. Do you think we might communicate further in slack (heredev)?
It depends on which channel our customer service prefers. Meanwhile, regarding "reproducing even without an SD card" - can you share the details including Android version and device, measures and any other details? Thanks.
Sure, later today I will try it on my end once again, share with you the sample project and all the details.
Here are some more details about the issue. Also shared with Ramirez, Manuel via email.
Title: Offline Maps - slow download/update speed Platform: Android SDK version: Navigate 4.13.3.0 Tested devices: Samsung Tab Active 3 (Android 12), Google Pixel 6 Pro (Android 13) Download region: United States - ID 21000001 - 9.2 GB Network speed: Stable, download speed 400-500 Mbps Description:
Downloading speed for Region United States (ID 21000001) is reasonably slower while using SDKOptions persistentMapStoragePath outside of the app default path.
The speed is quite good once we don’t specify persistentMapStoragePath and it is stored in the default location - /data/user/0/<APP-PACKAGE-NAME>/
Download speed:
Google Pixel6Pro - 15min
Samsung TabActive3 - 25min
But once we set the path to the public directory “Documents” /storage/emulated/0/Documents/here-maps-cache the download speed is slowed down much.
Download speed:
Google Pixel6Pro - 50min
Samsung TabActive3 - 1hour 30min
While using external storage (SD card) path to the public directory “Documents” /storage/9D6F-12F8/Documents/here-maps-cache the drop-down in speed is even more obvious.
9D6F-12F8 - is the SD card name.
Download speed:
Google Pixel6Pro - does not support SD cards
Samsung TabActive3 - 4hours
More technical details:
Tested on HERE Github sample project OfflineMaps, slightly modified it to download the United States instead of Switz.
The speed drop could be viewed in the app logs:
CachePrefetchJob - Region prefetch progress: job_id=0, region_id=21013672, status=[460/755]
Once the download with nondefault persistentMapStoragePath is started, it is running all right for the first couple of minutes, until it reached 20-30%. Then, while observing the logs, you will start noticing that the download process might pause for some time and resumes on its own (Region prefetch progress wasn’t showing any new data, while normally it is firing roughly once per second).
So the download process is pausing on its own for a few minutes and then resumes on its own as well. And the more total download progress you have, the more often pauses are happening and the more time it needs to resume the download.
If it hadn't these weird pauses, I think that the total download time might be quite similar to the download time with default persistentMapStoragePath.
Based on my testing, pauses are happening on both internal (public directory) and external (SD card). But on the external storage pauses are happening even more often and it takes more time to resume. Since the issue is reproduced in the same way on both internal and external storage I think they are related (or might even be the same).
Downloading offline maps is not the only issue that has weird pauses. Once I tried updating the maps, it had the same issue - my guess is because the update process is making the same downloading under the hood, but with replacing the old maps once the download is done. So I think once we fix the regular download speed, it will also speed up the update process.
Note:
-
be sure that the internet connection is stable and it is not the cause of download pauses
-
The SD card I was using is San Disk A1 and San Disk A2 U3 V30 - both of them didn’t have any difference in the download speed, and both of them had download pauses described above.
-
download pauses are reproducing on all our devices, so it is not a local issue that is only affecting specific devices
-
we want the persistent offline maps to be stored outside of the app directory, so it is not removed by the system once the app is deleted
Attaching the sample project to this message - again, the project is based on HERE examples OfflineMaps. If you have any other questions, please let me know, we will be happy to help and provide any details which might help determine and resolve the issue.
https://drive.google.com/file/d/1SKF2wCg7WdTJyIfnMHU7nCu-ZADa9dKy/view?usp=sharing
@romandanylyk Thanks for the informative details. We noticed that the internal path "/storage/emulated/0/Documents/here-maps-cache" has "emulated" directory in it and we came across a few articles that mentioned that the path "/storage/emulated/0" is treated as an external storage, so that it is compatible with older apps on devices that don't have a true external storage.
It would be really helpful if you can also share the logs so that we can understand the root cause of the slow downloads.
We have an assumption that the slowness may be caused due to processing of data on our side, but we can only verify when seeing the full logs. It would be great if you can share these either here or directly through customer service channel. Thanks!
@HERE-SDK-Support-Team Are you looking for some specific logs or do you want me to take all the logs when the slow download is happening? Have you been able to reproduce it on your side?