gpslogger icon indicating copy to clipboard operation
gpslogger copied to clipboard

java.lang.OutOfMemoryError

Open LOVE2CMOL opened this issue 11 months ago • 21 comments

java.lang.OutOfMemoryError: Failed to allocate a 1040 byte allocation with 1422696 free bytes and 1389KB until OOM, target footprint 268435456, growth limit 268435456; giving up on allocation because <1% of heap free after GC.
	at java.util.Arrays.copyOf(Arrays.java:3585)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:177)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:691)
	at java.lang.StringBuilder.append(StringBuilder.java:242)
	at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3519)
	at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3427)
	at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3239)
	at java.io.ObjectInputStream.readString(ObjectInputStream.java:1764)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1433)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1835)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1436)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
	at androidx.work.Data.fromByteArray(Data.java:432)
	at androidx.work.impl.model.WorkSpecDao_Impl.getAllEligibleWorkSpecsForScheduling(WorkSpecDao_Impl.java:3263)
	at androidx.work.impl.Schedulers.schedule(Schedulers.java:116)
	at androidx.work.impl.Schedulers.lambda$registerRescheduling$0(Schedulers.java:76)
	at androidx.work.impl.Schedulers.$r8$lambda$UldOQUXkk9IlUfICp9vyZBPK1cE(Schedulers.java:0)
	at androidx.work.impl.Schedulers$$ExternalSyntheticLambda1.run(R8$$SyntheticClass:0)
	at androidx.work.impl.utils.SerialExecutorImpl$Task.run(SerialExecutorImpl.java:96)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1251)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
	at java.lang.Thread.run(Thread.java:1012)

But I have 4GB of free memory, which is sufficient.

Image

Image

LOVE2CMOL avatar Feb 13 '25 13:02 LOVE2CMOL

Is this happening on its own, or after you performed a certain action?

Which loggers/senders do you have activated?

mendhak avatar Feb 13 '25 18:02 mendhak

Is this happening on its own, or after you performed a certain action?

Which loggers/senders do you have activated?

This is happening automatically in the background. 🤣🤣🤣🤣🤣 The timing of it happening is quite strange. It usually occurs when I am out running with the GPS location enabled on my watch.My watch has built-in GPS. The app is called 小米运动健康(com.mi.health).

This is my configuration properties file loggers/senders should also be in it.

#Warning: This file can contain server names, passwords, email addresses and other sensitive information.
#Tue Feb 18 01:49:32 GMT+08:00 2025
log_opengts=false
stoponapplaunch=false
log_network_locations=true
autoftp_enabled=false
google_drive_enabled=false
last_version_seen=1
log_customurl_basicauth_username=
log_satellite_locations=true
log_geojson=false
autoftp_port=21
osm_tags=
log_customurl_basicauth_password=
google_drive_folder_path=GPSLogger for Android
autosend_frequency_minutes=60
current_profile_name=************************
log_customurl_url=https\://************************/api/gpslogger/************************
owncloud_directory=/gpslogger
log_customurl_discard_offline_locations_enabled=false
owncloud_password=
new_file_creation=everystart
user_specified_locale=zh-CN
retry_time=60
absolute_timeout=120
log_kml=false
autosend_frequency_whenstoppressed=false
owncloud_server=
accuracy_before_logging=60
owncloud_username=
log_nmea=false
smtp_port=25
opengts_accountname=
smtp_server=
autoftp_username=
osm_visibility=private
log_customurl_headers=Content-Type\: application/x-www-form-urlencoded
log_customurl_body=%ALL&device\=XiaoMi13&*******\=************************
altitude_subtractgeoidheight=false
autoftp_implicit=false
autoftp_directory=GPSLogger
useImperial=false
autoopengts_enabled=false
log_plain_text=true
log_plain_text_csv_delimiter=,
log_customurl_method=POST
opengts_server=
autosend_wifionly=false
distance_before_logging=2
new_file_prefix_serial=false
autoftp_ssltls=
opengts_server_communication_method=HTTPS
log_plain_text_csv_decimal_comma=false
opengts_device_id=
owncloud_enabled=false
app_theme_setting=system
autoopengts_server_path=
smtp_username=
osm_promptfordetails_when_logging_starts=false
autosend_sendzip=false
autoftp_password=
gpslogger_folder=/storage/emulated/0/Download/GPSlogger
smtp_password=
smtp_ssl=true
hide_notification_buttons=true
log_gpx=false
time_before_logging=300
startonbootup=true
log_gpx_11=true
log_customurl_enabled=true
opengts_server_port=443
autoftp_server=
osm_enabled=false
startonapplaunch=true
altitude_subtractoffset=0
autoemail_target=
new_file_custom_name=gpslogger
dropbox_enabled=false
log_passive_locations=true
latlong_display_format=DEGREES_MINUTES_SECONDS
autoftp_useftps=false
retry_get_best_possible_accuracy=false
hide_notification_from_status_bar=false
annotations_buttons=
autosend_enabled=false
new_file_custom_each_time=true
osm_description=
hide_notification_from_lock_screen=true
autoemail_enabled=false
keep_fix=false
new_file_custom_keep_changing=true

LOVE2CMOL avatar Feb 17 '25 17:02 LOVE2CMOL

So I'm a bit puzzled as to how this is happening, I think it's a bit hard to say without a debug log but I can make an educated guess.

The problem for me is the stack trace isn't pointing at a specific part of GPSLogger, but it's pointing at WorkManager, which GPSlogger does use. So it could indicate that too many WorkManager items are being queued up... it is unlikely to be large items being queued because WorkManager automatically rejects large items (I think above 10kb).

I see that passive logging is enabled, and custom URL logging enabled. I am wondering if the GPS from the watch app is so frequent that a lot of items are being queued up. But then, sending on wifi is disabled so I'm not fully confident.

Something you could try, not run custom URL logging but instead choose to use the custom URL 'auto send' feature that only sends the logs after certain intervals or when logging stops.

mendhak avatar Feb 17 '25 21:02 mendhak

Passive logging refers to obtaining data from other applications rather than from the location services triggered by the application itself, correct? I might try disabling passive logging if it doesn't affect the passive uploads when I'm using maps

Thank you for your help.

LOVE2CMOL avatar Feb 18 '25 19:02 LOVE2CMOL

So I'm a bit puzzled as to how this is happening, I think it's a bit hard to say without a debug log but I can make an educated guess.

The problem for me is the stack trace isn't pointing at a specific part of GPSLogger, but it's pointing at WorkManager, which GPSlogger does use. So it could indicate that too many WorkManager items are being queued up... it is unlikely to be large items being queued because WorkManager automatically rejects large items (I think above 10kb).

I see that passive logging is enabled, and custom URL logging enabled. I am wondering if the GPS from the watch app is so frequent that a lot of items are being queued up. But then, sending on wifi is disabled so I'm not fully confident.

Something you could try, not run custom URL logging but instead choose to use the custom URL 'auto send' feature that only sends the logs after certain intervals or when logging stops.

When I activate the sports function on my smartwatch, it tends to crash after a while. I tested it out, and even when I turned off Passive logging, it still crashed. And there is a large amount of queued data being generated. Actually, I thought about it. Could we possibly control both active and passive triggers by increasing the recording interval? Any excess data beyond this interval could then be discarded. An optional feature that allows manual setting of the time interval and the ability to turn it off.This approach might help alleviate the issue to some extent.

LOVE2CMOL avatar Feb 19 '25 16:02 LOVE2CMOL

For the passive logging, this is by design, it's just there to passively receive everything and some of the filters are deliberately ignored.

But for standard non passive logging, this behaviour shouldn't be happening. Looking at the parameters above, you've got: time_before_logging=300 which means every 5 minutes. So for regular network or satellite logs, it'll only try to log after 5 minutes. If you're seeing that it's generating logs (and therefore custom URL requests) less than that, there's some other problem at play maybe.

I'm not understanding what the relation is between the smartwatch and GPSLogger though, what does sports function mean?

mendhak avatar Feb 19 '25 20:02 mendhak

For the passive logging, this is by design, it's just there to passively receive everything and some of the filters are deliberately ignored.对于被动日志记录,这是设计使然,它只是被动接收所有内容,并且某些过滤器被故意忽略。

But for standard non passive logging, this behaviour shouldn't be happening. Looking at the parameters above, you've got: time_before_logging=300 which means every 5 minutes. So for regular network or satellite logs, it'll only try to log after 5 minutes. If you're seeing that it's generating logs (and therefore custom URL requests) less than that, there's some other problem at play maybe.但对于标准的非被动日志记录,这种行为不应该发生。查看上面的参数,您得到:time_before_logging=300,这意味着每 5 分钟一次。因此,对于常规网络或卫星日志,它只会在 5 分钟后尝试记录。如果你看到它生成的日志(以及自定义 URL 请求)少于此数量,则可能存在其他问题。

I'm not understanding what the relation is between the smartwatch and GPSLogger though, what does sports function mean?不过,我不明白智能手表和 GPSLogger 之间的关系,运动功能是什么意思?

The sports function includes GPS data.The smartwatch app generates passive logging during the sports activity. I'm not sure about the logic of the smartwatch app, but it might be generating passive GPS logging with a large amount of data during the sports activity. Additionally, even if you disable passive logging, using other mapping software may still generate a significant amount of GPS data uploads.

Image

Image

LOVE2CMOL avatar Feb 20 '25 12:02 LOVE2CMOL

I think passive logging + smartwatch is simply too much so the app is becoming overwhelmed and crashing. I don't think there is a way around it, so for your case please don't use passive logging because GPSlogger can't handle it. Unfortunately I can't recreate the issue so this is the best I can say.

But if you are saying that even with passive disabled a lot of points are being generated, then I think there might be some other bug. Is your logging interval still set to 300 (5 minutes)? At 300 you should simply not be seeing such frequent

If, even with passive disabled, passive points are still being logged, I think you could do a reset of the app to see if passive logging stops?

mendhak avatar Feb 22 '25 14:02 mendhak

I think passive logging + smartwatch is simply too much so the app is becoming overwhelmed and crashing. I don't think there is a way around it, so for your case please don't use passive logging because GPSlogger can't handle it. Unfortunately I can't recreate the issue so this is the best I can say.

But if you are saying that even with passive disabled a lot of points are being generated, then I think there might be some other bug. Is your logging interval still set to 300 (5 minutes)? At 300 you should simply not be seeing such frequent

If, even with passive disabled, passive points are still being logged, I think you could do a reset of the app to see if passive logging stops?

It's strange that after resetting the software, the passive logging issue was resolved. It seems that the crashing issue with the sports watch has not occurred recently. My app version was upgraded from a very old version, and during this time, I also went through Android system updates. I need to take another look to see if it will crash again. 😂😂😂

update: The crashing issue still exists. However, for now, disabling passive logging will no longer record data. However, this crash seems to only occur in the background or when the phone is in sleep mode.I have observed for a long time in the foreground, and it has not occurred.

LOVE2CMOL avatar Feb 23 '25 14:02 LOVE2CMOL

I also recently discovered that passive GPS data from map navigation can also cause crashes. The logs are the same. 😂😂😂 Could it be that the issue is caused by Android 15 adjusting the background queue memory? I upgraded to Android 15 on my phone this year.

LOVE2CMOL avatar Feb 26 '25 13:02 LOVE2CMOL

Sorry for late reply. If that didn't help either then I am not sure what is causing it. Yes it is possible, it could be related to too much passive logging, it could be getting overwhelmed with too many points. I do use passive logging myself, but I don't think I have a similar setup where there are too many points being issued at once. I'm also on Android 15 so maybe it isn't related to OS version.

If you want you can try to record a debug log: https://gpslogger.app/#troubleshooting

and there may be clues in there about the crash, but if the application crashes without anything recorded in the debug log, then it's likely OS related - the app isn't getting a chance to record anything about the problem .

mendhak avatar Mar 13 '25 21:03 mendhak

Sorry for late reply. If that didn't help either then I am not sure what is causing it. Yes it is possible, it could be related to too much passive logging, it could be getting overwhelmed with too many points. I do use passive logging myself, but I don't think I have a similar setup where there are too many points being issued at once. I'm also on Android 15 so maybe it isn't related to OS version.很抱歉回复晚了。如果这也没有帮助,那么我不确定是什么原因造成的。是的,这是可能的,这可能与过多的被动日志记录有关,它可能会因太多的点而不堪重负。我自己确实使用被动日志记录,但我认为我没有类似的设置,一次发布太多积分。我也在使用 Android 15,所以可能与作系统版本无关。

If you want you can try to record a debug log: https://gpslogger.app/#troubleshooting如果需要,可以尝试记录调试日志:https://gpslogger.app/#troubleshooting

and there may be clues in there about the crash, but if the application crashes without anything recorded in the debug log, then it's likely OS related - the app isn't getting a chance to record anything about the problem .并且其中可能有关于崩溃的线索,但如果应用程序崩溃而调试日志中没有记录任何内容,则很可能与作系统有关 - 应用程序没有机会记录有关问题的任何内容。

Haha, actually, I'm also completely in the dark right now.😂 Actually, I've recently noticed a particular characteristic: when data accumulates to a certain extent and the network switches, such as transitioning from a cellular network to Wi-Fi, it triggers this issue. I will try to collect some debug logs .I will supplement the information once I have collected the logs from when the crash occurs.

LOVE2CMOL avatar Mar 18 '25 14:03 LOVE2CMOL

debuglog(2).txt The crash occurred at 21:11 and 21:12. The first prompt indicated a crash, but the application continued to run in the background. The second instance resulted in a complete exit. This issue arose during the transition from WiFi to cellular data. However, upon reviewing the debug logs, it appears that not much additional information was recorded.

LOVE2CMOL avatar Mar 19 '25 13:03 LOVE2CMOL

I am able to see right at 21:11, there's a burst of network activity. A lot of network activity.

In the beginning part, I can see that GPS points are being 'captured', but not necessarily being sent to custom URL, because there are several errors related to sockets and timeouts when trying to send the data, so it waits to retry. It seems in the beginning part, the connection is unreliable, and the requests are being queued up.

Then when switching network at 21:11, the connection seems a lot more reliable and there is a burst of queued activities. That burst of activities is too much, and is overwhelming the application and causing a crash.

Unfortunately I don't know why this particular behaviour occurs, because the library I'm using called WorkManager, it is supposed to manage the concurrency and make sure the phone doesn't get overwhelmed.

You said you are going wifi to cellular, but did you mean cellular to wifi? In case your wifi is more reliable then under 'auto send' menu you can check the 'Send on Wi-Fi only' option, which should avoid cellular network. But if it's WiFi that's unreliable then that won't help.

mendhak avatar Mar 19 '25 21:03 mendhak

I am able to see right at 21:11, there's a burst of network activity. A lot of network activity.

In the beginning part, I can see that GPS points are being 'captured', but not necessarily being sent to custom URL, because there are several errors related to sockets and timeouts when trying to send the data, so it waits to retry. It seems in the beginning part, the connection is unreliable, and the requests are being queued up.

Then when switching network at 21:11, the connection seems a lot more reliable and there is a burst of queued activities. That burst of activities is too much, and is overwhelming the application and causing a crash.

Unfortunately I don't know why this particular behaviour occurs, because the library I'm using called WorkManager, it is supposed to manage the concurrency and make sure the phone doesn't get overwhelmed.

You said you are going wifi to cellular, but did you mean cellular to wifi? In case your wifi is more reliable then under 'auto send' menu you can check the 'Send on Wi-Fi only' option, which should avoid cellular network. But if it's WiFi that's unreliable then that won't help.

You make a valid point. During the transition from cellular network to WiFi, the WiFi signal was relatively weak. If the app does not handle poor network conditions well, then adopting a dual-stack approach on my end might potentially mitigate this issue to some extent. However, my phone should not have been overwhelmed at that time. I suspect that the system imposed certain resource limitations on the WorkManager, leading to GC. Actually, I'm not very familiar with the mechanisms of WorkManager. Is it possible to impose some restrictions to prevent too many tasks from being added simultaneously? Nowadays, many Android devices should default to a dual-stack mode (cellular + WiFi). It seems that this potential issue has not yet been discovered by anyone. 😁 I am now attempting to re-enable the dual-stack mode to see if the crash still occurs.

LOVE2CMOL avatar Mar 20 '25 07:03 LOVE2CMOL

Is it possible to impose some restrictions to prevent too many tasks from being added simultaneously?

That's what I'm trying to figure out, but even when I go straight to the WorkManager source code, the 'concurrency' is already very low. This is right from Android's codebase you can see it is between 2 and 4, that's it.

https://android.googlesource.com/platform/frameworks/support/+/androidx-1.0-dev/work/workmanager/src/main/java/androidx/work/Configuration.java#120

To help with connectivity issues, WorkManager can be told to only do the work when there's any unmetered connection, or when there's wifi. That has been configured here.

And the only other thing I can do really is to say 'retry' when there's a failure. I've set the retry to 3 here.

There is unfortunately not much else that the library gives in terms of configuration.

mendhak avatar Mar 20 '25 08:03 mendhak

The dual-stack solution experiment failed because once WiFi is connected, it checks whether the network is normal before selecting the network. However, at this point, I infer that the WiFi is in an extremely poor state, such as being on the edge of the coverage area. As I walk all the way home, the network happens to be in a poor state, but the test network packets can still be sent and received normally. At this time, the system will judge that the WiFi is normal and will not use cellular data, leaving GPSLogger still in a state of network congestion. I am preparing to extract the system logs to see if there are any other findings. However, my system logs are only retained for about 1-2 hours, so I will have to wait until I have time to collect them.

To help with connectivity issues, WorkManager can be told to only do the work when there's any unmetered connection, or when there's wifi. That has been configured here.

Additionally, is the shouldAutoSendOnWifiOnly option effective for a single message, or does it apply to the entire file upload? I noticed that the option is placed within the auto-send settings.

That's what I'm trying to figure out, but even when I go straight to the WorkManager source code, the 'concurrency' is already very low. This is right from Android's codebase you can see it is between 2 and 4, that's it.

I actually have a thought: could it be an issue with WorkManager's thread management itself, not releasing resources correctly? If that's the case, then the whole problem becomes much more troublesome.

Looking at the crash logs, my understanding is that the maximum memory allocated to WorkManager is 256MB, and then an Out Of Memory (OOM) error was triggered.

LOVE2CMOL avatar Mar 20 '25 17:03 LOVE2CMOL

Yeah the placement of 'shouldAutoSendOnWifiOnly' is misleading but it should apply to all sending types of network activity including custom URL logging.

So at this point I'm not sure how to solve it. One thing you could consider is instead of custom URL logging you could do the custom URL sending. That is, instead of a custom URL request for each GPS point, use the auto-sending feature which will write to a CSV file and every 60 minutes by default (can be changed) send all the stored points to your custom URL.

It should be fewer queued up items. It will resend all the points each time, so your receiving server needs to be able to handle duplicates. It should also result in just one WorkerManager for each 'send' because that single worker will try to read from CSV and send all the points one after the other.

mendhak avatar Mar 20 '25 21:03 mendhak

systemlog logs2.txt The crash occurred at 20:56.

It should be fewer queued up items. It will resend all the points each time, so your receiving server needs to be able to handle duplicates. It should also result in just one WorkerManager for each 'send' because that single worker will try to read from CSV and send all the points one after the other.

I am using InfluxDB here, and it is quite troublesome to handle the data that is inserted midway.The need to determine data duplication complicates the code significantly.On my backend, the data submitted by the app is so dense that more than 60% of it is filtered out before being stored in the database.

I have also identified another issue: the rate at which the queue is being submitted is not keeping pace with the rate at which data is being generated. This discrepancy causes a significant backlog of data waiting to be submitted, thereby impacting the real-time nature of the data. The submission process only gradually completes after the GPS mapping software is closed, which also includes the real-time data from that period.Speaking of which, it seems I forgot to set a larger distance threshold in the app.Perhaps it could also alleviate the issue of excessive data.🤣

upate 2025.3.23 I've discovered a new issue: when I attempt to clear the CSV and force quit the app, upon reopening the app, I find that the queue is still running, and there may be a significant number of tasks waiting to be executed. I'm uncertain whether these queued tasks could potentially cause GC. The newly generated data has not been submitted in real-time, possibly because it is queued up in the queue, which affects the real-time performance. I have set a larger distance threshold, yet the queue speed still cannot catch up with the rate of data generation.

LOVE2CMOL avatar Mar 22 '25 13:03 LOVE2CMOL

I'm wondering if real-time data can be submitted directly without going through a queue. I've noticed that when using WorkManager for submission, it only processes tasks sequentially. I need real-time data handling, but queue-based submission introduces delays of several seconds between each submission. Typically, GPS generates multiple data points per second, causing further backlog. I'm not sure why it sometimes waits before sending the next batch.

The most frustrating issue I'm facing right now is that GPS data isn't updating in real time.I've also tried discarding offline data, but it still didn't solve the problem. Image

LOVE2CMOL avatar Apr 04 '25 13:04 LOVE2CMOL

Yeah the placement of 'shouldAutoSendOnWifiOnly' is misleading but it should apply to all sending types of network activity including custom URL logging.

So at this point I'm not sure how to solve it. One thing you could consider is instead of custom URL logging you could do the custom URL sending. That is, instead of a custom URL request for each GPS point, use the auto-sending feature which will write to a CSV file and every 60 minutes by default (can be changed) send all the stored points to your custom URL.

It should be fewer queued up items. It will resend all the points each time, so your receiving server needs to be able to handle duplicates. It should also result in just one WorkerManager for each 'send' because that single worker will try to read from CSV and send all the points one after the other.

The OOM (Out-of-Memory) issue seems resolved. Below are my observations and hypotheses (which might be incorrect): The DNS servers for the local network (LAN) and wide-area network (WAN) point to different backend servers. While my architecture supports multi-endpoint access, I suspect that WorkManager’s internal mechanism—or a similar factor—causes problems during network handover scenarios. When a device just connects to WiFi, the LAN’s DNS resolves to a local server and feeds the response back to the client. However, if the device is still in a network transition state (e.g., switching between cellular/WiFi), some tasks might enter an abnormal state, leading to backlogged requests and eventual resource exhaustion.

HA HA 😀Perhaps Android 15 introduced a new bug? I was able to reproduce this issue on another Android 15 device as well.

I’ve looked into WorkManager’s mechanics: Its tasks are persisted in SQLite, and concurrency is strictly limited. Under normal circumstances, this design should prevent OOM errors.

I conducted a test by changing the LAN DNS host IP to the WAN DNS host IP. As a result, no OOM errors occurred.

Additionally, I've considered another potential factor: WorkManager's retry mechanism. During retry operations, it appears to generate excessive concurrent data loads, which can also trigger OOM conditions. Similar to image processing scenarios, the system generates massive data volumes until triggering OOM errors. Image

😂😂😂😂😂After nearly two months of investigating this issue, I've also uncovered several other related problems along the way. 😀

Thank you once again for developing such an incredibly useful software.👍

LOVE2CMOL avatar Apr 05 '25 13:04 LOVE2CMOL

This issue has been mitigated through an alternative solution implemented in v134. Marking as closed.

LOVE2CMOL avatar Jul 08 '25 12:07 LOVE2CMOL