Change logging behavior and logfile location

[Gohtraw]

I'll give it another try.

• logfiles will be placed in Documents/aapsLogs so they are easily accessible.
• there will be 24 logs per day.
• logs will be kept for 5 days (120 logs).

I've been running versions of this setup since August 2024 and it works for me. If you sync your Documents folder in some cloud, you may want to exclude aapsLogs.

[Gohtraw]

Just noticed that KeepAliveWorker was still deleting logs - that should be fixed now.

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[vanelsberg]

Personally, I have a lot of problems with this PR. I think there is no need for this change and changes may negatively affect wear watch performance.

Logs will be kept for 5 days (120 logs).

Why changing this? And why 5 days? Imho, current setting is just fine?

Documents/aapsLogs

I do not think log files should at that location. In general, user should not be bothered with app logfiles. This is about "app system" files that should be located in a location designed for that.

... easily accessible.

They are already:

1. Though maintenance menu: "send by e-mail". On Android this also lets the user copy files locally or cloud.
2. By connecting the phone using an USB cable.

[vanelsberg]

Note: This PR also changes logging behavior for the Wear App. This negatively affects the Wear app possibly creating problems.

Current settings of 7 days is intentionally to prevent the watch to run out of storage resources.

[Gohtraw]

Personally, I have a lot of problems with this PR. I think there is no need for this change and changes may negatively affect wear watch performance.

Logs will be kept for 5 days (120 logs). > Why changing this? And why 5 days? Imho, current setting is just fine?

Currently, AAPS uses size and time based rollover which is discouraged by the creators of the logging framework because it's slow and error-prone https://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy . Nobody knows how many days / hours of logs are being retained. It varies from user to user. Some people report errors in the app but by the time they're at their PC with AS, the logs are already gone.

Documents/aapsLogs > I do not think log files should at that location.

Google doesn't give us a lot of options as I explained already last August. The Downloads directory is even worse.

In general, user should not be bothered with app logfiles. This is about "app system" files that should be located in a location designed for that.

AAPS logs contain treatment data. Some people need to access their logs for further evaluation. Some people need to see why AAPS made a decision so they can adjust their settings. I've seen plenty of situations where people reported errors, got asked to provide logs and then couldn't because they were either unable to access them or the logs were already gone.

... easily accessible. They are already:

No. It's a pain to access them in Android 14 and lower and even worse from A15 on.

1. Though maintenance menu: "send by e-mail". On Android this also lets the user copy files locally or cloud.

This is not viable for people who want to evaluate their logs because they need all the data.

2. By connecting the phone using an USB cable.

This is not viable for people who want to evaluate their logs because they need all the data.

Wear stuff

I just modified that logback.xml to get rid of the broken triggering policy. Tell me what limitations should be implemented and I'll modify the PR. PS: the current configuration doesn't limit total log size, just how many days are being kept.

[Philoul]

The 2 places used in the past for log files was

• current location Android/data/...
• AAPS/logs

I think Documents folder is the last place a user will look at to find log files, these files are not "Documents".

If we want to move log files in an accessible path, then I think AAPS/logs should be used and is much better.

[Gohtraw]

The 2 places used in the past for log files was

• current location Android/data/...
• AAPS/logs

I think Documents folder is the last place a user will look at to find log files, these files are not "Documents".

If we want to move log files in an accessible path, then I think AAPS/logs should be used and is much better.

AAPS/logs would be perfect but

• in order to access that directory, user consent must be provided (that's what the set AAPS directory button in maintenance does).
• logging is one of the first things that's being started. Way before we get to the point where the aaps path is defined / queried.
• the used logback-android repo is not being maintained - it's not really compatible with the changes in Android file management.

[vanelsberg]

Summarizing my review comments on PR changes above:

1) Wear logging: By designs: do not chnage! Current values are optimized and tested for preventing storage problems on Wear Watches.

2) maintenancePlugin.deleteLogs(30): by design. Do not change! Value changed by this PR will cripple "Delete logs" functionality in AAPS Maintenance menu which was designed to clean out the logfiles independent logfile policies.

3) Logback RollingPolicies: keep existing! I have never encountered problems with current policies, nor have I seen complaints of it not working as designed. No need to change.

4) There is no need for this PR

• Current log location is by design, needed for supporting A15 and above.
• With Android 14 and above logsfiles are accessible connecting USB. General users do not need logfiles and have "send by e-mail" when logfiles are requested. WiKi explains how to access logsfiles for A13 and lower versions.
• Dev's know how to access.

5) Documents location: By Google design users "Documents" location is for documents only. Storing logfiles clutters the location with non-user files may end up in users document apps. Highly annoying) and it impacts security and access policies.

Also note: This PR would be depreciated on integrating Google Goud sync for AAPS (planned)

Additional remark: I really appreciate you trying to improve AndroidAPS. Thanks for this. But regarding the above I personally have problems with functionality, implementation and your arguments for the validity of this PR. That said, if other devs think differently: please let us know!?

[ga-zelle]

Sorry for not joining the dicussion earlier.

The principle of the PR allows realtime access to the logs from other apps. It is a hybris from Google to know better than myself which of my own data I can access on my own phone in my own 4 walls. I was really shocked when I got my new phone with Android14.

I use python scripts to scan the logs for relevant up-to-date results like "carbs required within" and if found use Text- to-Speach for announcing them. Otherwise when out on my bike with the phone hidden away deep in my pocket I only hear a short beep sound. But with the speach announcement I know how many carbs to take and when. This is much better information and not possible with permanent, asynchronous downloading of the active logfile. I had started that script years ago and even learned some python for that purpose. Meanwhile, with AI tools around, it might be in reach for me to add the Text-to-Speach capability directly in AAPS for various alerts. If someone else wants to develop that he is very welcome.

Even more importantly there is another scenario for live access to the active logfile. After starting with AAPS I wanted to understand the inner workings of the loop and ported determine_basal from javascipt to python. I ended up with the emulator which runs on a PC and takes its input from a logfile. It can also take modified settings like targets, sensitivity or any other input to determine_basal to change the loops reaction. So I can run several what-if scenarios in silico to tune my settings. Before committing the new settings I can also run that same emulator on the loop phone where it will use he active logfile and the alternative settings. That still is an in-silico setup but will inform me by Text-to-Speach if the new settings would suggest a larger SMB. Like in the early objective phases I can then inspect the scenario and decide whether and how much of that extra SMB I want to give. That way I build confidence in the new seetings before actually using them alive. Many user Mind you s of AutoISF use that emulator, too. Many IOS users of AutoISF miss that capablility because they don't have the logfiles required. Mind you that approach can be used for the SMB plugin just as well, given DynISF is disabled.

Here is an example screenshot for a test case of strengthening sensitivity by 20% listing key values for the last hour:

Although live access is the most important aspect there are other benefits:

1. The new naming and grouping by hour of the day makes it much easier to find the relevant logfile for any after the fact analysis
2. Often users ask why the loop behaved in a certain way. They may become aware of a potential problem only the next morning or when comparing to a similar event a few days ago. But when asked for a logfile it got recycled already and the case is lost.

Which folder is actually used for the logfiles is of minor importance to me. Top level AAPS was a good place, too. It may need more attention under the scoped file access policy than somwhere underneath Documents. I have not tried whether python scripts could read files in the new AAPS working folder area but I doubt it will work. It is no good alternative anyway because when users change it there is a migration gap in the logs.

[Gohtraw]

Summarizing my review comments on PR changes above:

Why is your summary longer than your original comments?

1) Wear logging: By designs: do not chnage! Current values are optimized and tested for preventing storage problems on Wear Watches.

You still haven't answered what values are desired. I'll just leave the wear logging as is in the updated PR.

2) maintenancePlugin.deleteLogs(30): by design. Do not change! Value changed by this PR will cripple "Delete logs" functionality in AAPS Maintenance menu which was designed to clean out the logfiles independent logfile policies.

That's in KeepAliveWorker. It has nothing to do with the maintenance menu.

3) Logback RollingPolicies: keep existing! I have never encountered problems with current policies, nor have I seen complaints of it not working as designed. No need to change.

The people who made logback wrote that those rolling policies are bad and should be avoided.

[Philoul]

@ga-zelle hum, I think log files are not the best solution for your use cases and looks to me as a workaround just because it's currently the only available solution. As you explain in your comment, it remains a very weak solution: a new failure can occur on every android update or on every AAPS update, each time syntax within log files change a bit for the needs of debugging or code analysis.

A dedicated plugin for external loop analysis/emulator, with all required information managed in a structured and stable way would be a much more robust solution:

• no need to filter 95% of data, only usefull data for an emulator can be exported
• data can be structured (i.e. json format) to be directly imported within your python code (you could simplified code on phone or in your computer).
• you can provide live access within phone for an emulator (like AAPS <=> xdrip sync or tizen broadcast), and history data for computer usage.

Several examples are available within AAPS to write this kind of dedicated plugin.

[ga-zelle]

@Philoul thanks for your comment. The main use of the emulator is on the PC where I investigate lots of variations on a case and on its equivalent predecessors. I don't want to perform such a series on a comparatively tiny phone display with tiny keyboard. And then for the final verification against the live loop I can use the very same python code and input files. So this is a lot less coding effort and less error prone. Changing the logfile location is the eassiet way out at the moment and I cannot see any disadvantage.

Regarding logfile content, yes, I thought about feeding it with dedicated information. At least by deselecting unnessasary content the file can be reduced considerably already now.

In the long run adding your suggested plugin may offer even more, namely re-running IOB/COB and other calculations and end up with something like Scott's backtesting for OpenAPS.

[vanelsberg]

A dedicated plugin for external loop analysis/emulator, with all required information managed in a structured and stable way would be a much more robust solution:

Fully agree with that. This could enable acces to structured info (database, events, states, data) and give option fot integration with external processes?

[ga-zelle]

When the javascript code of determine-basal was ported to Kotlin @MilosKozak had created the ReplayAPSResultTest which ran the javascript variant and the Kotlin variant for comparing the resuls. It was fed by a json-files created after each javasrcipt loop which contained exactly the input for determine-basal. The same data set can be used by the emulator.

So, if creating those json-files could be re-activated that would be a big step and be equivalent to current capabiliy based on extracting info from the logfiles.