obsidian-livesync FileWatching STORAGE -> DB (plain) updates lead to lag on mobile devices like iPad

Abstract

Local updates to PouchDB cause a lag to Obsidian each time. Whilst this is barely noticeable on a PC, on e.g. an iPad this leads to unresponsiveness of the application ~~and duplication of characters entered~~ at the moment DB writes kick in - in my case to an extend that renders livesync unsuitable for note updates from my (6th gen) iPad. I tried to disable all sync (via the command "Toggle All Sync") but this option only seems to affect replication between PouchDB and CouchDB, not chunk collection from local store into the local PouchDB. I failed to identify a way to suspend this process, even when all replication is suspended.

Expected behaviour

Filewatching/Writes to local PouchDB should not impact user experience (lag/responsiveness)
~~Filewatching/Writes to local PouchDB should not impact the usability of the application (duplication of characters entered)~~
Plugin activity should be minimised to what is actually needed to enable the synchronisation capability. In a use case other than "LiveSync" I'd assume there is no need for constant FileWatching for as long as no replication event is triggered. I'd assume there is only a need to keep the local DB up-to-date with local storage right before a synchronisation takes place (but pls. do correct me in case I am wrong here).

Actually happened

Updates to local PouchDB take place even with synchronisation disabled
Increasing the Batch size in settings seems to redruce the number of updates, yet not to an extend that would remedy the impact on user experience and usability
Activating "suspendFileWatching" remedies the issue, but in line with the purpose of this parameter, renders the replication capability of the plugin partially defunct

Reproducing procedure

Configure LiveSync as in the attached material - all automated replication off.
On an iPad to update an existing note in a suffuciently sized vault (2000+ notes) to experience lag.
To verify the cause for 2. watch plugin notifications "STORAGE -> DB", than activate "Suspend FileWatching" to verify the usability issues disappear.

Report materials

N.A.

Report from the LiveSync

For more information, please refer to Making the report.

Report from hatch

---- Obsidian info ----
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.4.13 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36
---- remote config ----
cors:
  credentials: "true"
  headers: accept, authorization, content-type, origin, referer
  max_age: "3600"
  methods: GET,PUT,POST,HEAD,DELETE
  origins: app://obsidian.md,capacitor://localhost,http://localhost
chttpd:
  bind_address: any
  enable_cors: "true"
  max_http_request_size: "4294967296"
  port: "5984"
  require_valid_user: "true"
admins: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
vendor:
  name: The Apache Software Foundation
feature_flags:
  partitioned||*: "true"
chttpd_auth:
  authentication_redirect: /_utils/session.html
  hash_algorithms: sha256, sha
  require_valid_user: "true"
indexers:
  couch_mrview: "true"
prometheus:
  additional_port: "false"
  bind_address: 127.0.0.1
  port: "17986"
httpd:
  WWW-Authenticate: Basic realm="couchdb"
  bind_address: 0.0.0.0
  enable_cors: "true"
  port: "5986"
smoosh:
  state_dir: ./data
couch_httpd_auth:
  authentication_db: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  secret: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  authentication_redirect: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchdb_engines:
  couch: couch_bt_engine
couchdb:
  database_dir: ./data
  max_document_size: "50000000"
  single_node: "true"
  uuid: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  view_index_dir: ./data

---- Plug-in config ---
version:0.22.16
couchDB_URI: self-hosted(HTTP)
couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
liveSync: false
syncOnSave: false
syncOnStart: false
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: false
suspendFileWatching: false
trashInsteadDelete: true
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: false
encrypt: true
passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
usePathObfuscation: false
doNotDeleteFolder: true
resolveConflictsByNewerFile: false
batchSave: true
deviceAndVaultName: ""
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: true
showStatusOnStatusbar: true
showOnlyIconsOnEditor: false
usePluginSync: false
autoSweepPlugins: false
autoSweepPluginsPeriodic: false
notifyPluginOrSettingUpdated: false
checkIntegrityOnSave: false
batch_size: 50
batches_limit: 25
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
showMergeDialogOnlyOnActive: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/,\/workspace$ ,\/workspace.json$,\/workspace-mobile.json$
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: 38e762a543fe78f5
ignoreVersionCheck: false
lastReadUpdates: 0
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 50
readChunksOnline: true
watchInternalFileChanges: true
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: false
configPassphraseStore: ""
encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
permitEmptyPassphrase: false
useIndexedDBAdapter: true
useTimeouts: false
writeLogToTheFile: false
doNotPaceReplication: false
hashCacheMaxCount: 300
hashCacheMaxAmount: 50
concurrencyOfReadChunksOnline: 30
minimumIntervalOfReadChunksOnline: 25
hashAlg: xxhash64
suspendParseReplicationResult: false
doNotSuspendOnFetching: false
useIgnoreFiles: false
ignoreFiles: .gitignore
syncOnEditorSave: false
pluginSyncExtendedSetting: {}
syncMaxSizeInMB: 50
settingSyncFile: ""
writeCredentialsForSettingSync: false
notifyAllSettingSyncFile: false
isConfigured: true
settingVersion: 0

Obsidian debug info

Debug info

N.A.

Plug-in log

We can see the log by tapping the Document box icon. If you noticed something suspicious, please let me know. Note: Please enable Verbose Log. For detail, refer to Logging, please.

Plug-in log

N.A.

Network log

N.A.

Screenshots

If applicable, please add screenshots to help explain your problem.

Other information, insights and intuition.

Suggestion: a) For all replication methods other than LiveSync, suspend FileWatching until a replication is triggered. a) Only update the local PouchDB when an actual replication is triggered, rigth before the actual synchronisation with the remote DB takes place ... or - in case this impeeds some other use cases/capabilities of the plugin ... b) Provide (a) configuration option(s) to manage the behaviour of local database operations, so that the behaviour suggested in a) (no local updates until a replication is about to start) can be configured

Apologies in advance if I just failed to understand how to properly configure livesync for my use case. Please do not hesitate to reach out for any related queries - or in case you'd like support with documentation.

Thank you

06/04 - removed character duplication aspect - turns out this was owed to a separate issue.

Apr 05 '24 16:04 czenk

Thank you for opening the issue and detailed information!

If the file changes are not watched, we have to scan the files before synchronisation to see which files have actually been changed. This also can be a bit costly process with 2000 files.

a) Only update the local PouchDB when an actual replication is triggered, rigth before the actual synchronisation with the remote DB takes place ... or - in case this impeeds some other use cases/capabilities of the plugin ...

Actually, this is the Batch database update (And sorry, not related to the term Batch of Batch size in synchronisation. I regret to name it so misleading). Aeroplane icons indicate files which are being pended to read. This has got quite less cost since v0.22.16. Could you please check when the lag has happened? -- with an aeroplane, or with an hourglass?

And, to check one more possibility. Would you mind trying file editing by disabling all status icons in Appearance of General Settings?

Apr 09 '24 06:04 vrtmrz

Thank you so much for looking into my issue and for taking the time to respond. Incorporating your advise and explanations I will do some extended testing and report back once done.

Apr 09 '24 16:04 czenk

Hi, Update on my experience regarding STORAGE -> DB activity in "batch" mode:

In this mode on Windows, the logs show updates from storage to DB when the changed note loses focus (e.g. when switching to another tab)
In this mode on iPad, the logs show updates from storage to DB every 5-10 seconds when the note gets changed. During test on the iPad all status indicators were switched off.

The behaviour from 2. is undesired as it a) causes unnecessary battery drain, b) needs compute power which can cause lag depending on the load of the mobile device, vault configuration and vault size. Is the problem "in front of the keyboard" (<-which would be me not configuring your plugin correctly ;-) or is your plugin behaving like this by design? Do feel free to let me know with what additional information I can assist in tracking down the issue (I can imagine that you may need more info to work with).

Thank you very much for your efforts and for providing the community with a plugin that has the potential to extend Obsidian to business cases that require an alternative option for data governance.

Apr 15 '24 10:04 czenk

Thank you very much for the new release 0.22.17. Just did some testing against it. Unfortunately the situation has not improved. Both on Windows and iPad the 'STORAGE -> DB' activity can bee seen every few seconds when a note is changed. In the hope that it may help, I am attaching a new report and screenshot. livesync_report_0.22.17_20240416.txt Livesync_batch_UI Thank you and please do not hesitate to reach out in case I can be of further assistance. Kind regards

Apr 16 '24 08:04 czenk

Sorry for being late! Unfortunately, that behaviour also seems to occur in the correct settings.

To improve the performance in this situation, I have implemented a new feature, Incubate Chunks in Document. When enabled, the newborn chunks are created inside the document and transferred with the document. Therefore, we can avoid transferring the meaningless chunk. And, in the recent version, the minor bug of using cached chunks has been also fixed.

Would you mind if I ask you to check the latest version with enabling Incubate Chunks in Document, please?

May 27 '24 09:05 vrtmrz

Hi,

Thank you for your efforts and for getting back to me.

You wrote "newborn chunks are created inside the document" - care to elaborate? I assume you are not manipulating the actual .md file of the note?

I spent approx. three hours with the latest version of your plugin so far.

Config at the end of my first test session: Incubation chunks - on Batch update - on Sync Mode - on events User trash bin - on Keep empty folder - on Hidden file synchronization - Enabled Scan hidden files periodically - 0

Experience:

The logs seem to stay quiet on my iPad now - until I edit a document at least: As soon as I udpate a note I am getting a STORAGE -> DB (plain) entry in the log.
Scenario: Database name configured = 'livesyncV0': Initial DB connectivity test to the new database claimed successful, but synchronisation failed with a message that only lower case letters and numbers are allowed in the database name. Did a new remote db (couchdb on Linux) setup to get it working. (Windows 10)
Option "Discard local database to reset or uninstall Self-hosted LiveSync" now does an immediate rebuild of the local database instead of deleting it only (tried several times). This is according to the livesync log and verified by watching HD space consumption. (Windows 10)
The option "Scan Hidden files sync" does seem to ignore the option "Sync Mode" being set to to 'On events'. "Self-hosted LiveSync Log" continues to report hidden files scans every 60 seconds. Now, this might be me not understanding the intended use of, and dependencies between options correctly - I assume thought that hidden files will not be synced at all if I set "Scan hidden files peridoic" to 0? (Windows 10)
Option "Discard existing settings and databases" did not purge database settings from the settings dialog. (iPad)

All in all I must admit that I am getting increasingly confused about configuration options, their names and their dependencies. This might be just me. If on the other hand you'd like some assistance in terms of documentation, naming, arrangement of configuration options in the settings dialog, do let me know. I can't make any promises, but I maybe I am able to provide some kind of support.

Finally some statistics (Windows 10): Disk space usage of vault: 504MB Disk space usage for local db after initial built: 1.17GB Vault contents: 2636 files, 151 folders No. of docs reported in remote couchdb after first replication: 12976 Couchdb size: 0.8GB

I will not be able to conduct further testing until at least the weekend, but I will let you know once I should have more updated for you.

In the meantime hope this information is of some use to you. Please do not hesitate to let me know in case you need further details.

Cheers

May 28 '24 16:05 czenk