Investigate whether Tasks plugin affects editing speed in large vaults - Answer: it does, and will be worked on soon.

[claremacrae]

In conversation with @schemar I learned that the way that Tasks tracks edits to Obsidian notes is to reload all files. And that has been fast enough.

There have been some anecdotal comments that maybe Obsidian with the Tasks plugin uses more CPU - and that editing can be laggy.

This issue is to investigate if performance or CPU is a problem, and if it is, then to resolve the issue.

Possible Solution

Refactor the code so that when a note is edited, only that note is re-read, and not the whole vault.

This will need to be sure detect edits made in external editors.

[claremacrae]

Example observation from @elight in Discord

Is it just me or does Tasks eat up a fair amount of CPU? I'm noticing more lag in my data entry in my vault, since enabling it. I'll check more tomorrow AM when I'm on my Mac as opposed to my iPad.

[claremacrae]

From me:

I think it may not be just you, I think it may be worse on iPad and iPhone than on Mac My efforts to get a solid repro have not succeeded, hence wanting to start logging their setups, what plugins they have enabled etc.

TODO Add a note about the size of my vaults - numbers of files and tasks

I must admit that I have not yet thoroughly tested it with only the Tasks plugin enabled, and I'm so dependent on CustomJS as well.

[claremacrae]

@schemar I hope I've represented correctly what you said. If you have a mo, any info about the relevant bits of code would really be appreciated!

[schemar]

Thank you for tracking this, @claremacrae! The details a bit different, though. When a note changes, the cache only refreshes the tasks from that note, not all notes.

Here, the cache subscribes to the vault changes on a per file level and calls indexFile for changes: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L115

Here, the cache clears all tasks from that file before re-processing the file: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L221

However, what is currently non-optimized are propagation of changes to the cache's subscribers. For example, each query view subscribes to the cache to be informed of changes to update the view when tasks change. The Cache::notifySubscribers() method always sends all tasks to all subscribers: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L93

Potential optimization: Only send the tasks of the changed note. For that, all subscribers would need a map of note->tasks to update only partially.

[claremacrae]

Oooh, the internal label is very nice. Thank you.

And thank you for all the info, @schemar!

[elight]

Confirmed this morning that I don't notice any lag on data entry/manipulation on my i7 Mac mini but significant data entry/manipulation lag on my M1 iPad Pro. I'll experiment further on my iPad to attempt to determine if it is in fact the Task plugin or something else. Early signs point toward my recent introduction of the Task plugin to my Vault.

[elight]

And thanks again for reporting, @claremacrae !

[YannickTemplay]

Hello, Tasks (1.6.0) make my quite large vault (21k notes) almost unusable with lags of 5 to 10 seconds in the middle of edits. If I disable tasks then the performance switch back to almost instantaneous. I am on an Apple M1 with 16MB of Ram...

I am not sure how many tasks I have but if counting "dones" then quite a lot.

Maybe you could add an option to the tasks queries are only updated once when a page loads or something ?

The tasks plugin is central to my Obsidian usage but right now I am down to disabling it and reenabling it every time I need to see my tasks :-(

Thanks for any help !

[claremacrae]

@YannickTemplay Sorry to hear that. What version of Tasks were you on previously, and what other plugins do you have enabled?

[claremacrae]

@YannickTemplay any extra info that you can supply would be really helpful. When you say M1 I presume you mean a laptop rather than iPad?

For example:

is it all edits, or edits in files with tasks?

Does it differ if you are Live Preview or Source mode?

Is it worse if your edits are further from the start of the file?

[YannickTemplay]

Yes, Mac M1

All notes (but I think its much worse on notes with tasks queries)

See-> https://www.loom.com/share/c92f3a4b6339451d9ca62b0241d21ff9

[aubreyz]

Yes, Mac M1

All notes (but I think its much worse on notes with tasks queries)

See-> https://www.loom.com/share/c92f3a4b6339451d9ca62b0241d21ff9

Are any of the queries on the page dataview queries on tasks?

I had the same problem with task queries but more prominently with dataview ones. One thing I did not understand was why it had nothing to do with the mode (edit, live preview, reading). So two observations:

a) Why would searches ever refresh (for either plugin) when in edit mode -- but I can confirm that this definitely seemed to be happening). Only one pane open.

b) It seemed to happen when the output of the query was large (i.e. queries that produced many hundreds of hits that had to be rendered), not anything do do with the complexity of the query or the quantity of underlying data (bizarrely even if the result was not actually displayed in HTML). Whatever the case, at least in my hands, the intermittent severe lags on editing are a function of the page contents and not the vault itself.

I cannot see why tasks (or dataview for that matter) should be trying to process anything at all ever except when exiting a page (either by going to another page, or going from edit mode to preview). Arguably it should even do no-processing in live preview. It should certainly not do anything at all at each and every autosave of a file while editing things.

I'm on Windows 10

[YannickTemplay]

The video I sent you has no queries at all. Just a plain new note. And no other notes were opened simultaneously.

Of course, the lags are much worse on pages with a query or multiple queries, but the lag is still there on pages without queries so I think there is indeed a problem with some "refresh" that takes place in the background unnecessarily.

Typical queries I use on my daily note page look like that...

tasks
not done
due on {{date:YYYY-MM-DD}}
group by filename

I often also use in projet notes queries like this...

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

[YannickTemplay]

Could the problem come from the dataview Javascript queries?

I copied this query from Excalidraw Zsolt.

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

In his example, I think he mentioned that he put a comment with a "random number" in every query. I did not understood why so I did not do that. But maybe it's related to dataview refresh or something?

But again those dataview JS queries are not on those empty new pages so I don't understand why it would slow things down and when I turn off TASKS is works again while I keep DATAVIEW active... so it's not that I think.

[aubreyz]

To add to my last message -- the page that caused the most serious problems in my hands -- with editing lags in pure edit mode: Was a page of examples I use which has a host of (dozens of) personal dataview and tasks examples I have collected. This file is almost un-editable (but can be easily previewed/rendered).

I know this sounds bizarre, but there is something fishy going on under the hood.

[claremacrae]

Ah right, knowing that you also have dataview enabled, it would be good to understand if there is any interaction between your dataview and tasks setups.

Is there any chance you could disable dataview and restart Obsidian, and explore the behaviour, please?

[YannickTemplay]

Yes, the lags seem to come when both TASKS and DATAVIEW are enabled together...

Just TASKS = OK

Just DATAVIEW = OK

TASKS + DATAVIEW = lags

[claremacrae]

This is all tremendously helpful!

As much as possible, it would be really good to get approximate measures of timings with just dataview and tasks enabled an nothing else.

(Someone mentioned Excalidraw too. The number of potential interactions between plugins goes up exponentially with the number of plugins enabled.)

[aubreyz]

OK I should qualify my previous comments. The previous file I had containing a large amount of example dataview and task searches, and which (6 weeks ago) produced massive lags in pure edit mode, no longer does that. So perhaps some update to core obsidian has fixed it for me. I also use excalidraw occasionally (not in these pages though).

The page does still have massive problems on editing in live preview mode (much less of a problem, and perhaps understandable, but probably unnecessary). It is not clear why every search on a page should be re-done every few characters of typing if text that is being typed is completely outside of any search.

[claremacrae]

Thanks @aubreyz

So perhaps some update to core obsidian has fixed it for me.

Perhaps., but I'm not sure ... Every time I thought I had a good reproduction of the problem, and restarted Obsidian to verify it, the problem would not occur. And then some time later (hours or so) it would suddenly be slow again.

As a developer, things failing intermittently makes it almost impossible to work reliably on the bug, because you never know whether any individual change you made fixed the problem, if you have to keep testing for multiple hours to even be sure you really fixed it.

I also use excalidraw occasionally (not in these pages though).

Do you mean you turn excalidraw off when you are not using it?

The thing is that some plugins may do work behind the scenes, even if they are not used on the particular page you are editing.... Hence my request for testing to be done with other plugins turned off...

The page does still have massive problems on editing in live preview mode (much less of a problem, and perhaps understandable, but probably unnecessary). It is not clear why every search on a page should be re-done every few characters of typing if text that is being typed is completely outside of any search.

See the comment https://github.com/schemar/obsidian-tasks/issues/697#issuecomment-1140455291 above, from the author of this plugin, for comments about that.

Once we get to a reliable reproduction of this issue on a development machine, we can look at optimising that code.

[aubreyz]

I also use excalidraw occasionally (not in these pages though).

Do you mean you turn excalidraw off when you are not using it?

No I don't switch it off. But no point trying that now, unless I can reproduce the issue again on editing (not on rendering)

[aubreyz]

I think a related issue is the autosave interval in the obsidian core. This is (as far as I know) fixed at about two seconds, and is not changeable.

So if something is triggered on a save during editing, this is massively magnified. It was fairly obvious before when I was having files causing these problems, that editing lag was cycling at about that frequency -- so if I just typed xxxxxxxxxxxxxxxxxxxxxx I got about two seconds of typing in, before having to wait about 10 seconds.

It also creates problems for file versioning (by say Dropbox) when there are just thousands of versions.

It would be quite nice if the Obsidian developers allowed this to be altered within some sensible limits (say up to 30 seconds or when the pane was exited or previewed). This would also help to debug these types of issues because the timing aspect would become more clear.

[elight]

My steps to replicate:

1. Open Obsidian on my M1 iPad Pro.
2. Ensure DataView and Tasks are enabled
3. Open my Daily Note (Contains several Tasks queries and a DataView query)
4. Begin typing into the note,
5. Switch off DV
6. Repeat Steps 1-4 
7. Switch on DV
8. Switch off Tasks
9. Repeat Steps 1-4
10. Switch on Tasks
11. Open a new note with no queries
12. Type into the new note.

Results

Steps	Observations
Steps 1-4	Input lag was noticed. After typing a few characters, input seemed to buffer, rendering approximately every 2 seconds.
Steps 5-6	Same lag was observed.
Steps 7-9	No lag was observed
Steps 10-12	No lag was observed.

This lag has a significant negative impact to usability on iPadOS. I haven't interacted with the iOS version enough lately to comment.

[elight]

I think a related issue is the autosave interval in the obsidian core. This is (as far as I know) fixed at about two seconds, and is not changeable. ...

The observed 2 seconds aligns with my observations on iPadOS.

[YannickTemplay]

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

[elight]

I do use dataviewJS but I don't on that particular note, just DV's query language.

On May 31, 2022, Vipercat @.***> wrote:

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs const name = dv.current().file.name; dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

— Reply to this email directly, view it on GitHub <https://github.com/schemar/obsidian-tasks/issues/697#issuecomment- 1141707348>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAACPAC3S3AM5HLABHHCASTVMWU55ANCNFSM5XHYPSOQ>. You are receiving this because you were mentioned.Message ID: @.***>

[elight]

That is, I'm not doing anything that lends itself to tying up an event loop unusually.

On May 31, 2022, Evan Light @.***> wrote:

I do use dataviewJS but I don't on that particular note, just DV's query language.

On May 31, 2022, Vipercat @.***> wrote:

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs const name = dv.current().file.name; dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

— Reply to this email directly, view it on GitHub <https://github.com/schemar/obsidian-tasks/issues/697#issuecomment- 1141707348>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAACPAC3S3AM5HLABHHCASTVMWU55ANCNFSM5XHYPSOQ>. You are receiving this because you were mentioned.Message ID: @.***>

[YannickTemplay]

I just exchange messages with Zsolt, the author of Excalidraw about the issue he had with Dataview’js but it seems unrelated to our problem.

[YannickTemplay]

His problem was that if there is multiple exact same Dataview JS queries in a vault, sometimes it would not update properly.

I definitely have the same Dataview JS query multiple times in my vault and it I don’t have any update problems.

[YannickTemplay]

His solution was to add a random number as a comment in each JS queries to make them “different”.