Since getting to 1.4.1 task order changes when completing a task

[George20730]

Since updating to 1.41. the order of my open tasks keeps changing when I complete a task.

Is this a new setting I missed or a bug?

Thanks, George

[George20730]

Maybe I should add some more info :)

It happens in a note where I use the following code to get my tasks:

due today
not done
hide backlink
hide due date
hide recurrence rule
hide task count

It is really annoying especially when quickly completing a bunch of tasks as they keep jumping around making me complete wrong tasks.

I hope there is a solution George

[schemar]

Thank you for the report @George20730. Something similar was reported here: #383

I was not yet able to reproduce it. But I agree that it’s a bug that must be fixed.

[George20730]

Hi @schemar if it could help, I could send you my task related files.

[schemar]

Thank you very much, I appreciate it. For now, I'll try and reproduce it myself. I have to have a testcase covering this either way.

[therden]

Hi @schemar.

I'm wondering whether the issue reported by @George20730 -- which I've also experienced -- may be related to the algorithm underlying the "Sort by Urgency" default that was introduced in version 1.4.0 (which was released earlier on the same day as version 1.4.1).

What does Tasks use as a tie-breaker when multiple items have the same Urgency score?

(Asking because It occurs to me that if it were the file's last modified date, then that value could change the order -- relative to other tasks with the same Urgency score -- of any not yet completed tasks that are contained in the same note file as the just-completed task.)

[schemar]

It is related to that change. From the implementation, it should not alter the original behavior if everything else is the same. But somehow it does (non-deterministically). The only influencers are the ones listed in the docs here: https://schemar.github.io/obsidian-tasks/advanced/urgency/ After that, no other "tie breaker" is used. It baffles me a bit, but we'll get to the ground of this eventually :smile:

[keathmilligan]

This is happening primarily because status is included in the list of default comparators. We shouldn't sort by status unless the user explicitly requests it. Also, the way urgency was calculated yields a different value every time this could cause sort order to change depending on the initial state of the list. See PR above.

[alxgoldstn]

Right. I included urgency scores and got 1.95 on all my tasks. But changing to in_progress or scheduled moves the task.

## 5.08 Monday
- Monday thing 1
- Monday thing 2
- [ ] Monday task 1/1

## 5.09 Tuesday
- [x] Tuesday task 1/1 (DONE)

## 5.10 Wednesday
- [ ] Wednesday task 1/1

## 5.11 Thursday
- [x] Thursday task 1/4 (DONE)
- [x] Thursday task 2/4 (DONE)
- [/] Thursday task 3/4 (IN PROGRESS)
- [ ] Thursday task 4/4 (NOT DONE)

## 5.12 Friday
- [ ] Friday task 1/1

Shows up as:

tasks
not done
path includes planner/2023.05.Wk19.md
short mode
sort by heading
hide task count
show urgency

If I don't have the Thursday task 3/4 as in progress and change it back to not done [ ] then it gets the order right.

[claremacrae]

Right. I included urgency scores and got 1.95 on all my tasks. But changing to in_progress or scheduled moves the task.

~~@alxgoldstn I’m happy to help, but your question is not related to this topic.~~

~~Please start a new Q&A Discussion so we can discuss it there without spamming all the people who have contributed to this.~~

~~Thanks!~~

[claremacrae]

@alxgoldstn My apologies - I misread this as being about the 1.95 score - but on re-re-reading, see it really is about sorting.

I will have a closer look - don't re-post in Q&A....

[alxgoldstn]

Ah, true. I was meaning to highlight that the Urgency score was the same, so therefore any other sorting should only come from the rules I included. Also, thank you! You're doing massively amazing work and this plugin is mind-boggling! :)

[claremacrae]

OK I have done some experimenting and have a better understanding of how sorting actually works in practice, and it differs somewhat from the docs.

I'm struggling to write it up well in the docs, so for now I will place notes here, in case it helps anyone else....

The 5 default sorts, that are appended to every query

1. The default order is:

   sort by urgency
   sort by status
   sort by dueDate
   sort by priority
   sort by path
   

2. Those 5 lines are always appended to the end of any sort by instructions supplied by the user.

3. sort by status is a particular problem for people using custom statuses, and for searches that include done tasks - as for backwards compatibility reasons, it puts [ ] first, and all others afterwards, regardless of your custom status types.

4. So doing sort by status.type instead is more useful.

5. sort by path is important, as it prevents tasks in the most-recently edited file jumping to the end of the list after editing due to an implementation detail of how the Tasks caching code works

6. Others have pointed out that because urgency does not take account of status, this default sort order can put urgent-but-done tasks ahead of not-yet-done tasks.

What I think the default sort order should be

So, a better/safer default order would probably be:

sort by status.type
sort by urgency
sort by dueDate
sort by priority
sort by path

Ordinarily I don't like changing existing behaviour for fear of breaking existing searches, but I suspect that the current behaviour is sufficiently broken that in this case it is justified.

How to force your own sort order

If you don't like the default sort orders, your only choice is to add enough sort lines to your query that uniquely sort all tasks in to a unique order, so that the default 5 do not have any effect on the final task order.

[claremacrae]

But looking back to the original subject of this issue:

Since getting to 1.4.1 task order changes when completing a task

The reason that completing a task changes the sort order is because the status is included in the default sort ordering....

As already pointed out in https://github.com/obsidian-tasks-group/obsidian-tasks/issues/439#issuecomment-1062906678 above.

[alxgoldstn]

Wouldn't it be better to change the default so that it can be overridden? I would think that the minimum requirement should be to alter the user's tasks list as little as possible, with query options adding alterations.

Also, I'm new to the plugin, so how would changing the default sort order break existing searches? Sure, it might alter users' search result order, but they could either:

1. add in sort by status (or status.type, as proposed) to the task list
2. add in sort by status to the Global Query to get "pre-current" functionality

The biggest problem with this default is that there isn't (afaik) any way to remove the existing sort order. A cooking analogy: Why am I trying to remove salt from a meal? That's hard / impossible. Instead, just don't pre-salt the food, and let me add it in if I want.

[claremacrae]

@schemar Hallo, Any thoughts you have on todays messages would be very welcome.

[claremacrae]

Also, I'm new to the plugin, so how would changing the default sort order break existing searches?

For people who were happy with the previous default sort order for many but not all of their queries, any change in the default will require them to update all their queries where they didn't need to add any sort instructions, to now manually set an order.

So that's 'break' in the sense of:

1. Creating work for users who were previously happy
2. "There will be user complaints..." - you only have to look at the start of this ticket for an example - and user support emails == time-consuming effort for me!

[claremacrae]

Wouldn't it be better to change the default so that it can be overridden? I would think that the minimum requirement should be to alter the user's tasks list as little as possible, with query options adding alterations.

I do take the point about being able to override the defaults.

I didn't write this code - from reading it, I had always assumed from the docs that if the user provided any sort lines, then the defaults would not be used.

TIL.

So my initial inclination is:

1. Only apply the default 5 sorts if the user doesn't supply any sort instructions.
2. Make the 2 adjustments to the default sort order that I mentioned above.

One important thing though: without a rewrite of the internals, if sort by path isn't in the list, the order of tasks will change randomly for all tasks in any file that is edited. People aren't seeing that now because path is always in the list.

So maybe it should be:

1. If the user supplies any sort instructions, append only sort by path
2. Otherwise, apply the updated 5 default sorts, so minimise changes to existing tasks blocks.

[schemar]

From what I can tell and if I understand it correctly, the fix would be to make sorting stable.

I wouldn't remove the default sorting that gets appended. It is a reliable way of consistent ordering of tasks. The only issue here is the status, so I would only look at that.

Like @claremacrae already suggested, it makes sense to look at these things in my opinion:

1. Change how the status is part of the default sorting to make it more stable
2. Consider lowering the urgency for any task that is marked "done". (Maybe something like -20? It could still be interesting to order done tasks by their urgency.)

Now, I cannot really judge how status vs. status.type and the order of the default search affect the user-facing behavior. Let me know if you need more support here, @claremacrae.

Like @claremacrae said, if you remove the appended default sorting, tasks may change order with every interaction. That is a bad user experience in my opinion. Instead, if the appended default sorting would work as intended, it would never be in the user's way while at the same time producing a consistent order.

[claremacrae]

@schemar Thanks for a very insightful and helpful reply.

[claremacrae]

The upcoming 6.0.0 release has a few small breaking changes/bug fixes in it, so I'm looking at fixing the default sort order so that searches that include historical DONE and CANCELLED tasks appear lower down the task results.

Specifically, I'm changing the always-on sort order like this:

It basically means that if the user does not add any sort by, tasks will appear in the order:

• All IN_PROGRESS tasks
• Then all TODO tasks
• Then all DONE tasks
• Then all CANCELLED tasks
• Then all NON_TASK tasks

And within each of those 5 groups, tasks will be sorted by urgency, then due date, then priority, then file path.

It's a localised change - and from experimenting with it, it gives a much obviously correct sort order.

I have no idea whether it fixes the issue originally described in this ticket, as it does not have a a set of tasks that I can copy in order to reproduce the problem...

So I will mark this issue as closed when I have done the above - and if there is still a problem with unreliable sorting after the 6.0.0 release, then I will require a full reproduction is given in a new issue.

[claremacrae]

Just released in 6.0.0