Sort entries by due date when using org-ql-block

[jakejx]

Hello, thank you for this wonderful package, it makes my custom agendas useable again!

I am in the process of porting some of my custom agenda commands I was wondering whether it was possible to sort my results based on the deadline. I tried using the conventional way of setting org-agenda-sorting-strategy, but I don't think it works. Here is what my agenda command looks like

(org-ql-block '(and (tags "test")
                    (not (tags "CANCELLED" "HOLD"))
                    (not (scheduled))
                    (or (or (todo "NEXT") (todo "PROG"))
                        (deadline 3)))
              ((org-ql-block-header "TEST")
                (org-agenda-sorting-strategy '(deadline-up))))

Is setting the sorting strategy like this supported? Thanks!

[alphapapa]

Thanks for the kind words. I'm glad it's helpful to you.

Actually, there is no way to do that at the moment. That's an oversight on my part. I'll need to add additional arguments to the block function so :sort can be specified, like in org-ql-search. Thanks for letting me know!

[yantar92]

Seconding the request

[jakejx]

Thanks for the kind words. I'm glad it's helpful to you.

Actually, there is no way to do that at the moment. That's an oversight on my part. I'll need to add additional arguments to the block function so :sort can be specified, like in org-ql-search. Thanks for letting me know!

No apologies needed! I will try to take a look at the code too 😄

[akirak]

I'll need to add additional arguments to the block function so :sort can be specified, like in org-ql-search.

Or maybe you can support the feature by adding a variable for a sorting strategy, like org-ql-block-header does for the header. I personally don't want to contain something other than a predicate in the second argument, so I prefer this way.

[alphapapa]

@akirak That's a good point.

What do the rest of you think?

[yantar92]

What do the rest of you think?

I feel that it would be better to stick to built-in org-mode functions as much as possible. Otherwise, you may end up reimplementing the whole org-agenda.el from scratch. I believe that only the most performance-critical functions from org-agenda need to be reimplemented.

[alphapapa]

@yantar92 The question here is only how to present the sorting argument in the block definition, not the implementation.

In fact, org-ql is essentially a reimplementation of the agenda--that's the point, because org-agenda.el is very difficult to understand and modify, and nearly impossible to extend.

[jakejx]

I am ambivalent to either solution. But of course whichever method is easier to leverage upon the built in org mode functionality should be preferred, to make implementation easier.

[alphapapa]

@jakejx Sorting is already implemented in org-ql. org-agenda sorting functions are not used. (If you're curious as to why, look at the code for the function org-entries-lessp. @akirak you might find it amusing.)

The question here is whether to pass the sort argument like this, which is consistent with org-ql functions:

(setq org-agenda-custom-commands
      '(("ces" "Custom: Agenda and Emacs SOMEDAY [#A] items"
         ((org-ql-block ('(and (todo "SOMEDAY")
                               (tags "Emacs")
                               (priority "A"))
                         :sort '(date priority todo)))
          (agenda)))))

or like this, which is more like org-agenda block settings:

(setq org-agenda-custom-commands
      '(("ces" "Custom: Agenda and Emacs SOMEDAY [#A] items"
         ((org-ql-block '(and (todo "SOMEDAY")
                              (tags "Emacs")
                              (priority "A"))
                        ((org-ql-block-sort '(date priority todo))))
          (agenda)))))

I'm generally in favor of the first example, because it's consistent with org-ql's API, and it doesn't require an extra binding form in the block, which makes the list structure much simpler.

[akirak]

@alphapapa

(org-ql-block ('(and (todo "SOMEDAY")
                               (tags "Emacs")
                               (priority "A"))
                         :sort '(date priority todo)))

This looks better than I expected. Actually, I now even prefer it over the other, because it allows easy migration to/from normal org-ql forms and requires less memorization of variable names.

If you're curious as to why, look at the code for the function org-entries-lessp. @akirak you might find it amusing.

Thanks, I'll take a look at it.

@yantar92

I feel that it would be better to stick to built-in org-mode functions as much as possible. Otherwise, you may end up reimplementing the whole org-agenda.el from scratch.

org-ql started out as a library for org-agenda-ng, which was an experiment by alphapapa to improve upon the API of org-agenda. org-agenda-ng was merged into org-ql. As he develops org-ql, it is becoming a practical alternative to org-agenda.

[yantar92]

In a view on org-ql as a replacement to org-agenda, it indeed makes sense to stick to org-ql syntax rather than retaining the old org-agenda syntax. Mixing up the syntax will make things confusing.

Also, replacing org-ql-block-header with :header will make sense then.

[alphapapa]

I looked into making sorting in org-ql-block blocks sort according to org-agenda-sorting-strategy, but it doesn't appear to be possible because of the difference in how Org Agenda and org-ql are implemented.

Org Agenda performs sorting based on a type text property which is set on entry strings depending on why the entry was collected. For example, when preparing a daily/weekly agenda, an entry selected because of its deadline has the property set to "deadline", while one selected because it was scheduled in the past has "past-scheduled". This is possible because entries are selected in multiple passes through each buffer, e.g. once to collect deadline items, once to get scheduled items, etc. But org-ql selects entries based on a given query in a single pass through a buffer, so it's not possible to know the Org Agenda-equivalent of the "reason" that an entry is collected, so the corresponding type property can't be set, so there's no such property for org-entries-lessp to sort on.

Anyway, I found the way to make org-ql's sorting work within an org-ql-block block.

However, I realized that changing the way org-ql-block's arguments are passed will break the configs of users who are running master (which is 99.9% of users, of course, since virtually no one uses MELPA Stable). So I think this change should be postponed to 0.5, so at least we can have a changelog entry, and maybe some code to warn about the changing argument format. (Not that many people read the changelog, anyway, but at least it will be there if someone notices the breakage and needs to find out why.)

So I've pushed this WIP branch: https://github.com/alphapapa/org-ql/tree/wip/issue-79 for 0.5.

[bitclick]

this could be another bug, but for me it serves as a workaround:

org-ql-block applies the sorting-strategy of the preceding normal agenda-block.

i have a dummy block before the org-ql-block and apply the sorting strategy there. the results from the org-ql-block are sorted like i want them to:

...
(todo "nothing here"
      (
       (org-agenda-overriding-header "")
       (org-agenda-sorting-strategy '(category-keep))
       )
  )
(org-ql-block  ...) 
...

maybe this is related to #121, in the sense that there is some cleanup missing when initializing the agenda-buffer for org-ql.

anyways, now i can sort the way i like :-)

[alphapapa]

@bitclick I'm afraid that's not the case. As I explained in https://github.com/alphapapa/org-ql/issues/79#issuecomment-575969925, org-ql blocks are not compatible with org-agenda sorting.

You can test this for yourself as follows: In a clean Emacs instance:

1. Create file test79.org with these contents:

* Heading 1
<2020-07-23 Thu 09:00>

* Heading 2
<2020-07-24 Thu 09:00>

2. Add that file to the agenda list.
3. Evaluate this expression:

(setq org-agenda-custom-commands
      '(("A" "Today"
	 ((todo "nothing here"
		((org-agenda-sorting-strategy '(timestamp-down))))
	 (org-ql-block '(ts)
		       ((org-ql-block-header "TEST")))))))

4. Display that custom agenda.
5. Note the order of the headings: 1 followed by 2.
6. Change the sorting strategy to timestamp-up.
7. Display that custom agenda anew.
8. Note that the order of the headings is unchanged.

[YujiShen]

New to org-ql and org-super-agenda. I am trying to sort items by priority within a group defined by org-ql-block in org-agenda-custom-commands. Just saw this is still open. Does it mean I can not sort items within group of org-super-agenda with org-ql?

I saw the doc of org-super-agenda said:

The order of items may not be preserved after grouping due to the implementation’s using hash tables. Future versions may address this shortcoming.

But I also saw a Reddit thread said:

Sorting is controlled by Org Agenda settings (or org-ql, if you're using it).

So I am very confused: Is it possible to sort within a group of org-super-agenda (using org-ql or not)?

[xanalogica]

Anyway, I found the way to make org-ql's sorting work within an org-ql-block block.

So I've pushed this WIP branch: https://github.com/alphapapa/org-ql/tree/wip/issue-79 for 0.5.

Thanks for this work. I've been trying it out and found it tricky to properly declare org-agenda-custom-command so I'm pasting what I got working for others who make wander by. Pardon the spread-out syntax -- I had to do that to figure out what I kept doing wrong -- which was an incorrect combination of quoting.

  ("c" "<c>aptured things"        ;; [1] key  [2] description
    org-ql-block                  ;; [3] function that select items
    '(                            ;; [4] match/query expression
      (property "CAPTURED")       ;;   arg1: org-query expression
      :sort (priority date)       ;;   keyword arg
      :header "xyzzy"             ;;   keyword arg
     )
    (                             ;; [5] list of extra settings
    )
    ()                            ;; [6] list of files to export to
  ) ;; end of agenda entry definition

[alphapapa]

Since this involves changing the format, what should probably be done is:

1. [ ] In 0.6, warn about changing argument format when the old format is used.
2. [ ] In 0.7, only accept the new format. (Since this change will be made on master, it should probably be delayed for at least a few weeks after 0.6 is released.)

[alphapapa]

Retargeting this for 0.7. 0.6 has been delayed for too long.

[valsdav]

Do you have news about when this nice feature will land in master?

Anyway, I found the way to make org-ql's sorting work within an org-ql-block block. So I've pushed this WIP branch: https://github.com/alphapapa/org-ql/tree/wip/issue-79 for 0.5.

Thanks for this work. I've been trying it out and found it tricky to properly declare org-agenda-custom-command so I'm pasting what I got working for others who make wander by. Pardon the spread-out syntax -- I had to do that to figure out what I kept doing wrong -- which was an incorrect combination of quoting.

  ("c" "<c>aptured things"        ;; [1] key  [2] description
    org-ql-block                  ;; [3] function that select items
    '(                            ;; [4] match/query expression
      (property "CAPTURED")       ;;   arg1: org-query expression
      :sort (priority date)       ;;   keyword arg
      :header "xyzzy"             ;;   keyword arg
     )
    (                             ;; [5] list of extra settings
    )
    ()                            ;; [6] list of files to export to
  ) ;; end of agenda entry definition

This block is giving me "Invalid Org-ql query" message in master at the moment. Thanks a lot for this!

[alphapapa]

Do you have news about when this nice feature will land in master?

No, my time for working on Emacs-related things is limited, and I've been spending it on other projects lately. If you need it sooner, you should be able to use that WIP branch. When I work on this package again, I'll try to get to this, but there are a number of things waiting for my attention.

[Pentaquark1]

I also tried to sort org-ql-block with various code snippets from this and related threads unsuccessfully.

[ParetoOptimalDev]

Since the order is lexographically sorted by default I've been appending "a" to group names I want to be first, "z" to the last, and so on.

Not ideal, but a workaround.

[alphapapa]

I've rebased this branch on top of master: https://github.com/alphapapa/org-ql/tree/wip/issue-79 If it still works (and I think it should), then all that's needed is to document the changes to the arguments, and then it can be merged.

[ptn]

warn about changing argument format when the old format is used.

By old format, do you mean the org-ql-block-sort option, and new format would mean :sort? If that's the case, does that mean the former is already implemented somewhere but not merged?

[maikol-solis]

If that's the case, does that mean the former is already implemented somewhere but not merged?

Yes. The fix for this issue is available in this branch: https://github.com/alphapapa/org-ql/tree/wip/issue-79/

Hoping that this will be merged soon! 🙌🏻

[ptn]

That implements the :sort option, not the org-ql-block-sort option.

[alphapapa]

That implements the :sort option, not the org-ql-block-sort option.

I don't understand what you're asking. The intention is to replace that option with the :sort argument.

[ptn]

I had misunderstood then, I think it's safe to disregard my previous comment and I am now on the same page as everyone else.