Issue with org-ql-search and narrowed org subtree

[arvindsv]

First of all: Thank you very much for this package (and org-super-agenda). They're great and a lot of effort seems to have gone into them.

I'm not sure if this behavior is expected or if I am understanding this package wrong.

Context:

• Emacs: 27.0.91 (9.0)
• org-ql version: org-ql-20200713.309
• org-super-agenda version: org-super-agenda-20200310.1337

Setup:

I did this in emacs --execute '(setq user-emacs-directory "/tmp/emacs.d")' -Q:

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

(package-refresh-contents)
(package-install 'org-ql)

(setq org-directory "/tmp/org")

These are the contents of /tmp/org/test.org:

* Test 1 (Deadline today - Schedule 2 days ago)
SCHEDULED: <2020-08-07 Fri> DEADLINE: <2020-08-09 Sun>

* Test 2 (Deadline futute - Hidden) :hide:
DEADLINE: <2020-08-10 Mon>

* Test 3 (Deadline future - Not scheduled)
DEADLINE: <2020-08-10 Mon>

Reproducing this issue:

1. Run:

   (org-ql-search '("/tmp/org/test.org")
     '(not (tags-local "hide"))
     :super-groups `((:name "Today"
                      :deadline today)))
   

   You will see something like this (this is correct and expected):

   

2. Go into /tmp/org/test.org and run org-narrow-to-subtree on any of the three subtrees. Then, re-run the org-ql-search run in step 1 above. You will see something like this:

   

This seems both wrong (the "Today" group is gone) and somehow dependent on the narrowing.

Stuff I've tried:

1. I've verified that :narrow t does the right thing (what it says it does). But, in this case: :narrow is nil and so it shouldn't matter.

2. I've verified that :narrow is being used properly. Running org-ql-select (like shown below) works fine, with or without narrowing. It returns the two items it should.

   (org-ql-select '("/tmp/test.org")
     '(not (tags-local "hide"))
     :action 'element-with-markers)
   

So, it looks like it's something around https://github.com/alphapapa/org-super-agenda/blob/dd0d104c269fab9ebe5af7009bc1dd2a3a8f3c12/org-super-agenda.el#L749 but I'm not sure. Can you help?

[alphapapa]

First of all: Thank you very much for this package (and org-super-agenda). They're great and a lot of effort seems to have gone into them.

Thanks for the kind words. Indeed, I have worked on them a lot.

Regarding the issue you reported: note that org-super-agenda is a separate package, and it alone is responsible for grouping results in these views, which are designed to imitate (but not necessarily be identical to) org-agenda views.

I don't know why narrowing the buffer would affect the grouping, but it's probably related to the metadata that is added to results when preparing the org-ql-view buffer or inserting the results into it. That could indicate an issue in either org-super-agenda or in this package. Or perhaps it's an interaction with Org itself. What version of Org are you using?

[arvindsv]

@alphapapa org-version says 9.3. It seems to be built-in. I don't see it installed in /tmp/a/.emacs.d.

As an aside: I realised that I hadn't isolated the config properly, in the earlier comment. I needed to do this:

$ mkdir /tmp/a
$ export HOME=/tmp/a
$ emacs --execute '(setq user-emacs-directory "/tmp/a/.emacs.d")' -Q

The issue is still reproducible with that.

[alphapapa]

Hi again,

Thanks for your patience, as I've been working on other things.

If you're still interested in this, it would be helpful if you would do the following: For each of the two screenshots you showed, run this command and paste the code shown in the buffer:

(defun org-ql-view-debug ()
  (interactive)
  (cl-assert org-ql-view-query)
  (let ((string (buffer-string)))
    (with-current-buffer (get-buffer-create "*org-ql-view-debug*")
      (erase-buffer)
      (emacs-lisp-mode)
      (prin1 string (current-buffer))
      (pp-buffer)
      (pop-to-buffer (current-buffer)))))

You will need to manually remove the very large keymap structures, which are not relevant.

This will probably show what's going on.

Thanks.

[arvindsv]

Hello!

Thanks for your patience, as I've been working on other things.

Oh, of course. I understand. It's not fair for me to expect having this looked at, and certainly not within some timeline! Some kindness towards the efforts of (all) open-source creators and maintainers goes a long way. :) Thank you for getting back.

I did what you said. I've attached four files:

1. good.nokeymaps.txt - Corresponds to the first screenshot (with the keymap structures removed)
2. bad.nokeymaps.txt - Corresponds to the second screenshot (with the keymap structures removed)
3. good.original.txt - Same as good.nokeymaps.txt with the keymap structures NOT removed
4. bad.original.txt - Same as bad.nokeymaps.txt with the keymap structures NOT removed

Context:

Emacs version: Emacs 27.1 org-ql version: org-ql-20201120.1422 org-super-agenda version: org-super-agenda-20201121.941

[alphapapa]

Thanks, I'll look at this, hopefully, soon. I have some other things I need to get to first.

[alphapapa]

I appreciate your patience. I've been busy with other things for a while.

I'd like to release a new stable version soon, so since this bug is relatively minor and may require some digging, I'm going to defer it to 0.7.

[arvindsv]

No problem. Since it's very niche and seemingly affecting no one else, please feel free to close it. I will understand. If there are higher priority, more impactful things you can be working on, you should. :)

[alphapapa]

It may be a real bug, just a relatively obscure one, and perhaps even dependent on the Org version or something like that. It deserves investigation eventually. :) Thanks.

[alphapapa]

I'm planning on some enhancements to narrowing in the next version (you'll be able to pass an ID or outline path to search in a subtree), so I'll defer this to 0.9.