alphapapa
Query syntax in ancestors/parent predicate
I tried to use ancestors predicate to define custom predicate like the following:
(org-ql-defpred project ()
"Match a project - task with a todo subtask or :project: tag."
:body (or (tags-local "project")
(and
(todo)
(descendants `(and
(or (todo) (done))
(level < ,org-inlinetask-min-level))))))
(org-ql-defpred subproject ()
"Match a subproject"
:body (and (project)
(ancestors '(project))))
However, such definition failed when searching for (subproject) query.
Instead, I had to use non-intuitive definition:
(org-ql-defpred subproject ()
"Match a subproject"
:body (and (project)
(ancestors 'project)))
It is either a bug or I missed something in the documentation.
There appears to be a misunderstanding about the meaning of the :body argument to org-ql-defpred. You should use the :normalizers to rewrite the query, rather than defining a body to be run on each heading. Please see the docstring for org-ql-defpred as well as the tutorial.
There appears to be a misunderstanding about the meaning of the :body argument to org-ql-defpred. You should use the :normalizers to rewrite the query, rather than defining a body to be run on each heading.
I clearly misunderstood the purpose of :normilizers.
Please see the docstring for org-ql-defpred as well as the tutorial.
Actually, I tried to read the docstring, but failed to understand the purpose of :normalizers. Let me elaborate. Below is roughly my thought process when I tried to understand how to define custom predicates.
First, let start from the docstring:
PREAMBLES and NORMALIZERS are lists of pcase forms matched against Org QL query sexps. They are spliced into pcase forms in the definitions of the functions org-ql--query-preamble and org-ql--normalize-query, which see.
Ok... Looking into the docstring of org-ql--normalize-query:
org-ql--normalize-query is a compiled function without a source file.
Signature (org-ql--normalize-query QUERY)
Documentation Return normalized form of QUERY expression.
This function is defined by calling org-ql--define-normalize-query-fn, which uses normalizer forms defined in org-ql-predicates by calling org-ql-defpred.
And the definition is not available since the function is dynamically defined... I guess, I had to look into org-ql--define-normalize-query-fn?
Scrolling through the source code of org-ql--define-normalize-query-fn, I can finally see some pcase forms (note that source code for functions is not shown by built-in help buffer):
(cl-labels ((rec (element)
(pcase element
(`(or . ,clauses) `(or ,@(mapcar #'rec clauses)))
(`(and . ,clauses) `(and ,@(mapcar #'rec clauses)))
(`(not . ,clauses) `(not ,@(mapcar #'rec clauses)))
...
,@normalizer-patterns
(reading through CL manual about cl-labels...)
The takeout so far is that :normalizers are "lists of pcase forms", though I still don't understand what "matched against Org QL query sexps" refers to. Does it mean that I can use rec symbol in normlizers? I guess that docstring could just state it in words instead of sending me into digging through docstring and source code. At least docstring could directly refer to org-ql--define-normalize-query-fn, which actually have a readable source code.
In any case, the purpose of :normalizers is still not clear and I continue reading the docstring:
NORMALIZERS are used to normalize query expressions to standard forms.
What does "normalize" refer to? What does "standard forms" refer to?
For example, when the predicate has aliases, the aliases should be replaced with predicate names using a normalizer.
Does it mean that when NAME is a list of symbols, I also have to define normalizers to make things work (as shown in the example at the end of the docstring)?
Also, predicate arguments may be put into a more optimal form so that the predicate has less work to do at query time.
It is not really clear what "optimal form" refers to. I can only vaguely guess that I can optimize the query depending on the predicate arguments.
At the end, my understanding is mostly that I am confused and that :normalizers should be used in some special cases when complex arguments are given to the predicate. Nothing to do with the predicate I wanted to write given that it does not even accept arguments.
So, trying to read through examples:
(org-ql-defpred person (name)
"Search for entries with the \"person\" property being NAME or having the tag \"personNAME\"."
:body (or (property "person" name)
(tags (concat "person" name))))
The example uses normal query syntax inside :body argument. So, I tried to do the same and hence encountered the problem with ancestors.
Since I clearly misunderstood something, let's try to look into the example of normalizers:
(org-ql-defpred person (&rest names)
"Search for entries about any of NAMES."
:normalizers ((`(person . ,names)
`(or (tags ,@(cl-loop for name in names
collect (concat "person" name)))
,@(cl-loop for name in names
collect `(property "person" ,name)))))
:body (cl-loop for name in names
thereis (or (property "person" name)
(tags name))))
We have both :body and :normalizers here. This is really confusing. Is normalizers expanded before running the query? If so, how does it speedup search if :body is still present and unchanged in comparison with earlier example?
Only later, the example file mentions that the predicate "will still work" without body:
(org-ql-defpred (person p) (&rest names)
"Search for entries about any of NAMES."
:normalizers ((`(,predicate-names . ,names)
`(or (tags ,@(cl-loop for name in names
collect (concat "person" name)))
,@(cl-loop for name in names
collect `(property "person" ,name))))))
Does :body make any difference? It is not clear. It seems that it is simply ignored when :normalizers is present, according to the last example (which does not make sense and I probably misunderstand again).
Maybe some clue is given in
Yep, same result as the non-normalized query. And look at how much simpler it is to write (person "Alice" "Bob") than to write (or (tags "personAlice" "personBob") (property "person" "Alice") (property "person" "Bob")).
I fail to understand why one needs to write that long string if "person" predicate is defined without :normalizers. Example showed that (person "Alice" "Bob") would work either way...
To summarize my (little) understanding and my confusion:
- :normalizer forms seems to be expanded once before running query
- :body form is executed with point at tested heading for every heading
- Each :normalizer form is a plist matcher expanding into another org-ql query
- :body form is also an org-ql query (or eval-able elisp sexp)
For my case, it seems that I can define subproject predicate either using :normalizer or :body with :body syntax being more straightforward (I do not need to write the (,predicate-names) part). So, I still fail to understand why using :normalizers is necessary and what was the problem with ancestors.
I believe that docstring could benefit from explaining the purpose of :normalizers better and providing more examples. Also, the usage of existing predicates in :body form could be clarified. From docstring, :body must be normal elisp, but example somehow uses org-ql sexps, which I would not expect to be a valid eval-able elisp sexp.
It seems like you haven't read the primary documentation about org-ql queries, which states:
An org-ql query is a Lisp expression which may contain arbitrary expressions, as well as calling certain built-in predicates. It is byte-compiled into a predicate function which is tested with point on each heading in an Org buffer; when it returns non-nil, the heading matches the query. When possible, certain built-in predicates are optimized away to whole-buffer regular expression searches, which are much faster to search for than testing the predicate on each heading.
I'm not a database expert, but AFAIK, the concept of normalizing a query to a more optimal form is common.
Docstrings are not intended to be comprehensive references, especially for a macro like org-ql-defpred. If the documentation you find isn't enough, you could look at the source code in org-ql.el to see how the macro is used. You could also look at the tests to see how certain queries are normalized, and how some are optimized to regexp searches.
I'm not aware of any Emacs packages that comprehensively describe every facet of internal design in documentation--even many parts of Emacs require reading the source code to understand. The design of org-ql is relatively straightforward. If you want to understand how queries are run, you could start by instrumenting org-ql-select with Edebug and stepping through the code. You'll quickly see how a query is pre-processed, then run in each buffer.
I don't know why you had problems using the ancestors predicate. I'd suggest looking at the uses of it in the tests.
I do the best I can to imagine a new user's perspective and write to inform it, but there's always room to improve the documentation. But for some users, it's too much, and for others, it's never enough. shrug
I do the best I can to imagine a new user's perspective and write to inform it, but there's always room to improve the documentation. But for some users, it's too much, and for others, it's never enough. shrug
First of all, I am sorry if my last message sounded harsh. It was not an intention. I am thankful for all your (almost single-handed) work on the package. I merely tried to illustrate some of my confusion I had while trying to understand how to use newly introduced org-ql-defpred macro.
Note that I also tried to provide suggestions at the end. They do not necessarily belong to the docstring itself, but might as well benefit manual or examples/defpred.org.
Also, I do not urge you to implement my suggestions yourself. Once I am confident in my understanding and have spare time, I will be happy to provide patches (if you also think that improvements are useful).
Docstrings are not intended to be comprehensive references, especially for a macro like org-ql-defpred. If the documentation you find isn't enough, you could look at the source code in org-ql.el to see how the macro is used. You could also look at the tests to see how certain queries are normalized, and how some are optimized to regexp searches.
I agree. Too long docstring might be not a good idea in practice. However, org-ql-defpred is expected to be used by user, so it seems to me that explaining more details about it can be useful. I think Emacs just points user (via link at the end of the docstring) to relevant manual page in such cases.
It seems like you haven't read the primary documentation about org-ql queries, which states:
I did read the info pages of the org-ql, the readme, the examples of org-ql-defpred you provided in examples/defpred.org, and some of the source code. Now, I have also read org-ql.el in more details looking into more examples how org-ql-defpred is used.
I understand that org-ql queries can contain both built-in predicates and arbitrary Lisp. I also improved my earlier intuitive understanding what :normalizers are supposed to do. Though I still find the wording of that sentence awkward. I would say something like
NORMALIZERS are used to transform query expressions to other query expressions. It is different from BODY that is not an org-ql query, but ordinary Elisp sexp that will be evaluated.
Instead of
NORMALIZERS are used to normalize query expressions to standard forms.
I don't know why you had problems using the ancestors predicate. I'd suggest looking at the uses of it in the tests.
My confusion came from defpred.org example:
(org-ql-defpred person (name)
"Search for entries with the \"person\" property being NAME."
:body (property "person" name))
Org-ql query is used in the :body form here. Note that none of the predicates defined in org-ql.el use queries as :body form.
Is the above example not correct? If so, I can try to provide a patch changing defpred.org person predicate to something like
(org-ql-defpred person (name)
"Search for entries with the \"person\" property being NAME."
:body (string= name (org-entry-get (point) "person")))
First of all, I am sorry if my last message sounded harsh. It was not an intention. I am thankful for all your (almost single-handed) work on the package. I merely tried to illustrate some of my confusion I had while trying to understand how to use newly introduced
org-ql-defpredmacro.Note that I also tried to provide suggestions at the end. They do not necessarily belong to the docstring itself, but might as well benefit manual or examples/defpred.org.
Also, I do not urge you to implement my suggestions yourself. Once I am confident in my understanding and have spare time, I will be happy to provide patches (if you also think that improvements are useful).
I understand. No need to apologize. I appreciate your feedback, for without it I wouldn't know what a user's perspective really is.
Regarding queries, let me try to explain another way: org-ql queries are merely Lisp expressions. While a query is being executed, the defined org-ql predicates are bound to their respective symbols so they can be called directly, like property.
Normalizers are used to transform query expressions into more optimal or standard forms. I can't give a comprehensive explanation of what that means, but you can see in the source code how they are used for that, e.g. in the timestamp-related predicates.
Is the above example not correct?
The example is correct. I was very careful to execute the code in the tutorial as-written. And were it not correct, the purpose of the example would be defeated.
If a predicate is effectively just calling another predicate with certain arguments, as in the tutorial's example, it's clearer, and potentially more efficient, to normalize it into calling the other predicate (perhaps considering it a "virtual" predicate), rather than actually defining another predicate function.