Suggestion: Searching the contents of the notes

[Fourchaux]

I'm trying (the very impressive) zk program ( https://github.com/localauthor/zk ).
Maybe you already know localauthor (Grant Rosson)?

Among the basic functions, zk offers a search on the contents of the notes ( zk-find-file-by-full-text-search ). Could be a good addition in Denote.

Small videos ( https://www.youtube.com/channel/UCirnOrHNy7N_CIA_dS7XWug/videos ) present some functionalities of zk. They are all instructive.

In addition, the "philosophy" of zk is the same as denote. No dependency on external packages and heavy use of embark, consult, vertico, orderless ...

[protesilaos]

Hello @Fourchaux!

Yes, I know zk---it is linked to from Denote's manual. In essence, the difference between it and Denote is the latter's file-naming scheme.

To your suggestion: do you know how this search mechanism differs from existing built-in solutions, such as grep, project-find-regexp, find-grep-dired as well as what other packages provide such as consult, consult-notes, and deft?

[Fourchaux]

It's not so much a search mechanism issue ( zk uses grep for example https://github.com/localauthor/zk/blob/main/zk.el#L342 )
but more a convenience/usability issue.

As a basic user (or a lazy one too...) I would like to have denote functions that allow searching in the body of the notes. (same behavior as denote-link-add-links and denote-link-dired-marked-notes). I may be wrong but this kind of search seems important.

[protesilaos]

I understand the convenience argument, though I must ask: how do you now search through file contents in a regular file that was not created by Denote?

The reason I ask this is because I consider search a foundational feature that belongs directly to core Emacs or a specialised package (e.g. consult). It should work in all regular files, not just those of Denote (and Denote files are regular files, anyway).

To me, it feels more convenient to have an interface that works everywhere, rather than a tweaked grep for Org, another for Denote, a third for a different package, and so on.

What I fail to see right now is how a Denote wrapper around grep would add value. Maybe I am missing something from zk's approach, so please excuse my ignorance. I am genuinely curious to figure out what the problem is and whether the solution should be provided by Denote.

If you or anyone else reading this has an insight, please let me know.

(same behavior as denote-link-add-links and denote-link-dired-marked-notes).

The difference here is that the linkining functionality is specific to Denote: we use the denote: link type. We are not wrapping around the behaviour of another command, as would be the case with grep.

I may be wrong but this kind of search seems important.

It is super important for all files, not just Denote. Which suggests that a general approach is more likely to be the right one.

[basaran]

zk is a great package too. I watched some of the videos, and I think it is more in the lines of bundled software vs denote being just denote, an interface for you to quickly write buffers.

To search for things through denote, all it takes is to invoke (consult-ripgrep denote-directory) and you could search however you like. That is, if you are using consult.

I understand one might righteously think "it would be great if X just worked out of the box", but we are Emacs users. The only thing that works out of the box is Emacs itself, for everything else we add, I noticed the approach is to try not to break it.

It's probably unwise for me to add anything about vi, but I suppose the war started decades ago and I won't make it worse. In my understanding with vi, nothing works out of the box and the user is expected to make it work :)

None the less, it appears they are both ruthless entities who require one to offer their time before giving you their blessings.

[Fourchaux]

Okay, here is a suggestion: zk also has a function zk-search (which defaults to zk-grep - a basic wrapper around 'rgrep') https://github.com/localauthor/zk/blob/main/zk.el#L870 . Could we have an equivalent 'denote-search' function?

A general question: what type of development is envisaged for denote?

Will it stay more or less as is or features can be incorporated?

Examples:

• finding dead links or unlinked notes
• consult-grep for alternative search functions (as suggested by @basaran )
• embark integration
• ebib access
• biblio.biz access
• ...

[protesilaos]

zk also has a function zk-search (which defaults to zk-grep - a basic wrapper around 'rgrep') https://github.com/localauthor/zk/blob/main/zk.el#L870 . Could we have an equivalent 'denote-search' function?

I noticed. Though I still don't see the value of it. Why do we need a Denote-specific rgrep? How would it differ from any other grep that we already use to search file contents?

As a matter of principle, I don't think it is a good practice for each package to define its own little variant of standard mechanisms. There has to be a compelling reason for it, some exception to the rule, which I do not see right now.

To explain how I work with all files:

1. If I already know what I want and do not need live feedback, I have a simple wrapper around rgrep and lgrep, which is also mentioned in the Denote manual: https://protesilaos.com/emacs/denote#h:76198fab-d6d2-4c67-9ccb-7a08cc883952

2. If I need live feedback, I call consult-grep, which I can then export to a grep buffer with Embark.

3. To search only specific files, I have a command that operates on marked files. Copying it here for your convenience from my dotfiles (GitHub mirror):

(defvar prot-dired--find-grep-hist '()
"Minibuffer history for `prot-dired-grep-marked-files'.")

;; Also see `prot-search-grep' from prot-search.el.
;;;###autoload
(defun prot-dired-grep-marked-files (regexp &optional arg)
 "Run `find' with `grep' for REGEXP on marked files.
When no files are marked or when just a single one is marked,
search the entire directory instead.

With optional prefix ARG target a single marked file.

We assume that there is no point in marking a single file and
running find+grep on its contents.  Visit it and call `occur' or
run grep directly on it without the whole find part."
 (interactive
  (list
   (read-string "grep for PATTERN (marked files OR current directory): " nil 'prot-dired--find-grep-hist)
   current-prefix-arg)
  dired-mode)
 (when-let* ((marks (dired-get-marked-files 'no-dir))
             (files (mapconcat #'identity marks " "))
             (args (if (or arg (length> marks 1))
                       ;; Thanks to Sean Whitton for pointing out an
                       ;; earlier superfluity of mine: we do not need
                       ;; to call grep through find when we already
                       ;; know the files we want to search in.  Check
                       ;; Sean's dotfiles:
                       ;; <https://git.spwhitton.name/dotfiles>.
                       ;;
                       ;; Any other errors or omissions are my own.
                       (format "grep -nH --color=auto %s %s" (shell-quote-argument regexp) files)
                     (concat
                      "find . -not " (shell-quote-argument "(")
                      " -wholename " (shell-quote-argument "*/.git*")
                      " -prune " (shell-quote-argument ")")
                      " -type f"
                      " -exec grep -nHE --color=auto " regexp " "
                      (shell-quote-argument "{}")
                      " " (shell-quote-argument ";") " "))))
   (compilation-start
    args
    'grep-mode
    (lambda (mode) (format "*prot-dired-find-%s for '%s'" mode regexp))
    t)))

Will it stay more or less as is or features can be incorporated?

More or less, yes. It is feature complete. Though there still are refinements to be made.

This does not preclude extensions which must be implemented as different packages. I am happy to help anyone who want to write such a package.

• finding dead links or unlinked notes

Yes, this could be done.

• consult-grep for alternative search functions

I am not sure what you mean here. You mean some kind of integration with consult-grep? It seems to work fine on my end.

• embark integration

Yes, provided we don't introduce dependencies. Though this is not a high priority right now. Someone who uses Embark more than me must suggest what their idea for this is.

• ebib access
• biblio.biz access

These belong to an external package. For example: https://github.com/pprevos/denote-citar.

[Andre0991]

Let me also share a bit of my emacs.el, as this can be useful to others. I created a git repo in my denote dir (as suggested by https://github.com/protesilaos/denote/blob/main/README.org#107-treat-your-notes-as-a-project).

With that, I use C-S-d for dispatching project.el:

(use-package denote
  :init
  (defun apt-denote-project
      ()
    (interactive)
    (project-switch-project "~/dropbox/denote"))
  :bind
  ("C-S-d" . apt-denote-project))

Result:

I frequently use f for finding files as well.

The ripgrep action shows up because of this:

(add-to-list 'project-switch-commands '(consult-ripgrep "Ripgrep") t)

(although I doubt rg performance makes any difference compared to regular grep in my notes directory).

[protesilaos]

Thank you @Andre0991! Making your notes a project is a good idea and I'm doing it as well.

Just to add to this, in Emacs 28 we get this option:

(defcustom xref-search-program 'grep
  "The program to use for regexp search inside files.

This must reference a corresponding entry in `xref-search-program-alist'.

This variable is used in `xref-matches-in-files', which is the
utility function used by commands like `dired-do-find-regexp' and
`project-find-regexp'."
  :type '(choice
          (const :tag "Use Grep" grep)
          (const :tag "Use ripgrep" ripgrep)
          (const :tag "Use ugrep" ugrep)
          (symbol :tag "User defined"))
  :version "28.1"
  :package-version '(xref . "1.0.4"))

The command project-find-regexp is bound to C-x p g by default. It is another built-in option in case someone needs it.

EDIT: minor rewording.

[Andre0991]

Oh, indeed, I didn't know about that customization. Thanks!

[Fourchaux]

Thanks @prot for your comments.

Regarding Denote itself, it is clear that it will never be an independent application with many features. Moreover, by re-reading your excellent manual, I see that you are addressing this use ( § 16.8 In short: let Denote (or equivalent) create notes and link between them, the file manager organise and provide access to files, search programs deal with searching and narrowing, and version control software handle the tracking of changes). Denote is not a direct replacement for Org Roam. That's OK. People with little knowledge of emacs/elisp will use the basic functions available (which are obviously sufficient).

Denote is therefore aimed at emacsians who are invited to use the emacs tools and/or write their own functions.

Now with this type of development a problem is where are the resources? Didn't someone write a wonderful search function? If so, is the code available somewhere?

A possible improvement would be to group (a meta-note hehe...) in a single list (or paragraph) the resources CURRENTLY available for Denote. Ideally this list should be easily accessible (to inform people and also to encourage them to show their code).

Example:

FOR DENOTE:

• denote-citar (pprevos) Citar for denote https://github.com/pprevos/denote-citar

Videos:

• protesilaos https://www.youtube.com/watch?v=mLzFJcLpDFI
• System Crafters https://www.youtube.com/watch?v=QcRY_rsX0yY

Emacs configs:

• protesilaos https://github.com/protesilaos/dotfiles/blob/master/emacs/.emacs.d/prot-emacs-modules/prot-emacs-write.el
• Live System Crafters https://systemcrafters.net/live-streams/july-15-2022/
• Andre0991 https://github.com/Andre0991/dotfiles/blob/master/emacs.el

Various:

• https://jao.io/blog/2022-06-19-simple-note-taking.html
• https://lucidmanager.org/productivity/testing-denote-package/

USABLE by DENOTE:

• consult-notes (mclear-tools) Use consult to search notes https://github.com/mclear-tools/consult-notes

• zk (as a great source of examples) https://github.com/localauthor/zk implements many of the features of Zetteldeft package. The presentation videos are also very interesting https://www.youtube.com/channel/UCirnOrHNy7N_CIA_dS7XWug/videos

That's it for now.

Thanks again @prot (and @Andre0991 too).

You can close this (long) issue if you want.

[protesilaos]

Thanks prot for your comments.

You are welcome! Note that @ + prot is not my username here: it is @protesilaos. Just so you don't ping the wrong person.

Regarding Denote itself, it is clear that it will never be an independent application with many features.

Correct. It is minimal so that it works with standard Emacs and, by extension, can adapt to the tools the user already has. I personally dislike packages that pull in irrelevant dependencies because they ask me to use stuff I do not want: I want to do things my way.

Moreover, by re-reading your excellent manual, I see that you are addressing this use ( § 16.8 In short: let Denote (or equivalent) create notes and link between them, the file manager organise and provide access to files, search programs deal with searching and narrowing, and version control software handle the tracking of changes).

Indeed. Emacs already provides everything the user needs and there are packages which enhance that experience.

Denote is not a direct replacement for Org Roam. That's OK.

I have never used org-roam, so I do not know what "replacing" would mean in this case. For someone who uses org-roam to produce notes, then yes, Denote is a replacement for it. If, however, org-roam provides X, Y, Z extra features then no, Denote is not a replacement for those---and shall never be.

I consider Denote's limited scope to be a strength, not a weakness. We let the user use the tools they already have. Re-using code and being a good Emacs citizen is the right thing to do.

People with little knowledge of emacs/elisp will use the basic functions available (which are obviously sufficient).

People do not need any knowledge of Elisp to use Denote, M-x grep, M-x consult-grep, M-x project-find-regexp, Vertico, Ivy, Helm, and all the others. Writing wrapper functions is entirely optional and only for those who want to tinker with the code.

You can close this (long) issue if you want.

I say we keep it open for a while, in case others want to read it.

Thank you for your time!

[protesilaos]

Hello again everyone! I am about to release version 0.5.0. Its change log mentions this page, so the user can be informed about the contents of our discussion. I am thus closing the issue.