Offer migration paths for existing notes

[tmalsburg]

I imagine many prospective users of this package already have notes in one of the two formats offered by helm-bibtex or some other format. Is there something that could be offered to these users to make the transition easier? Doesn't have to be a part of the package; some hints and sample code in README.org would go a long way.

[zaeph]

I imagine many prospective users of this package already have notes in one of the two formats offered by helm-bibtex or some other formats. Is there something that could be offered to these users to make transition easier?

1. If the user was using different files, the only thing we'd need to ensure is the presence of the file-metadata tag #+ROAM_KEY: (and possibly #+TITLE:).
2. If the user was using a single file, we also need to isolate each heading into a separate file.

In theory, it's possible, since bibtex-completion provides us with everything we need. In practice, this is quite a bit finicky, especially for creating files.

In case 1., we could prepend the content of the files with the relevant file-metadata tags, checking if a title has already been defined and, if not, creating it.

In case 2., however, things can get hairy quite fast. We'd have to vet every heading to make sure that it matches a BibTeX cite-key in your .bib files. But what if you switch .bib files, and use the same the directory for all of it, thereby shadowing the files which aren't currently in your .bib file?

If we suppose that, still for case 2., every heading in the file is linked to a BibTeX cite-key, then things might be a little easier. We create a file out of each tree, adjust the headings level accordingly, add the file-metadata tags from the BibTeX cite-key, and call it a day.

And obviously, we make all this non-destructive to prevent fuck-ups.

How does that sound to you?

[tmalsburg]

Alterntive case 2 solution: Don't iterate over sections in org-file but iterate over entries returned by bibtex-completion-candidates. If entry has notes, use existing edit notes code to grab them, and insert into new notes file created by org-roam-bibtex. A variant of this also works for case 1.

[zaeph]

Alterntive case 2 solution: Don't iterate over sections in org-file but iterate over entries returned by bibtex-completion-candidates.

Then, we need to make it clear that the users should have all their .bib files loaded before migrating.

I don't have much experience writing tests, let alone migration scripts. I'm going to think about it, but if anyone's feeling up to the task, feel free to let us now.

[tmalsburg]

I will look into it. Should be easy. But assuming I have an entry's note in a string and the BibTeX key, what's the easiest way to create a new note in org-roam-bibtex programmatically, i.e. without manual user interaction?

[zaeph]

But assuming I have an entry's note in a string and the BibTeX key, what's the easiest way to create a new note in org-roam-bibtex programmatically, i.e. without manual user interaction?

You'd want to use org-roam-bibtex-edit-notes:

• Store the string in a variable.
• Set org-roam-bibtex-template in a let-form to a template which will have org-capture expand the variable via %(sexp).

This should do it nicely.

[tmalsburg]

Cool, thanks.

[tmalsburg]

I have this sketch below (for just one entry). Idea is to copy the notes for the entry to the kill ring and then to have the notes automatically inserted from there by including %c in the template. However, it doesn't work. The org-roam entry is created but the notes are not inserted. I'm afraid I must be missing something about org-roam-bibtex-edit-notes or how the template is used. Can you spot the mistake?

(let ((key "Haeussler2012"))
  ;; First copy notes to kill ring. Then open org-roam notes file with
  ;; a org-roam-bibtex-template that uses to `%c` to insert notes at
  ;; the desired position.
  (let ((org-roam-bibtex-template
         '(("r" "ref" plain #'org-roam-capture--get-point ""
            :file-name "${slug}"
            :head "#+TITLE: ${title}\n#+ROAM_KEY: ${ref}\n\n%c"
            :unnarrowed t))))
    (bibtex-completion-edit-notes (list key))
    (copy-region-as-kill
     (progn (beginning-of-buffer) (point))
     (progn (end-of-buffer) (point)))
    (bibtex-completion-exit-notes-buffer)
    (org-roam-bibtex-edit-notes key)))

[zaeph]

On it.

[zaeph]

Good news: it's just a typo!

We changed the name of the variable from org-roam-capture-template to org-roam-capture-templates (i.e. plural).

This should work:

(let ((key "Haeussler2012"))
  ;; First copy notes to kill ring. Then open org-roam notes file with
  ;; a org-roam-bibtex-template that uses to `%c` to insert notes at
  ;; the desired position.
  (let ((org-roam-bibtex-templates                                            ; <---
         '(("r" "ref" plain #'org-roam-capture--get-point ""
            :file-name "${slug}"
            :head "#+TITLE: ${title}\n#+ROAM_KEY: ${ref}\n\n%c"
            :unnarrowed t))))
    (bibtex-completion-edit-notes (list key))
    (copy-region-as-kill
     (progn (beginning-of-buffer) (point))
     (progn (end-of-buffer) (point)))
    (bibtex-completion-exit-notes-buffer)
    (org-roam-bibtex-edit-notes key)))

[tmalsburg]

Just tried it but now I get a new buffer that's completely empty. Hm ...

[zaeph]

Just tried it but now I get a new buffer that's completely empty. Hm ...

Can you get this to work?

(let ((key "Haeussler2012"))
  (let ((org-roam-bibtex-templates
         '(("r" "ref" plain #'org-roam-capture--get-point ""
            :file-name "${slug}"
            :head "#+TITLE: ${title}\n#+ROAM_KEY: ${ref}\n\n%c"
            :unnarrowed t))))
    (kill-new "Testing 123")
    (org-roam-bibtex-edit-notes key)))

[tmalsburg]

Nope, still an empty buffer (with name "haessler2020.org").

[zaeph]

Strange, it's working on my end. Have you pulled the repo?

[tmalsburg]

I installed org-roam-bibtex two hours ago from Melpa (via straight.el). Will have to test with emacs -Q to rule out interference from other stuff I suppose.

[zaeph]

I installed org-roam-bibtex two hours ago from Melpa (via straight.el). Will have to test with emacs -Q to rule out interference from other stuff I suppose.

Yeah, sorry, quite the annoying problem you've got there. I've run into some issues with helm-bibtex being loaded by another package before its use-package block in my config, and that was a nightmare to debug.

[tmalsburg]

I cannot even get this to work: (org-roam-bibtex-edit-notes "Haeussler2012")

I think the org-roam db was perhaps corrupted when I first made a notes file for this entry but then didn't save it? Reason: When I try a different key, it works.

[tmalsburg]

Raises the question: how can notes be deleted?

[zaeph]

Raises the question: how can notes be deleted?

You could run org-roam-db--clear and org-roam-db-build-cache to recreate it.

[tmalsburg]

Could it be that an entry in org-roam's db is created even before the capture process is finalized? In this case an aborted capture process should trigger some clean-up.

[zaeph]

Could it be that an entry in org-roam's db is created even before the capture process is finalized? In this case an aborted capture process should trigger some clean-up.

I'm pretty sure we're only writing to the db on filesave in Org-roam. Let me check.

Edit: Yes, this is what we're running with org-capture-after-finalize-hook: https://github.com/jethrokuan/org-roam/blob/fbdda33c6a9bcc01f1bf5550f0bf7d3ea817670e/org-roam.el#L760-L767 L765.

[tmalsburg]

Hm, but when I start org-roam-bibtex-edit-notes, then abort (C-c C-k), and then run it again, I have the content of the template twice. Something must be left over. But there is no file on disk.

[zaeph]

Hm, but when I start org-roam-bibtex-edit-notes, then abort (C-c C-k), and then run it again, I have the content of the template twice. Something must be left over. But there is no file on disk.

Yes, I can reproduce it. Sadly, this is an Org-roam problem rather than an orb problem. Org-roam has addressed it by disabling nested/parallel captures, but we're not running those checks in orb. As of now, there isn't a very robust way to check if a capture buffer is already acting on the citekey, and whatever structure we could incorporate (like an alist listing the current captures), I think we're better waiting for upstream to fix it.

Well, when I say 'waiting', I'm in charge of it, so I'll try to tackle it next week, cf. jethrokuan/org-roam#527.

[tmalsburg]

Got it. Luckily it's not a show-stopper in the current use case. Will continue to work on the migration script over the next couple of days.

[zaeph]

Will continue to work on the migration script over the next couple of days.

Thanks for your work, and good luck for the new semester!

[myshevchuk]

  (let ((org-roam-bibtex-templates
         '(("r" "ref" plain #'org-roam-capture--get-point ""
            :file-name "${slug}"
            :head "#+TITLE: ${title}\n#+ROAM_KEY: ${ref}\n\n%c"
            :unnarrowed t))))
    (kill-new "Testing 123")
    (org-roam-bibtex-edit-notes key)))

@tmalsburg @zaeph I think the problem here is that :head is Org-roam's homegrown addition to org-capture-templates and it uses a different wildcards syntax, so Org-capture's %... wildcards do not work here.

If I understood the intent of the function correctly, this may be fixed as follows, just move %c into the template string (5th element of the list):

  (let ((org-roam-bibtex-templates
         '(("r" "ref" plain #'org-roam-capture--get-point "%c"
            :file-name "${slug}"
            :head "#+TITLE: ${title}\n#+ROAM_KEY: ${ref}\n\n"
            :unnarrowed t))))
    (kill-new "Testing 123")
    (org-roam-bibtex-edit-notes key)))

[tmalsburg]

Below is a script that worked for me. Definitely use at your own risk :)

(seq-do
 (lambda (key)
   ;; First copy notes to kill ring. Then open org-roam notes file with
   ;; a org-roam-bibtex-template that uses to `%c` to insert notes at
   ;; the desired position.
   (let ((org-roam-bibtex-templates
          '(("r" "ref" plain #'org-roam-capture--get-point ""
             :file-name "${citekey}"
             :head "#+TITLE: ${author-abbrev}${title}\n#+ROAM_KEY: ${ref}\n\n%c"
             :unnarrowed t))))
     (bibtex-completion-edit-notes-default (list key))
     (copy-region-as-kill
      (progn (goto-char (point-min)) (point))
      (progn (goto-char (point-max)) (point)))
     (bibtex-completion-exit-notes-buffer)
     (org-roam-bibtex-edit-notes key)
     (org-capture-finalize)))
 (list (car (seq-map (lambda (candidate) (cdr (assoc "=key=" candidate)))
          (seq-filter
           (lambda (candidate) (assoc "=has-note=" candidate))
           (bibtex-completion-candidates))))))

If you want to experiment with different templates, just run the code above, check results, delete new note files in org-roam-directory, then reset org roam cache via:

(progn
  (org-roam-db--clear)
  (org-roam-db-build-cache))

Rinse and repeat.

[zaeph]

Nice! As much as I'd love to run the risk of having things catastrophically fail on the eve of submission, I'll pass.

If anyone gives it a go—and please back-up your files before—please tell us here if it worked for you!

[tmalsburg]

Good luck with your submission! And congrats in advance. Got to get back to actual work as well. :)

[zaeph]

Thank you, and good luck to you as well! :)