nyxt icon indicating copy to clipboard operation
nyxt copied to clipboard
verified publisher icon atlas-engineer

Recursive search

Open aadcg opened this issue 3 years ago • 14 comments

This is a follow-up to #1246.

Still some work of investigation ahead, but it's coming as recursive-search-from-current-buffer with depth set to 1 is already working.

Updates soon.

aadcg avatar May 05 '22 21:05 aadcg

Nice! Looking forward to it, this is an amazing feature!

Ambrevar avatar May 06 '22 06:05 Ambrevar

Moving slowly. quri is playing some tricks on me that weren't there roughly a year ago.

I will address you valuable comments @aartaka, but there are more fundamental and pressing issues at the moment in this PR!

aadcg avatar May 13 '22 16:05 aadcg

Moving slowly. quri is playing some tricks on me that weren't there roughly a year ago.

I've been maintaining Quri, so if you think you've spotted a bug, please report :)

Ambrevar avatar May 13 '22 17:05 Ambrevar

I finally found the bug in the quri library.

aadcg avatar May 18 '22 09:05 aadcg

Hooray! :D

Ambrevar avatar May 18 '22 10:05 Ambrevar

Here's the proposed solution to fix the bug in the quri library.

@aartaka, an extra pair of eyeballs wouldn't hurt, if you have time and interest!

aadcg avatar May 20 '22 09:05 aadcg

[Note to self.]

I've spotted a subtle bug.

There are two ways to fetch links: using the renderer and dexador:get. Why do I need both, if the latter would suffice? If a buffer corresponding to a certain URL is already loaded, why not use it?

Something funny is going on. Let the current buffer correspond to https://nyxt.atlas.engineer/article/dark-mode.org.

Then:

(let ((link-list))
  (loop for link in (fetch-links)
        unless (member link link-list :test #'quri:uri-equal)
          do (push link link-list))
  (print link-list))

(#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/contact-work>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/contact>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/legal>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/auto-mode.org>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/login>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/documentation>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/feed>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/newsletter>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/faq>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/download>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/applications>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/>)

(let ((link-list))
  (loop for link in (http-fetch-links)
        unless (member link link-list :test #'quri:uri-equal)
          do (push link link-list))
  (print link-list))

(#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/contact-work>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/contact>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/legal>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/auto-mode.org>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/login>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/documentation>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/feed>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/newsletter>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/faq>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/download>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/applications>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/>)

What???

The following behaves well.

(let ((link-list))
  (loop for link in (list (quri:uri "https://nyxt.atlas.engineer/")
                          (quri:uri "https://nyxt.atlas.engineer/foo")
                          (quri:uri "https://nyxt.atlas.engineer/bar")
                          (quri:uri "https://nyxt.atlas.engineer")
                          (quri:uri "https://nyxt.atlas.engineer"))
        unless (memeber link link-list :test #'quri:uri-equal)
          do (push link link-list))
  (print link-list))

(#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/bar>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/foo>
 #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/>)


(member (quri:uri "https://nyxt.atlas.engineer") 
        (list (quri:uri "https://nyxt.atlas.engineer/")) 
        :test #'quri:uri-equal)

(#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/>)

I must be doing smth really stupid. No, I didn't.

FINALLY!!!!! The issue is that quri:uri-equal behaves differently when the URI's path is the empty string or NIL.

aadcg avatar Jun 09 '22 14:06 aadcg

I don't understand what the problem is.

Note there is quri:uri= and quri:uri-equal, the latter normalizes the path (see the docstring).

Also I presume your code is trying to delete the duplicates in the list. If so, use remove-duplicates or delete-duplicates, then your code is a one-liner:

(print (delete-duplicates (fetch-links) :test #'quri:uri-equal))

Ambrevar avatar Jun 09 '22 17:06 Ambrevar

@Ambrevar, my code is indeed silly but I had reasons to write it the way I did. It was hard to trace back the issue.

aadcg avatar Jun 09 '22 17:06 aadcg

Second bug in the quri library found and solved. Moving on.

aadcg avatar Jun 10 '22 19:06 aadcg

The link computing engine seems to be correct, after fixing some bugs in the quri library.

So now I'm able to compute all links starting from Nyxt's homepage. In this case, the maximum distance between links is 3. So if I ask it compute all links up to distance 9999, I get the same result.

(recursive-links 3 (list "https://nyxt.atlas.engineer"))

(recursive-links 9999 (list "https://nyxt.atlas.engineer"))
If you're curious to see the boring output, click to expand. 
((#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/>)
 (#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/contact-work>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/contact>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/legal>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/login>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/documentation>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/feed>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/newsletter>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/faq>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/download>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/applications>)
 (#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/application/demeter.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/command-line-programs.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/hooks.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/technical-design.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/emacs-hacks.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-1.3.1.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-1.3.2.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-1.3.4.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/the-nyxt-thesis.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-1.4.0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-1.5.0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/next-nyxt-rename.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/reading-line.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/element-hints.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/autofills.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-1.3.3.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/debconf-presentation.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-1.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/auto-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/fosdem-presentation.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/common-settings.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-2.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/class-based-functional-configuration.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/bookmarklets.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/hooks-implementation.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-3.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/nyxt-powerline.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/dark-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-4.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/continuous-testing-and-packaging.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/dashboard.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/reduce-to-buffer.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/dbscan.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/dark-theme.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/package-manager.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-5.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/visual-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/fosdem-presentation-2021.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/global-history-tree.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/watch-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/the-thin-line-between-users-and-collaborators.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/diff-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-6.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/radiostudent.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/nyxt-extensions-tutorial.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/sharing-files.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2-pre-release-7.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/prompt-buffer.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/prompt-buffer-customization.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.0.0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/cruise-control-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/enable-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.1.0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/process-mode-and-repeat-mode.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.1.1.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/element-frame.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/sustainable-open-source.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/macro-edit.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/status0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/status1.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/qr-url.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/panel-buffers.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.2.0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/nyxt-versus-plugins.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/getting-started-on-nyxt-hacking-1-bookmarklet.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.2.1.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.2.2.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/record-input-field.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/article-how-can-i-make-emacs-my-web-browser.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.2.3.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/demeter-montezuma.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/emacs-conf.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/why-building-nyxt-instead-of-an-emacs-package.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/nyxt-showcase-and-configuration-demo.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/demeter-release-0.1.0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/release-2.2.4.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/annotations.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/gavin-freeborn-interview.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/distrotube-video-about-nyxt.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/its-FOSS.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/terminal-root-press-release.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/welcome-to-the-nyxt-academy.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/aliquote.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/gopher.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles/lisp>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles/press>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles/release>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/articles/feature>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/newsletter-0.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/newsletter-01.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/newsletter-02.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/newsletter-03.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/article/newsletter-04.org>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/learn-lisp>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/forgot-password>
  #<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/register>)
 (#<QURI.URI.HTTP:URI-HTTPS https://nyxt.atlas.engineer/application/demeter.org/apply>)
 NIL)

Now that it's proven correct, I can work on performance (memoization and parallelization).

I'm still not working on the search itself, but computing the search space (the URLs).

Don't look too much at the code yet. It's mostly nasty.

aadcg avatar Jun 13 '22 14:06 aadcg

Woohoo, looks exciting already! :)

Ambrevar avatar Jun 13 '22 14:06 Ambrevar

@aadcg, good news: I'll quite likely open a PR with a prototype for configurable pure-Lisp search, so we can stich those two together and make it recursive :D

aartaka avatar Jun 13 '22 16:06 aartaka

It's going well. Still exploring.

I need to talk to @aartaka to understand a bit better what he's cooking with respect to search-buffer.

aadcg avatar Jun 20 '22 22:06 aadcg

Please reopen when ready.

jmercouris avatar Oct 28 '23 20:10 jmercouris