Importing browsing history, bookmarks and downloads' history from Luakit

[MuhammedZakir]

1. There is an option to import bookmarks from HTML. So that seems straightforward. However,

   1. when I tried creating a new bookmark in Nyxt (for testing), I couldn't create tags with hiphen(-) or underscore(_). Is it not supported?

2. In Luakit, history and session history (opened/closed tabs & windows) are kept in seperate files. But it looks like Nyxt keep them in a single file?

3. Downloads' history: where does Nyxt store this?

---

Hopping between browsers, syncing these things is the main problem! So I am thinking about using external software for these, so...

4. How difficult is it to use an external bookmarks manager like buku?
5. I would like to use an SQlite database for browsing history and downloads' history. How difficult is to do this?

[MuhammedZakir]

Actually, the question is not Luakit-specific, but rather, about how Nyxt stores these and how to manipulate these storages. And also about creating custom backends for these.

[aartaka]

1. Bookmarks are HTML-importable, yes. i. Yes, tags are not supporting underscores or hyphens yet. Should be trivial to add, though.
2. History and session history are not only united in Nyxt, they are also stored in a Nyxt-specific format. Thus, I'm not sure you can import Luakit history. By the way, what's the format Luakit uses for those?
3. Downloads history is not stored anywhere. It's only downloaded files that are loft on disk. Someday we may store downloads to history, but no promises.
4. I've not used external bookmark managers, but it should be hackable, for sure :)
5. SQLite database... We used to have DBs for data storage a long time ago, but it's long gone, and we don't even have SQLite as a dependency. So, I guess, it's only one-shot individual hacks that one can do that with :/

[MuhammedZakir]

• Bookmarks are HTML-importable, yes. i. Yes, tags are not supporting underscores or hyphens yet. Should be trivial to add, though.

• I've not used external bookmark managers, but it should be hackable, for sure :)

• SQLite database... We used to have DBs for data storage a long time ago, but it's long gone, and we don't even have SQLite as a dependency. So, I guess, it's only one-shot individual hacks that one can do that with :/

I will try hacking then! :-) I am still learning Lisp, so hopefully it won't be too hard.

• History and session history are not only united in Nyxt, they are also stored in a Nyxt-specific format. Thus, I'm not sure you can import Luakit history.

Is there a reason why it was done this way? IMHO, although browsing history and session are connected, they are a little different. I don't know how to put into words though.

By the way, what's the format Luakit uses for those?

Luakit uses SQlite dbs extensively -- for bookmarks, downloads' history, browsing history, cookies, etc. However, some stuff like session history, Luakit-managed settings (similiar to auto-config.lisp), etc. are saved in Lua-table-syntax-like structure (kinda similiar to how Nyxt does).

DB for history contains URI, title, no. of visits, and last visited date. Session file contains a list of windows -> each of them will have a list of opened tabs and a list of closed tabs -> each of them will have a navigation history. IIRC, it's a flat-history, rather than tree-like.

• Downloads history is not stored anywhere. It's only downloaded files that are loft on disk. Someday we may store downloads to history, but no promises.

I see. So list-downloads only show a list of files that are currently being downloaded?

[Ambrevar]

2. Thus, I'm not sure you can import Luakit history.

Yes we can, simply as history entries. The GHT entries alone are equivalent to a traditional history.

3. Downloads history is not stored anywhere. It's only downloaded files that are loft on disk. Someday we may store downloads to history, but no promises.

A download history would be really easy to implement.

In any case, our download manager is one of the weakest points of Nyxt and is in urgent need of an overhaul. I've got little time to work on it at the moment, but help is welcome of course!

[Ambrevar]

Is there a reason why it was done this way? IMHO, although browsing history and session are connected, they are a little different. I don't know how to put into words though.

See https://github.com/atlas-engineer/nyxt/tree/d14314817b8bd7ce39d9490f4577a403658082cd/libraries/history-tree for a rationale. This gives us strictly more power than the traditional approach.

[MuhammedZakir]

About history: If I have a list of URLs, how can I generate/create history file in GHT format? I don't mind creating "fake" session details.

[aartaka]

If you're on the bleeding edge Nyxt (or are willing to wait for the 3-pre-release-1 that heppens really soon), open lisp-repl, input this and hit C-return:

(nfiles:with-file-content (history (nyxt:history-file (nyxt:current-buffer)))
  (let ((urls (uiop:read-file-lines #p"/full/path/to/your/history.txt")))
    (dolist (url urls)
      (htree:add-entry history (make-instance 'nyxt::history-entry :url url)))))

This code implies that the URL file has all the URLs on separate lines.

---

For 2.*, the code is almost the same and should also be runnable in lisp-repl:

(nyxt:with-data-access (history (nyxt:history-path (nyxt:current-buffer))
                           :default (make-history-tree))
  (let ((urls (uiop:read-file-lines #p"/full/path/to/your/history.txt")))
    (dolist (url urls)
      (htree:add-entry history (make-instance 'nyxt::history-entry :url url)))))

[MuhammedZakir]

As a test, I tried this:

(nfiles:with-file-content (history (history-file (current-buffer)))
  (let ((url "https://example.com/?q=test"))
    (htree:add-entry history (make-instance 'nyxt::history-entry :url url))))

but, I am getting:

v2312.0 = nil
v2312.1 = #<undefined-function current-buffer {100F1E00D3}>

Edit: nyxt --version: Nyxt version d143148

Edit-2: Installed using

make all
make doc
DESTDIR= PREFIX=~/.local make install

I think I may have used DESTDIR=~/.local PREFIX= instead; I don't remember. Would that cause this problem?

[Ambrevar]

Are you running from the REPL? If so, you must switch to the nyxt-user package first:

(in-package :nyxt-user)

[MuhammedZakir]

From Nyxt > lisp-repl.

This works:

(nyxt-user::current-buffer)

This doesn't (gettting undefined function current-buffer):

(in-package :nyxt-user)
(current-buffer)

---

FWIW, directly using nyxt-user:: works. It looks like some strange issue with my installation?

(nfiles:with-file-content (nyxt-user::history (nyxt-user::history-file (nyxt-user::current-buffer)))
  (let ((url "https://example.com/?q=test"))
    (htree:add-entry nyxt-user::history (make-instance 'nyxt::history-entry :url url))))

---

Whenever I evaluate a Lisp expression, I am seing this warning printed to the console:

<WARN> [16:50:27] Warning: JavaScript error: GError: Domain: "WebKitJavascriptError", Code: 699, Message: nyxt:nyxt/repl-mode:lisp-repl:1:50: TypeError: null is not an object (evaluating 'document.querySelector('#evaluation-result-1882')')

[aartaka]

No, that might be a wrong default package used by Nyxt image. @Ambrevar, do we set it directly anywhere?

Those are interesting JavaScript errors... Seems like the cell focusing logic is somewhat broken there (;¬_¬)

EDIT: directory -> directly

[MuhammedZakir]

Here is the error printed to stdout:

; in:
;      FILES:WITH-FILE-CONTENT (COMMON-LISP::HISTORY
;                           (COMMON-LISP::HISTORY-FILE
;                            (COMMON-LISP::CURRENT-BUFFER)))
;     (COMMON-LISP::CURRENT-BUFFER)
;
; caught WARNING:
;   The function COMMON-LISP::CURRENT-BUFFER is undefined, and its name is reserved
;   by ANSI CL so that even if it were defined later, the code doing so would not
;   be portable.

;     (COMMON-LISP::HISTORY-FILE (COMMON-LISP::CURRENT-BUFFER))
;
; caught WARNING:
;   The function COMMON-LISP::HISTORY-FILE is undefined, and its name is reserved
;   by ANSI CL so that even if it were defined later, the code doing so would not
;   be portable.
;
; compilation unit finished
;   Undefined functions:
;     COMMON-LISP::CURRENT-BUFFER COMMON-LISP::HISTORY-FILE
;   caught 2 WARNING conditions
<WARN> [16:44:24] Warning: Error in s-exp evaluation: The function COMMON-LISP::CURRENT-BUFFER is undefined.

[aartaka]

Yes, then it uses common-lisp as the package for evaluation. Weird...

[hendursaga]

I for one wouldn't mind an SQLite storage option for Nyxt history, bookmarks, etc.

[Ambrevar]

current-buffer is exported. The following works for me in a REPL:

> (in-package :nyxt-user)
#<PACKAGE "NYXT-USER">
> (current-buffer)
NIL

@MuhammedZakir Can you share a complete recipe to reproduce your issue? You might be doing something wrong in the middle.

@hendursaga It's not hard to store as SQLite, but we would need a use case first. What would SQLite bookmark or history bring you?

[aartaka]

I for one wouldn't mind an SQLite storage option for Nyxt history, bookmarks, etc.

It looks to me as an extensions idea more than a core feature. The only reason I can think of to include this in the core is to make importing things from other browsers easier.

[MuhammedZakir]

current-buffer is exported. The following works for me in a REPL:

Which REPL? In Nyxt > REPL?

> (in-package :nyxt-user)
#<PACKAGE "NYXT-USER">
> (current-buffer)
NIL

@MuhammedZakir Can you share a complete recipe to reproduce your issue? You might be doing something wrong in the middle.

I have posted expressions I have tried above. Here is what I have tried to evaluate just now (in Nyxt > REPL).

Note: consecutive lines starting with ↑↓ means those lisp expressions are entered in a single "box".

↑↓ (in-package :nyxt-user)

v980 = #<package "NYXT-USER">

↑↓ (current-buffer)

v1038.0 = nil
v1038.1 = #<undefined-function current-buffer {100C7F3D33}>

↑↓ (in-package :nyxt-user)
↑↓ (current-buffer)

v1099.0 = nil
v1099.1 = #<undefined-function current-buffer {100D764FA3}>

An example of what Nyxt outputs to the console is posted above.

---

I for one wouldn't mind an SQLite storage option for Nyxt history, bookmarks, etc.

It looks to me as an extensions idea more than a core feature. The only reason I can think of to include this in the core is to make importing things from other browsers easier.

If the code you posted is valid in the Nyxt-sense, i.e. it is fine to add history entries like that, then importing is already easy. I don't understand why SQLite storage would make that easier. It would, however, make exporting easier, as reading a SQLite database is easier/simpler than reading some other custom format.

[aartaka]

@MuhammedZakir, @Ambrevar most probably meant the SLY/SLIME REPL that you get when you connect to Nyxt from Emacs (see documents/README for details). Nyxt built-in REPL has nyxt-user as a default (and, so far, unchangeable) package, so it might interfere with the expectations there.

[aartaka]

I've edited my code so that it should be safe to use from any package... hopefully :sweat_smile:

[hendursaga]

It looks to me as an extensions idea

@aartaka I did a little work on making an extension, actually, but I think it's out-of-date by now, and I'm sure the other devs would have a better idea of how to architect it than I at this point..

we would need a use case first.

@Ambrevar facilitating import / export is one idea, but also interoperability, I'd say. Ideally, yes, I could modify, augment, whatever to history and bookmarks and such, but I have other software and scripts I wrote that aren't in CL. In addition, isn't everything in-memory at this point? Like, sure, things can get (de)serialized, but how well would Lisp expressions in memory and on disk scale?

[Ambrevar]

In-memory data is necessarily Lisp expressions for our concern (history, bookmarks).

Importers and interop with other browser's data may fit more as an extension. What do you think?

One feature which is quite popular in other browsers is to import data from another browser, e.g. Firefox can (or could?) import from Chrome. Maybe for this use case adding SQLite support by default would make sense. But first we must work on the importer.

[aartaka]

It looks to me as an extensions idea

@aartaka I did a little work on making an extension, actually, but I think it's out-of-date by now, and I'm sure the other devs would have a better idea of how to architect it than I at this point..

Well, the best I can imagine is a couple of commands, not even belonging to a mode, so it's not much architecture there :)

I can help you with making it up-to-date if you point me at it!

we would need a use case first.

@Ambrevar facilitating import / export is one idea, but also interoperability, I'd say. Ideally, yes, I could modify, augment, whatever to history and bookmarks and such, but I have other software and scripts I wrote that aren't in CL. In addition, isn't everything in-memory at this point? Like, sure, things can get (de)serialized, but how well would Lisp expressions in memory and on disk scale?

Adding a bit more context there: it's not Lisp expressions in memory, it's Lisp data structures (which compilers know how to optimize, I'm guessing) in memory :)

[hgluka]

@Ambrevar @aartaka Hey, I'm working on an extension for this (importing history from other browsers). Currently, I have an import from Firefox that works!

Anyway, @jmercouris mentioned that maybe the importing function is valuable enough to justify being in Nyxt core, even though it introduces another dependency. He can probably explain it better.

What do you all think?

[aartaka]

If that dependency does not depend on some C library and is portable across OS-es and CL implementations, then why not!

[hgluka]

Well, it's cl-dbi (more specifically, datafly, which is a convenient wrapper around cl-dbi), so it needs libsqlite3 to work. If I'm reading things correctly though, WebKitGTK depends on it as well, so maybe it's not an issue?

[jmercouris]

It should not be an issue.

[hgluka]

Cool. In that case, I'll create a PR so I can get some feedback and we'll go on from there.

[jmercouris]

Perfect, thank you!

[Ambrevar]

Oh, sorry, that thread fell under my radar.

So to answer the question about cl-dbi, I would use cl-sqlite instead since it's much lighter. If I'm not mistaken we don't need a wrapper around Postgres or MySQL.