chenyanming
New Wallabag client
Hi @chenyanming,
Thank you for writing wallabag.el! This post is about a different Emacs package, but I don't think it would exist in its current form without this one.
I use Wallabag regularly (outside of Emacs) and tried to use wallabag.el, but discovered that it was unusable for my rather particular situation and was missing features I needed. Specifically:
-
I have been using read-it-later style apps since 2005 and been migrating my data every time I switch apps. As a result, I have ~15,000 articles in my Wallabag DB. This causes multiple problems:
-
I can't initialize the Wallabag DB because it tries to fetch all the entries in one HTTP request, which obviously fails.
-
Wallabag.el loads the entire database into memory twice, the second time as text properties in the wallabag-search buffer. So this consumes 500+ MB of RAM and the Wallabag buffer is unusable. Since Emacs does not like a 500+ MB font-locked buffer, Emacs itself becomes unusable.
-
-
There is no way to query the database except using a general full text search. This makes it difficult to filter by date, author etc, which I rely on to find things in a listing of 15K.
-
The DB update logic often misses entries or changes to entries if they are added/changed outside of wallabag.el. (This depends on the time-order in which entries are added/changed.) It does a full sweep to track certain changes but this is a bad idea when you have ~15,000 entries. The full sweep also fails because of the HTTP request thing.
-
Poor http request error handling and recovery.
Earlier this year I started writing some patches for wallabag.el to fix these issues, but this required pretty fundamental design changes, and I eventually had to give up on the idea of submitting a PR. I ended up writing a new Wallabag client from scratch.
Things that it has in common with wallabag.el:
- The visual design and UI is inspired by wallabag.el and Elfeed.
- The sqlite db schema is mostly reused from wallabag.el (for compatibility), but I added some new fields I needed.
The rest is all different:
- No shared code with wallabag.el, except for three face definitions.
- No emojis or sidebar menu.
- Incomplete tag support. (I don't use Wallabag tags)
- Fully featured search, including debouncing and some other tweaks to ensure Emacs is never unresponsive:
https://github.com/chenyanming/wallabag.el/assets/8607532/8761c303-4d7c-4350-9ed2-1fa698d0a999
-
Minimal footprint: Does not load the db into memory, queries whatever is required instead. The wallabag-search buffer memory size with all 15000 entries showing is now about 25 MB, although with the new search filters there's never more than ~100 entries showing, which is < 800 KB.
-
Automatically saves your place in articles so you can resume where you stopped reading.
-
Handles updating any number of entries without http errors.
-
Different update logic. Using some API endpoints available in newer versions of the Wallabag API, all actions are consistent across clients (no time-ordering dependence), and much less data is transferred compared to wallabag.el.
-
Wallabag http requests: Better error handling, automatic oauth token refresh, retries on failure etc.
-
Many more quality of life improvements (improved header status/messages, bookmark support, pixel-precision smooth scrolling, imenu support, Embark actions support, better EWW integration etc)
I have been using this client for a few months, and am considering making it public now. I hope you don't mind me publishing it -- I see that wallabag.el is GPLv3 licensed, but wanted to check with you in person since my package is clearly inspired by it.
Thanks again for your work on wallabag.el!
@karthink Sure! I would love to see the new wallabag client! I did not have much time to add new features for my package as well. By the way, I like your gptel!
I might be late to the party, but it would be cool to add wallabag to https://github.com/fasheng/elfeed-protocol
Wow! This is awesome. See this kind of collaboration in real time makes me recover my faith in humanity. Thank you both @karthink and @chenyanming
@karthink my workflow and experience is very similar to you. Even the android client takes forever in my devices the first time (and in some devices it never loads :-( )
@karthink when are you planning to release the new client? It will be available in melpa? Will support integration with vertico, consult, etc?
@karthink Sure! I would love to see the new wallabag client! I did not have much time to add new features for my package as well. By the way, I like your gptel!
Thank you @chenyanming.
I uploaded it to github here.
Even the android client takes forever in my devices the first time (and in some devices it never loads :-( )
@karthink when are you planning to release the new client? It will be available in melpa? Will support integration with vertico, consult, etc?
@nanounanue The new package takes 4-5 minutes to fetch ~15K entries and populate an empty DB in my case.
The package is now live. I don't have any immediate plans to add it to MELPA. It doesn't have any vertico/consult integration, I'm not sure what that would mean.
I might be late to the party, but it would be cool to add wallabag to https://github.com/fasheng/elfeed-protocol
Wallabag provides RSS feeds that you can just subscribe to with Elfeed:
I might be late to the party, but it would be cool to add wallabag to https://github.com/fasheng/elfeed-protocol
Wallabag provides RSS feeds that you can just subscribe to with Elfeed:
Thanks for sharing that! It's more of using the API for a first class integration with wallabag as the back end like wombag, rather than having tags stored in elfeed, it's just using elfeed as a front-end(my simple understanding at-least).
Thanks @karthink pointing out issues, haha, I recently developed https://github.com/chenyanming/paw, and integrated it with wallabag.el. I addressed the memory issues with paginations support. Now it is able to handle very large database, since wallabag-search-page-max-rows is the maximum entries it will load. Real time filting (unlike wombag, wallabag.el only use space to seperate keywords for searching multiple columns) is also handled properly. Cheers~