skeeto
How to automatically update feeds in the background?
Is there a way to automatically update feeds in the background with Elfeed (every hour, for example)?
My wishes:
- I'd like to replace Tiny Tiny RSS, which uses a systemd service to update its feeds but also requires an entire LAMP stack
- I prefer to avoid missing any published feeds, which can happen when you forget to update them for days
- Feeds should be updated in the background if possible, without interfering with Emacs usage
There is currently no automatic update mechanism in Elfeed. You could use a run-at-time timer to do it:
;; Run `elfeed-update' every 8 hours (run-at-time nil (* 8 60 60) #'elfeed-update)
Since you're worried about missing things, you might want to follow this up with an elfeed-db-save to explicitly write out what was fetched. Since elfeed-update is asynchronous, you'd need to wait until it was done. Put a function in elfeed-update-hooks that saves the database when the queue is has been emptied (elfeed-queue-count-total).
If you do this in your active Emacs session, this would disrupt Emacs while you're using it. Maybe you could work around this with a generous idle timer. Further, saving a large database (one large blocking operation) will be more disruptive than fetching feeds (lots of little blocking operations). Emacs 26 threads cannot help with any of this.
Alternatively you could use a cron job and run an isolated Emacs instance to fetch everything, then exit. In theory you could do this in a batch instance by using accept-process-output to "block" the main thread while it runs, allowing you to do it without a controlling pty (it's pretty complicated):
Turning Asynchronous into Synchronous in Elisp https://nullprogram.com/blog/2013/01/14/
However, you must not have the Elfeed database loaded in another instance of Emacs while this happens. The database is essentially single user, and the different Emacs instances won't see each others' changes. (It won't corrupt the database, you'll see one version or the other.) That database race condition is probably worse than disrupting your running Emacs instance, though.
As "Elfeed was inspired by notmuch" I wonder if following the notmuch model of keeping the emacs interface, and the actual "backend" implementation separate would help here? Ie. factoring out the feed updating logic to a separate program which can be called by emacs or independently.
Elfeed is already split up this way, though without any native bits. At one point I considered making a diagram for the README, and perhaps I should.
At the bottom level is the database, "elfeed-db". It defines entry and feed structs, it persists instances of these structs, and it provides an index for efficient look-ups. It doesn't know anything about RSS, Atom, XML, HTTP, or any other part of Elfeed. It just knows about these Lisp objects.
Off to the side is a URL (well, HTTP-only at the moment) fetching engine, "elfeed-curl". It doesn't know about RSS, Atom, XML, feeds, the database, or the rest of Elfeed. It just knows how to asynchronously request curl to fetch content at a particular URL, and a little bit about HTTP headers. If curl isn't available, Elfeed will fall back on url-retrieve instead for this role (which doesn't work nearly as well).
At the core is "elfeed". It knows how to parse various versions of RSS and Atom to produce entry and feed structs. It passes these objects over to elfeed-db for storage. It knows about your feed list, and when you ask it to update, it uses elfeed-curl to each feed, which it then parses. It knows nothing at all about the user interface. It's the other way around: the UI uses public hooks in Elfeed in order to follow the action. (There's a tiny bit of trickery to make this work properly under byte compilation, and to allow "elfeed" to be the public entrypoint.)
The last two parts, "elfeed-search" and "elfeed-show", are the UI. They know all about the other parts of Elfeed and hook into them in order to keep the display up to date. They also know about each other, working in conjunction with one another — though "elfeed-search" just barely knows about "elfeed-show". In theory you could completely toss elfeed-search and elfeed-show and replace it with your own custom interface without changing anything about Elfeed. The only thing special about them is that they're automatically loaded when you say "(require 'elfeed)" or "M-x elfeed".
To truly have a separate database like notmuch, accessible in parallel from multiple clients, would require either a native backend or a standalone Emacs database server (e.g. serving client database requests over sockets). Emacs Lisp itself lacks the primitives to coordinate with peer processes on a common database, necessitating a native component. My original plan was to use EmacSQL for this, but I abandoned that idea since it cost more than it was worth.
@skeeto following your reply I implemented the following script below. Can you please examine if there isn't anything missing? I try to implement background auto feeds fetching every 4h.
(add-to-list 'elfeed-update-hooks 'elfeed-update)
(run-with-timer 0 (* 60 60 4) 'elfeed-update)
Definitely don't to put elfeed-update in elfeed-update-hooks. That's like calling itself recursively: an update will trigger multiple more updates immediately.
You've got the right idea with run-with-timer.
Definitely don't to put elfeed-update in elfeed-update-hooks. That's like calling itself recursively: an update will trigger multiple more updates immediately.
You've got the right idea with run-with-timer.
Thank you. I ended up updating feeds manually, since it gives more control of feeds view.