nfiles

nfiles copied to clipboard

#+TITLE: NFiles

NFiles is a Common Lisp library that deals with customizable path resolution, file persistence and loading.

Its main use case is help manage user-centric files like configuration files.

In some aspects, it can be seen as "Common Lisp 'logical pathnames' over CLOS".

** Goals

• Performance ::
  • Data is not persisted to disk if it does not need to.
  • Files are read only once (unless modified externally).
• Extensibility :: Persist any data structure the way you want it.
• Reliability :: no corruption and no data loss should occur.

** Features

• Dynamic and customizable path expansion.
• Extensible serialization and deserialization.
• Cached reads and writes :: When a =file= object expands to the same path as another one, a =read= or =write= on it won't do anything in case there was no change since last write.
• (Experimental!) On-the-fly PGP encryption.
• Profile support.
• On read errors, existing files are backed up.
• On write errors, no file is written to disk, the existing file is preserved.
• A =remote-file= can point to a URL, which is automatically downloaded if the local file is not found.

** Motivation

This package was developed after dealing with a problem when delivering Common Lisp images: when an image is generated, path expansion may already be resolved and thus hard-coded within the image, which makes it unfit for delivery. Consider this:

#+begin_src lisp

(defvar foo-path (uiop:xdg-config-home)) FOO-PATH foo-path #P"/home/johndoe/.config/" #+end_src

Now if I ship this image to my friend Kaboom, =foo-path= will expand to

#+begin_src lisp #P"/home/johndoe/.config/" #+end_src

on their machine instead of the expected

#+begin_src lisp #P"/home/kaboom/.config/" #+end_src

** Examples

A basic session:

#+begin_src lisp (defvar config-file (make-instance 'nfiles:config-file :base-path #p"my-app/init.lisp"))

(nfiles:expand config-file) ; => #P"/home/johndoe/.config/my-app/init.lisp"

(setf (nfiles:content config-file) "Hello file!") ; The file is written to disk.

(nfiles:content config-file) ; => "Hello file!" #+end_src

The following convenience macro ensures the file is updated when done with the body:

#+begin_src lisp (nfiles:with-file-content (content config-file) (format t "Length: ~a~%" (length content)) (setf content (serapeum:string-replace "file" content "config"))) #+end_src

The =with-paths= helper allows for let-style bindings of the expanded paths:

#+begin_src lisp (let ((file1 (make-instance 'nfiles:file)) (file2 (make-instance 'nfiles:file :base-path #p"alt"))) (nfiles:with-paths ((path1 file1) (path2 file2)) (list path1 path2))) #+end_src

A =remote-file= works the same but needs some specialization:

#+begin_src lisp (defmethod nfiles:fetch ((profile nfiles:profile) (file remote-counter-file) &key) (dex:get (nfiles:url file)))

;; Optional: (defmethod nfiles:check ((profile nfiles:profile) (file remote-counter-file) content &key) (let ((path (nfiles:expand file))) (ironclad:byte-array-to-hex-string (ironclad:digest-file :sha3 path))))

(let ((file (make-instance 'nfiles:remote-file ;; The URL to download from if the file is not found on disk. :url (quri:uri "https://example.org") ;; Without base-path, the file won't be saved to disk. :base-path #p"/tmp/index.html" :checksum "794df316afac91572b899b52b54f53f04ef71f275a01c44b776013573445868c95317fc4a173a973e90addec7792ff8b637bdd80b1a6c60b03814a6544652a90"))) ;; On access, file is automatically downloaded if needed and the checksum is verified: (nfiles:content file) ;; ... ) #+end_src

See the [[file:package.lisp][package]] documentation for a usage guide and more examples.

** Configuration

NFiles was designed with configurability in mind. All configuration happens through subclassing of the =file= and =profile= classes together with method specialization.

All configuration methods are specialized against =profile= and =file= so that the user can easily compose the behaviour:

• Profile-customization impacts all files using that profile;
• File-customization impacts the files of that specific type (or subtype) regardless of their profile.

Of course you can specialize against both!

The specialization methods are divided into the following:

• =resolve= :: This is where path resolution is done. On call site, prefer the =expand= convenience wrapper.

• =deserialize= and =serialize= :: This is how the content is transformed to the file on disk. These functions are meant to be called by the =read-file= and =write-file= methods.

• =read-file= and =write-file= :: This is how the file is read and written to disk. These functions are responsible for calling the =deserialize= and =serialize= methods.

• =fetch= :: This generic function is only called for =remote-file= objects. You must define its methods. It does not have any method by default so as to not burden NFiles with undesirable dependencies.

• =check :=: Like =fetch=, this generic function is only called for =remote-file= objects to test the integrity of the downloaded file. You must define its methods. It does not have any method by default so as to not burden NFiles with undesirable dependencies.

** Conditions and restarts

Some NFiles-specific conditions are raised in case of exceptional situations to provide for interactive and customizable behaviour:

• =external-modification= :: The file was modified externally. See the =on-external-modification= slot to automate what to do in this case.

• Read error restarts can also customized, see the =on-read-error= slot to automate what to do in this case.

• =process-error= :: This may be raised for instance when =gpg= fails to encrypt. The =use-recipient= restart is provided to retry with the given recipient.

** Shadowing

NFiles 1 shadows =cl:delete=, thus you should not =:use= the package (as with any other library anyways).

** Platform support

It's pure Common Lisp and all compilers plus all operating systems should be supported.

Some notes:

• All compilers but SBCL depend on [[https://github.com/sionescu/iolib][IOlib]] to preserve file attributes.
• Android devices also depend on [[https://github.com/sionescu/iolib][IOlib]] to preserve file attributes, regardless of the compiler.
• File attributes might not be preserved on Windows.

** Roadmap

• Improve PGP support.
• Support OS-level locks (à-la Emacs / LibreOffice).
• Improve portability.
  • In particular, preservation of file attributes may not work on Windows.
• Compressing =write-file= and =read-file= (for instance with zstd / lz).
  • But is it such a good idea? Users should prefer compression at the level of the file system.

** Change log

*** 1.1.4

• Remove =NASDF= as a dependency.

*** 1.1.3

• Ensure a =cl:pathname= is returned from =resolve=.
• Complete some missing documentation.

*** 1.1.2

• Add restarter functions for =ask=, =reload=, etc. Mind that =cl:delete= is now shadowed (this was necessary to preserve backward compatibility). Do not =:use= the package!

• Switch from =hu.dwim.defclass-star= to [[https://github.com/atlas-engineer/nclasses/][nclasses]].

*** 1.1.1

• Allow path expansion for =virtual-file= (as in 1.0.0).

  This restores the usefulness of virtual-files, namely to handle the path-expansion business while deferring the read/write business to a third-party.

  =virtual-profile= still nullifies path expansions (as in 1.1.0).

*** 1.1.0

• Add support for Android.
• Nullify path expansion for =virtual-file= and =virtual-profile=.
• Ensure that the =deserialize= method of =virtual-file= and =virtual-profile= return nil.
• Fix =basename= corner case.
• Add report messages to all restarts.

** History

NFiles was originally developed for user file management in [[https://nyxt.atlas.engineer][Nyxt]], so the "N" may stand for it, or "New", or whatever poetic meaning you may find behind it!