graven-image

graven-image copied to clipboard

#+TITLE:Graven Image

#+begin_quote Thou shalt not make unto thee any graven image, or any likeness of any thing that is in heaven above, or that is in the earth beneath, or that is in the water under the earth. #+end_quote

Graven Image is a Common Lisp portability library (a less fancier name might've been =trivial-debugging=) for better interaction and debugging of a running Lisp image. One can inspect and debug all the things under heaven and [[https://www.corecursive.com/lisp-in-space-with-ron-garret/][above it]]—all in their own REPL-resident Lisp image!

Graven Image reuses compiler internals to improve/redefine the existing standard functions. This "improvement" often comes at a cost of changing the API of a function (for better customizability) or making it slightly less reliable (due to unstable, compiler-internal, or otherwise hacky implementation.)

The library is purposefully limited in scope:

• Improving the standard functions has a priority over introducing new ones.
  • [[https://github.com/m-n/repl-utilities][repl-utilities]] as a contrasting approach: it defines lots of "DWIM" functions, significantly altering the REPL interaction.
  • There are helpers like =function-lambda-list*= in Graven Image, but these are merely aliases/one-liners over standard/improved functions.
    • I get carried away sometimes, though. =benchmark*= is one of such [[https://xkcd.com/356/][nerd snipes]]. But still, it's quite close to the underlying =with-time*= 😉
• The interaction paradigm of the improved functions should stay standard (i.e. use =query-io= where standard requires =query-io=.)
  • This is to ensure that libraries like [[https://github.com/atlas-engineer/ndebug/][Ndebug]] can rely on standard facilities safely when used with Graven Image.
• Function arglist can be modified for convenience, but it should preferably stay as close to standard/implementation-specific arglist as possible.
• Graven Image strives to not modify the REPL/image/shell in any way, relying on implementation defaults instead.
  • [[https://github.com/ciel-lang/CIEL][CIEL]] is taking a different direction: redefining the REPL for increased utility.
  • [[https://github.com/vseloved/flight-recorder][flight-recorder]] uses a separate shell script to add a new functionality.
• Portably reusing implementation-specific functionality is better than re-implementing it. But if some functionality is e.g. unique to SBCL, it might be dropped as non-portable.
  • [[https://github.com/TeMPOraL/tracer][tracer]], [[https://github.com/fukamachi/supertrace][supertrace]], [[https://github.com/40ants/cl-flamegraph][cl-framegraph]], and other non-portable SBCL-specific improvements.

• Getting started

Clone the Git repository: #+begin_src sh git clone --recursive https://github.com/aartaka/graven-image ~/common-lisp/ #+end_src

And load =:graven-image= in the REPL: #+begin_src lisp (asdf:load-system :graven-image) ;; or, if you use Quicklisp (ql:quickload :graven-image) #+end_src

You can also install Graven Image via Guix, using the bundled =guix.scm= file: #+begin_src sh guix package -f guix.scm #+end_src

Someday (after at least minor version bump) I'll send a patch to Guix so that you can also install it with =guix install=. Until then—stay tuned (or send the patch yourself—I don't mind.)

• Convenient =use=

If you want your REPL to start with Graven Image constructs accessible without package prefix, then simply use the package directly. #+begin_src lisp (asdf:load-system :graven-image) ;; Imports external symbols of Graven Image into the current package ;; (CL-USER?) Safe, because Graven Image shadows no CL symbols. (use-package :graven-image) #+end_src

Or, if you're more orderly and disciplined about packages than I am, you can always use =trivial-package-local-nicknames= and define a shorter Graven Image nickname: #+begin_src lisp (trivial-package-local-nicknames:add-package-local-nickname :g :graven-image :cl-user) #+end_src

If you want to replace the standard utilities with Graven Image ones, you can safely do so with the familiar: #+begin_src lisp ;; For functions (fmakunbound 'apropos) (setf (fdefinition 'apropos) (fdefinition 'gimage:apropos*)) ;; For macros (setf (macro-function 'time) (macro-function 'gimage:time*)) #+end_src

• Enhanced functions (mostly from ANSI CL [[https://cl-community-spec.github.io/pages/Debugging-Utilities.html][Debugging Utilities]] chapter)

The functions that Graven Image exposes are safely =:use=-able star-appended functions/macros (i.e. =describe*= instead of =describe=). Currently improved ones are:

• =y-or-n-p*= / =yes-or-no-p*=
• =apropos*= / =apropos-list*=
• =function-lambda-expression*=
• =time*=
• =describe*= / =inspect*=
• =dribble*=
• =documentation*=

All of the functions exposed by Graven Image are generics, so one can easily define =:around= and other qualified methods for them.

** =y-or-n-p*=, =yes-or-no-p*= (generic functions)

Signature: #+begin_src lisp y-or-n-p* &optional control &rest arguments => generalized-boolean yes-or-no-p* &optional control &rest arguments => generalized-boolean #+end_src

Improvements are:

• Both functions accept options from =graven-image:yes-or-no-options=, thus allowing for "nope" or "ay" to be valid responses too.
• Both functions mean the same now, because it makes no sense in differentiating them (and because most Emacs users use a magical =(fset 'yes-or-no-p 'y-or-n-p)= in their config, setting the precedent for shorter yes/no queries).
• No beeps (just define a =yes-or-no-p* :before= method to add beeps if you like 'em; see the "Customization" section below).

** =apropos-list*=, =apropos*= (generic functions)

Signature: #+begin_src lisp apropos-list* string &optional (package nil) exported-only docs-too => list of symbols apropos* string &optional (package nil) exported-only docs-too => no values #+end_src

=apropos-list*= now allows listing exported symbols only (with =exported-only=), which was a non-portable privilege of SBCL/Allegro until now. Search over docs (more intuitive for =apropos(-list)*= than mere name search) is possible with =docs-too=.

Based on this foundation, =apropos*= lists symbols with their types, values, and documentation, so that implementation-specific formats are gone for a better and more unified listing:

#+begin_src lisp (apropos* :max) ;; MAX [FUNCTION (NUMBER &REST ;; MORE-NUMBERS) : Return the greatest of its arguments; among EQUALP greatest, return...] ;; :MAX [SELF-EVALUATING] ;; CFFI::MAX-ALIGN ;; SB-ASSEM::MAX-ALIGNMENT [CONSTANT = 5] ;; ... ;; SB-C::MAXES ;; ALEXANDRIA:MAXF [MACRO (#:PLACE &REST NUMBERS) : Modify-macro for MAX. Sets place designated by the first argument to the...] ;; SB-KERNEL::MAXIMAL-BITMAP ;; ... ;; SB-LOOP::LOOP-ACCUMULATE-MINIMAX-VALUE [MACRO (LM OPERATION FORM)] ;; SB-LOOP::LOOP-MAXMIN-COLLECTION [FUNCTION (SPECIFICALLY)] ;; SB-LOOP::LOOP-MINIMAX [CLASS (STRUCTURE-OBJECT)] ;; ... #+end_src

** =function-lambda-expression*= (generic function)

Signature: #+begin_src lisp function-lambda-expression* function/macro/method/symbol &optional force => list, list, symbol, list ;; Alias: lambda-expression* function/macro/method/symbol &optional force => list, list, symbol, list #+end_src

This function tries to read source files, process the definitions of functions, and build at least a barebones lambda from the arglist and documentation of the function. So that CL =function-lambda-expression= returns: #+begin_src lisp (function-lambda-expression #'identity) ;; => NIL, T, IDENTITY (function-lambda-expression #'print-object) ;; => NIL, T, PRINT-OBJECT #+end_src

While the new Graven Image =function-lambda-expression= now returns: #+begin_src lisp (function-lambda-expression* #'idenitity) ;; => (LAMBDA (THING) "This function simply returns what was passed to it." THING), ;; NIL, IDENTITY, (FUNCTION (T) (VALUES T &OPTIONAL)) (function-lambda-expression* #'print-object t) ; Notice the T for FORCE, to build a dummy lambda. ;; => (LAMBDA (SB-PCL::OBJECT STREAM)), NIL, PRINT-OBJECT, (FUNCTION (T T) *) #+end_src

Which means:

• =identity= is actually not a closure, and has a reliable source!
• =print-object= is a generic and thus is not really inspectable, so we build a dummy lambda for it when =force= argument is provided.
  • This might be a questionable choice, but it at least allows us to get function arglists from =function-lambda-expression= in a portable-ish way. The standard doesn't provide us with much ways to know an arglist of a function beside this.

*** Return values

Things that =function-lambda-expression*= now returns are:

• Lambda expression.
  • For lambda functions, their source.
  • For regular functions, their =defun= turned into a =lambda=.
  • For anything else, a constructed empty =(lambda (arglist...) documentation nil)= (only when =force= is T).
  • Or, in case all the rest fails, NIL.
• Whether the thing is a closure
  • If it is, might return an alist of the actual closed-over values, whenever accessible (not for all implementations).
  • If closed-over values are not accessible, returns T.
  • If it's not a closure, returns NIL.
• Function name. Mostly falls back to the standard =function-lambda-expression=, but also inspects implementation-specific function objects if necessary.
• Function type, whenever accessible (SBCL and ECL).

*** Helpers

Based on these new features of =function-lambda-expression*=, here are some Graven Image-specific helpers:

• =function-lambda-list*= :: Get the lambda list of a function.
  • =function-arglist*= :: Alias.
  • =lambda-list*= :: Alias for =function-lambda-list*=.
  • =arglist*= :: Alias.
• =function-name*= :: Get the name of a function.
• =function-type*= :: Get its ftype.

#+begin_src lisp function-lambda-list* function => list function-arglist* function => list lambda-list* function => list arglist* function => list function-name* function => symbol function-type* function => list #+end_src

** =time*= (macro)

Signature: #+begin_src lisp time* &rest forms => return-values #+end_src

The improved =time*= from Graven Image reuses as much implementation-specific APIs as possible, with the predictable output format.

And it also allows providing several forms, yay!

*** =benchmark*= (macro)

Signature: #+begin_src lisp benchmark* (&optional (repeat 1000)) &body forms => return-values #+end_src

While =time*= is the standard benchmarking/profiling solution, it's almost always too simple for proper benchmarking. Most systems getting complex enough end up with some form of custom benchmarking. Shinmera's [[https://github.com/Shinmera/trivial-benchmark/][trivial-benchmark]] is one such example. Graven Image =benchmark*= is heavily inspired by =trivial-benchmark=, but has a more portable foundation in the form of =with-time*=.

As many other benchmarking macros, =benchmark*= repeats its body a certain number of times, collecting timing stats for every run, and then prints aggregate statistics for the total runs. #+begin_src lisp (gimage::benchmark* (20) ;; Repeat count. (loop for i below 1000 collect (make-list i) finally (return 1))) ;; Benchmark for 20 runs of ;; (LOOP FOR I BELOW 1000 ;; COLLECT (MAKE-LIST I) ;; FINALLY (RETURN 1)) ;; - MINIMUM AVERAGE MAXIMUM TOTAL ;; REAL-TIME 0.0 0.00175 0.019 0.035 ;; USER-RUN-TIME 0.000668 0.0016634 0.016315 0.033268 ;; SYSTEM-RUN-TIME 0.0 0.00021195 0.003794 0.004239 ;; GC-RUN-TIME 0.0 0.00085 0.017 0.017 ;; BYTES-ALLOCATED 7997952.0 8008154.5 8030464.0 160163090.0 #+end_src

*** =with-time*= (macro)

Signature: #+begin_src lisp with-time* (&rest time-keywords) (&rest multiple-value-args) form &body body #+end_src

As the implementation detail of =time*= and =benchmark*=, =with-time*= allows to get the timing data for interactive querying. =time-keywords= allow =&key=-matching the timing data (like =:gc= time or bytes =:allocated=) for processing in the body. While =multiple-value-args= allow matching against the return values of the =form=. So we get best of the both worlds: timing data and return values. This flexibility enables =time*=, with its requirements of printing the data and returning the original values at the same time.

For example, here's how one would track the allocated bytes and garbage collection times when running a cons-heavy code: #+begin_src lisp (gimage:with-time* (&key aborted gc-count gc allocated) (lists lists-p) (loop for i below 1000 collect (make-list i :initial-element :hello) into lists finally (return (values lists t))) (unless aborted (format t "Bytes allocated: ~a, GC ran ~d times for ~a seconds" allocated gc-count gc))) ;; Bytes allocated: 7997952, GC ran NIL times for 0 seconds #+end_src

** =describe*= (generic function)

Signature: #+begin_src lisp describe* object &optional (stream t) respect-methods #+end_src

Describes the =object= to the stream, but this time with portable format of description (determined by =graven-image:description*= and specified for many standard classes) and with predictable set of properties (=graven-image:fields*=). In Graven Image, both =describe= and =inspect= have the same format and the same set of fields.

As a note of respect to the original =describe=, Graven Image one allows to reuse the =describe-object= methods defined for user classes. To enable this, pass T to =respect-methods=.

*** =graven-image:fields*= (generic function)

Signature: #+begin_src lisp fields* object &key strip-null &allow-other-keys #+end_src

Returns an undotted alist of properties for the =object=. Custom fields provided by Graven Image are named with keywords, while the implementation-specific ones use whatever the implementation uses. Arrays and hash-tables are inlined into fields to allow indexing these right from the inspector.

See =fields*= documentation for more details.

*** =graven-image:description*= (generic function)

Signature: #+begin_src lisp description* object &optional stream #+end_src

Concise and informative description of =object= to the =stream=. Useful information from most of the implementations tested—united into one description header.

** =inspect*= (generic function)

Signature: #+begin_src lisp inspect* object &optional strip-null #+end_src

New'n'shiny =inspect*= has:

• Most commands found in other implementation, with familiar names.
• Abbreviations like =H -> HELP= (inspired by SBCL).
• Ability to set object field values with =(:set key value)= command (inspired by CCL).
• Built-in pagination with ways to scroll it (=:next-page=, =:previous-page=, =:home=) and change it (=:length=).
• Property indexing by both integer indices and property names (with abbreviations for them too!).
• Ability to ignore =nil= properties with =strip-null= argument (inspired by SBCL). On by default!
• And the ability to evaluate arbitrary expressions (with =:evaluate= command or simply by inputting something that doesn't match any command).

And here's a help menu of the new =inspect*= (in this case, inspecting =readtable=), just to get you teased:

#+begin_src This is an interactive interface for 5 Available commands are: :? Show the instructions for using this interface. :HELP Show the instructions for using this interface. :QUIT Exit the interface. :EXIT Exit the interface. (:LENGTH NEW) Change the page size. (:WIDTH NEW) Change the page size. (:WIDEN NEW) Change the page size. :NEXT Show the next page of fields (if any). :PREVIOUS Show the previous page of fields (if any). :PRINT Print the current page of fields. :PAGE Print the current page of fields. :HOME Scroll back to the first page of fields. :RESET Scroll back to the first page of fields. :TOP Scroll back to the first page of fields. :THIS Show the currently inspected object. :SELF Show the currently inspected object. :REDISPLAY Show the currently inspected object. :SHOW Show the currently inspected object. :CURRENT Show the currently inspected object. :AGAIN Show the currently inspected object. (:EVAL EXPRESSION) Evaluate the EXPRESSION. :UP Go up to the previous level of the interface. :POP Go up to the previous level of the interface. :BACK Go up to the previous level of the interface. (:SET KEY VALUE) Set the KEY-ed field to VALUE. (:MODIFY KEY VALUE) Set the KEY-ed field to VALUE. (:ISTEP KEY) Inspect the object under KEY. (:INSPECT KEY) Inspect the object under KEY. :STANDARD Print the inspected object readably. :AESTHETIC Print the inspected object aesthetically.

Possible inputs are:

• Mere symbols: run one of the commands above, matching the symbol.
  • If there's no matching command, then match against fields.
    • If nothing matches, evaluate the symbol.
• Integer: act on the field indexed by this integer.
  • If there are none, evaluate the integer.
• Any other atom: find the field with this atom as a key.
  • Evaluate it otherwise.
• S-expression: match the list head against commands and fields, as above.
  • If the list head does not match anything, evaluate the s-expression.
  • Inside this s-expression, you can use the `$' function to fetch the list of values under provided keys. #+end_src

** =dribble*= (generic function)

Signature: #+begin_src lisp dribble* &optional pathname (if-exists :append) #+end_src

Dribble the REPL session to =pathname=. Unlike the implementation-specific =dribble=, this one formats all of the session as =load=-able Lisp file fully reproducing the session. So all the input forms are printed verbatim, and all the outputs are commented out.

Beware: using any interactive function (like =inspect= etc.) breaks the dribble REPL. But then, it's unlikely one'd want to record interactive session into a dribble file.

** =documentation*= (generic function)

Signature: #+begin_src lisp documentation* object &optional (doc-type t) doc* object &optional (doc-type t) #+end_src

Improved version of =documentation=. Two main improvements are: =doc-type= is now optional, and =doc*= alias is available for convenience.

documentation.lisp also defines more =documentation= methods (and respective =setf= method) to simplify documentation fetching and setting. In particular, method on =(symbol (eql t))= to simplify symbol documentation search; and =(t (eql 'package))= with a new doc-type for package documentation convenience.

** =break*= (macro)

Signature: #+begin_src lisp break* &rest arguments #+end_src

A more useful wrapper for =break=, listing the function it's called from and the provided symbol values. See examples in the docstring.

• Customization

Graven Image is made to be extensible. That's why most of the improved functions are generic: one can define special methods for their data and patch the behavior with =:before=, =:after=, and =:around= methods. Most of Graven Image functions mention the variables/things influencing them in the docstring. Here's a set of useful customizations:

** Beeping before =yes-or-no-p*=

Restoring the standard-ish (beeping with bell (ASCII 7) character) behavior: #+begin_src lisp (defmethod gimage:yes-or-no-p* :before (&optional control &rest arguments) (declare (ignore control arguments)) (write-char (code-char 7) query-io) (finish-output query-io)) #+end_src

** Changing the accepted yes/no options for =yes-or-no-p*= and =y-or-n-p*= #+begin_src lisp ;; Make it strict yes/no as per standard. (defmethod gimage:yes-or-no-p* :around (&optional control &rest arguments) (declare (ignore control arguments)) (let ((gimage:yes-or-no-options '(("yes" . t) ("no" . nil)))) (call-next-method)))

;; Add more yes/no options (Russian, for example). (defmethod gimage:y-or-n-p* :around (&optional control &rest arguments) (declare (ignore control arguments)) (let ((gimage:yes-or-no-options (append gimage:yes-or-no-options '(("да" . t) ("ага" . t) ("нет" . nil) ("не" . nil) ("неа" . nil))))) (call-next-method))) #+end_src

** Sorting =apropos-list*= lists

Implementations are not good at sorting things, and their results are not often useful. Sorting things the way one needs is a useful extension. Here's a simple yet effective =:around= method that sorts things by =string= occurence: #+begin_src lisp (defmethod gimage:apropos-list* :around (string &optional packages external-only docs-too) "Sort symbols by the relation of subSTRING count to the length of symbol." (declare (ignorable packages external-only docs-too)) (let ((result (call-next-method))) (sort (remove-duplicates result) ;; For more comprehensive matching, see ;; a1b4ebd649e0268b1566e80709e7cea41363d006 and other commits ;; before c090d6dc14e05c561cf5c39cf5f6cc02e8cd04c5. #'> :key (lambda (sym) (let ((match-count 0)) (uiop:frob-substrings (string sym) (list (string string)) (lambda (sub frob) (incf match-count) (funcall frob sub))) (/ match-count (length (string sym)))))))) #+end_src

** Changing printer settings for Graven Image output

Graven Image =inspect*= function uses =interface-lines= for the number of properties to list. If your screen is more than 20 lines high, you might want to add more lines:

#+begin_src lisp (defmethod gimage:inspect* :around (object) (declare (ignore object)) (let ((gimage:interface-lines 45)) (call-next-method))) #+end_src

Most of Graven Image functions also rely on implementation/REPL-specific printer variables, which might be un-intuitive, overly verbose, or too short. Binding printer variables around Graven Image functions helps that too:

#+begin_src lisp (defmethod gimage:apropos* :around (string &optional package external-only docs-too) (declare (ignore string package external-only docs-too)) ;; Note that you can also use ;; sb-ext:*compiler-print-variable-alist*' and ;; sb-ext:debug-print-variable-alist' on SBCL. (let ((print-case :downcase) (print-level 2) (print-lines 2) (print-length 10)) (call-next-method))) #+end_src

A noisy apropos function listing like #+begin_src lisp X86::X86-OPERAND-TYPE-NAMES [VARIABLE = ((:REG8 . 1) (:REG16 . 2) (:REG32 . 4) (:REG64 . 8) (:IMM8 . 16) (:IMM8S . 32) (:IMM16 . 64) (:IMM32 . 128) (:IMM32S . 256) (:IMM64 . 512) (:IMM1 . 1024) (:BASEINDEX . 2048) (:DISP8 . 4096) (:DISP16 . 8192) (:DISP32 . 16384) (:DISP32S . 32768) (:DISP64 . 65536) (:INOUTPORTREG . 131072) (:SHIFTCOUNT . 262144) (:CONTROL . 524288) (:DEBUG . 1048576) (:TEST . 2097152) (:FLOATREG . 4194304) (:FLOATACC . 8388608) (:SREG2 . 16777216) (:SREG3 . 33554432) (:ACC . 67108864) (:JUMPABSOLUTE . 134217728) (:REGMMX . 268435456) (:REGXMM . 536870912) (:ESSEG . 1073741824) (:INVMEM . 2147483648) (:REG . 15) (:WORDREG . 14) (:IMPLICITREGISTER . 75890688) (:IMM . 1008) (:ENCIMM . 464) (:DISP . 126976) (:ANYMEM . 2147547136) (:LLONGMEM . 2147547136) (:LONGMEM . 2147547136) (:SHORTMEM . 2147547136) (:WORDMEM . 2147547136) (:BYTEMEM . 2147547136) (:LABEL . 4294967296) (:SELF . 8589934592))] #+end_src turns into a much more readable #+begin_src lisp x86::x86-operand-type-names [variable = ((:reg8 . 1) (:reg16 . 2) (:reg32 . 4) (:reg64 . 8) (:imm8 . 16) (:imm8s . 32) (:imm16 . 64) (:imm32 . 128) (:imm32s . 256) (:imm64 . 512) ...)] #+end_src

** Suppressing documentation errors in =documentation*=

Several implementations throw errors when trying to get documentation for non-existent method combinations, classes, etc. It's convenient to suppress these: #+begin_src lisp (defmethod gimage:documentation* :around (object &optional doc-type) (ignore-errors (call-next-method))) #+end_src

Actually, one can try to write an =:around= method for regular =documentation=, but this modification is not guaranteed to work on all implementations.

• Contributing

You can help with any of the [[https://github.com/aartaka/graven-image/issues?q=is%3Aopen+is%3Aissue][open issues]] most are well-described and split into bite-sized tasks. See .github/CONTIBUTING.md for the contributing guidelines.