jzon

A correct and safe(er) JSON RFC 8259 parser with batteries-included.

Table of Contents

• Overview
• Usage
  • Type Mappings
  • Reading
  • Writing
    • Symbol key case
    • Custom Serialization
      • coerced-fields
  • Features
    • Unambiguous values
    • Strict spec compliance
    • Safety
    • Simplicity
    • Object key pooling
• Dependencies
• License

Usage

Type Mappings

jzon maps types per the following chart:

Note the usage of symbol cl:null as a sentinel for JSON null

Reading

There's a single entry point: parse:

(jzon:parse "
{
  \"name\": \"Rock\",
  \"coords\": {
    \"map\": \"stony_hill\",
    \"x\": 5,
    \"y\": 10
  },
  \"attributes\": [\"fast\", \"hot\"],
  \"physics\": true,
  \"item\": false,
  \"parent\": null
}")

parse reads input from its single argument and returns a parsed value per Type Mappings.

in can be any of the following:

• string
• (vector (unsigned-byte 8)) - octets in utf-8
• stream - character or binary in utf-8
• pathname - parse will open the file for reading

parse also accepts the follwing keyword arguments:

• :allow-comments This allows the given JSON to contain //cpp-style comments
• :max-depth This controls the maximum depth to allow arrays/objects to nest. Can be a positive integer, or nil to disable depth tests.
• :max-string-length This controls the maximum length of strings. This applies for both keys and values. Must be a positive integer no larger than array-dimension-limit.
• :key-fn A function of one argument responsible for 'interning' object keys. Should accept a simple-string and return the 'interned' key

Tip: key-fn can be supplied as #'identity in order to disable key pooling:

(parse "[ { \"x\": 1, \"y\": 1 }, { \"x\": 1, \"y\": 1 } ]" :key-fn #'identity)

Tip: alexandria:make-keyword or equivalent can be used to make object keys into symbols:

(jzon:parse "[ { \"x\": 1, \"y\": 1 }, { \"x\": 1, \"y\": 1 } ]" :key-fn #'alexandria:make-keyword)

Writing

stringify will serialize an object to JSON:

(stringify #("Hello, world!" 5 2.2 #(null)))
; => "[\"Hello, world!\",5,2.2,[null]]"

stringify accepts the following keyword arguments:

• :stream A destination like in format, or a pathname. Like format, returns a string if nil.
• :pretty If true, output pretty-formatted JSON
• :coerce-element A function for coercing 'non-native' values to JSON. See Custom Serialization
• :coerce-key A function for coercing key values to strings. See Custom Serialization

In addition to the mappings defined in Type Mappings, stringify accepts the following types of values:

*: Heuristic depending on the key values - Detects alists/plists by testing each key to be a character, string, or symbol.

†: On supported implementations where structure slots are available via the MOP.

These coercion rules only apply when using the default :coerce-element and :coerce-key.

Symbol key case

When symbols are used as keys in objects, their names will be downcased, unless they contain mixed-case characters.

For example:

(let ((ht (make-hash-table)))
  (setf (gethash 'all-upper ht) 0)
  (setf (gethash '|mixedCase| ht) 0)

  (stringify ht))

shall result in:

{
  "all-upper": 0,
  "mixedCase": 0
}

This is particularly important when serializing CLOS objects per Custom Serialization.

Custom Serialization

stringify allows serializing any values not covered in the Type Mappings in an few different ways.

By default, if your object is a standard-object, it will be serialized as a JSON object, using each of its bound slots as keys.

Consider the following classes:

(defclass coordinate ()
  ((reference
    :initarg :reference)
   (x
    :initform 0
    :initarg :x
    :accessor x)
   (y
    :initform 0
    :initarg :y
    :accessor y)))

(defclass object ()
  ((alive
    :initform nil
    :initarg :alive
    :type boolean)
   (coordinate
    :initform nil
    :initarg :coordinate
    :type (or null coordinate))
   (children
    :initform nil
    :initarg :children
    :type list)))

If we stringify a fresh coordinate object via (stringify (make-instance 'coordinate)), we'd end up with:

{
  "x": 0,
  "y": 0
}

And if we (stringify (make-instance 'coordinate :reference "Earth")):

{
  "reference": "Earth",
  "x": 0,
  "y": 0
}

Similarly if we (stringify (make-instance 'object)):

{
  "alive": false,
  "coordinate": null,
  "children": []
}

Note that here we have nil representing false, null, and []. This is done by examining the :type of each slot. If no type is provided, nil shall serialize as null.

stringify recurses, so if we have:

(stringify (make-instance 'object :coordinate (make-instance 'coordinate)))

We'll have:

{
  "alive": false,
  "coordinate": {
    "x": 0,
    "y": 0
  },
  "children": []
}

coerced-fields

If you wish more control over how your object is serialized, the most straightforward way is to specialize coerced-fields.

Consider our previous coordinate class. If we always wanted to serialize only the x and y slots, and wanted to rename them, we could specialize coerced-fields as follows:

(defmethod coerced-fields ((coordinate coordinate))
  (list (list "coord-x" (x coordinate))
        (list "coord-y" (y coordinate))))

This results in:

{
  "coord-x": 0,
  "coord-y": 0
}

coerced-fields should a list of 'fields', which are two (or three) element lists of the form:

(name value &optional type)

The name can be any suitable key name. In particular, integers are allowed coerced to their decimal string representation.

The value can be any value - it'll be coerced if necessary.

The type is used as :type above, in order to resolve ambiguities with nil.

Including only some slots

If the default coerced-fields gives you most of what you want, you can exclude/rename/add fields by specializing an :around method as follows:

(defmethod coerced-fields :around ((coordinate coordinate))
  (let* (;; Grab default fields
         (fields (call-next-method))
         ;; All fields except "children"
         (fields (remove 'children fields :key #'first))
         ;; Include a 'fake' field "name"
         (fields (cons (list 'name "Mary") fields)))
    fields))

This would result in the following:

{
  "name": "Mary",
  "alive": false,
  "coordinate": {
    "x": 0,
    "y": 0
  }
}

write-value

For more fine-grained control, you can specialize a method on jzon:write-value.

jzon:write-value writer value

writer is a json-writer on which any of the writer functions may be called to serialize your object in any desired way.

(defclass my-point () ())

(defmethod jzon:write-value (writer (value my-point))
  (jzon:write-array writer 1 2))

See json-writer for the available functions.

json-writer

In addition to stringify, jzon also provides an imperative, streaming writer for writing JSON.

The following are the available functions for writing:

General

• jzon:write-value - Writes any value to the writer. Usable when writing a toplevel value, object property value, or array element.

Object

• jzon:with-object
• jzon:begin-object
• jzon:write-key
• json:write-property
• json:write-properties
• json:end-object
• jzon:write-object

Array

• jzon:with-array
• jzon:begin-array
• jzon:write-values
• json:end-array
• jzon:write-array

Example

(let ((writer (jzon:make-json-writer :stream *standard-output* :pretty t)))
  (jzon:with-object writer
    (jzon:write-properties writer :age 24 "colour" "blue")
    (jzon:write-key writer 42)
    (jzon:write-value writer #(1 2 3))

    (jzon:write-key writer "an-array")
    (jzon:with-array writer
      (jzon:write-values writer :these :are :array :elements))

    (jzon:write-key writer "another array")
    (jzon:write-array writer :or "you" "can" "use these" "helpers")))

produces:

{
  "age": 24,
  "colour": "blue",
  "42": [
    1,
    2,
    3
  ],
  "an-array": [
    "THESE",
    "ARE",
    "ARRAY",
    "ELEMENTS"
  ],
  "another array": [
    "OR",
    "you",
    "can",
    "use these",
    "helpers"
  ]
}

It's worth noting that every function returns the writer itself for usage with arrow macros:

(let ((writer (jzon:make-json-writer :stream *standard-output*)))
  (jzon:with-object writer
    (-> writer
        (jzon:write-key "key")
        (jzon:write-value "value")
        (jzon:begin-array)
        (jzon:write-value 1)
        (jzon:end-array))))`

Features

This section notes some of jzon's more noteworthy features.

In general, jzon strives for (in order):

• Safety
• Correctness
• Simplicity
• Interoperability
• Performance

Unambiguous values

Values are never ambiguous between [], false, {}, null, or a missing key, as in some other json parsers.

Strict spec compliance

This parser is written against RFC 8259 and strives to adhere strictly for maximum compliance and little surprises.

Also, this has been tested against the JSONTestSuite. See the JSONTestSuite directory in this repo for making & running the tests.

Safety

RFC 8259 allows setting limits on things such as:

• Number values accepted
• Nesting level of arrays/objects
• Length of strings

jzon is meant to be safe in the face of untrusted JSON and will error on otherwise 'reasonable' input out-of-the-box.

jzon's parse is also type-safe, and shall not, for example:

CL-USER> (parse 2)
; Debugger entered on #<SB-SYS:MEMORY-FAULT-ERROR {1003964833}>

.. as in some other libraries.

jzon also chooses to (by default) keep object keys as strings. This is done rather than using symbols via intern because over time, symbols will continue to be allocated and because they are in a package, will not be collected by the garbage collector, causing a memory leak.

Simplicity

You call parse, and you get a reasonable standard CL object back.

• No custom data structures or accessors required
• No worrying about key case auto conversion or hyphens/underscores being converted.
• No worrying about what package symbols are interned in (no symbols).
• No worrying about dynamic variables affecting a parse as in cl-json, jonathan, jsown. Everything affecting parse is given at the call-site.

parse also accepts either a string, octet vector, stream, or pathname for simpler usage over libraries requiring one or the other, or having separate parse functions.

Object key pooling

jzon will use a key pool per-parse, causing shared keys in a nested JSON object to share keys:

(parse "[{\"x\": 5}, {\"x\": 10}, {\"x\": 15}]")

In this example, the string x is shared (eq) between all 3 objects.

This optimizes for the common case of reading a JSON payload containing many duplicate keys.

Dependencies

• closer-mop
• flexi-streams
• uiop

Alternatives

There are many CL JSON libraries available, and I defer to Sabra Crolleton's definitive list and comparisons https://sabracrolleton.github.io/json-review.

But for posterity, included in this repository is a set of tests and results for the following libraries:

• cl-json
• jonathan
• json-streams
• jsown
• yason

No ill-will is meant for these other libraries. I simply want jzon to be better and become a true de-facto library in the world of JSON-in-cl once and for all.

License

See LICENSE.

jzon was originally a fork of st-json, but I ended up scrapping all of the code except for for the function decoding Unicode.