Improve the stringification of `collections` objects

[dwkns]

Operating system

Mac OS 15.3.2 (24D81)

Eleventy

3.0.0 & 3.0.1-alpha.5

Describe the bug

Ultimate goal improve eleventy-plugin-console-plus to support expand/collapse style view

I have a shortcode I'm using in a Nunjucks template:

 {% jsonToHtml collections.all[0] %}

collections.all has circular references.

In the shortcode I attempt to remove the circular references:

  eleventyConfig.addShortcode("jsonToHtml",  (obj) => {

  // various methods to clone `obj` to avoid editing references. 

    const cache = new Set();
     const jsonString = JSON.stringify(obj, (key, value) => {
       if (typeof value === 'object' && value !== null) {
         // If the object has already been seen, return undefined
         // to avoid circular reference
         if (cache.has(value)) {
           return; // Remove circular reference
         }
         // Store the object in the cache
         cache.add(value);
       }
       return value;
     });

     cache.clear(); // Clear the cache after serialization
  })
}

But Eleventy errors out with:

[11ty] 1. Having trouble rendering njk template ./src/index.njk (via TemplateContentRenderError)
[11ty] 2. (./src/index.njk)
[11ty]   EleventyNunjucksError: Error with Nunjucks shortcode `jsonToHtml` (via Template render error)
[11ty] 3. Tried to use templateContent too early on ./src/index.njk (via TemplateContentPrematureUseError)
[11ty] 
[11ty] Original error stack trace: TemplateContentPrematureUseError: Tried to use templateContent too early on ./src/index.njk
[11ty]     at Object.get [as templateContent] (file:///Users/dazza/Desktop/demo-project/node_modules/@11ty/eleventy/src/Template.js:626:14)
[11ty]     at JSON.stringify (<anonymous>)
[11ty]     at Object.<anonymous> (file:///Users/dazza/Desktop/demo-project/eleventy.config.js:15:30)
[11ty]     at Object.fn (file:///Users/dazza/Desktop/demo-project/node_modules/@11ty/eleventy/src/Benchmark/BenchmarkGroup.js:37:23)
[11ty]     at ShortcodeFunction.run (file:///Users/dazza/Desktop/demo-project/node_modules/@11ty/eleventy/src/Engines/Nunjucks.js:232:29)
[11ty]     at Template.root [as rootRenderFunc] (eval at _compile (/Users/dazza/Desktop/demo-project/node_modules/nunjucks/src/environment.js:527:18), <anonymous>:14:70)
[11ty]     at Template.render (/Users/dazza/Desktop/demo-project/node_modules/nunjucks/src/environment.js:454:10)
[11ty]     at file:///Users/dazza/Desktop/demo-project/node_modules/@11ty/eleventy/src/Engines/Nunjucks.js:437:9
[11ty]     at Template._render (file:///Users/dazza/Desktop/demo-project/node_modules/@11ty/eleventy/src/TemplateContent.js:587:25)
[11ty]     at async file:///Users/dazza/Desktop/demo-project/node_modules/@11ty/eleventy/src/TemplateMap.js:360:8

I suspect this is becuase I'm modifing the orignal obj when running JSON.stringify.

However all attempts to clone the incomming object beforehand fail.

• Cloning obj via structuredClone(obj) — Original error stack trace: DataCloneError: read() {} could not be cloned.
• Cloning obj via `_.cloneDeep(obj) — TemplateContentPrematureUseError
• Flatted —TemplateContentPrematureUseError
• circular-structure-stringify — TemplateContentPrematureUseError
• Use util.inspect - no error but produces string output not JSON.
• safe-stringify — TemplateContentPrematureUseError

Reproduction steps

As above

Expected behavior

Be able to remove circular references in an object so I can serialize it to JSON within a shortcode.

Reproduction URL

No response

Screenshots

No response

[dwkns]

More on this.

It seems even reading the values of content and templateContent from collections.all[0] causes the TemplateContentPrematureUseError.

I understand that content and templateContent are potentially not complete and may change , but erroring out just on reading them seems harsh.

eleventyConfig.addShortcode("getKeysShortcode", async function (obj) {
    let allKeys = Object.keys(obj) // works as expected
    console.log("allKeys: ", allKeys)

    for (let key in obj) {
      if (key != "content" && key != "templateContent") {
        console.log("key ", key  ) // works as expected
        console.log("value: ", obj[key] ) // works as expected
      } else {
        console.log("key ", key  ) // works as expected
        console.log("value: ", obj[key] ) // fails with TemplateContentPrematureUseError error
      }
    }
    return 
  })
}

Called from a Nunjucks Template with:

{% getKeysShortcode collections.all[0] %}

We get the same error with using a standard function(){}, an async function(){} and a ()=>{} and async ()=>{}

[zachleat]

You’re right that collections is not currently JSON serializable, sorry! Can you delete content/templateContent as a workaround?

[zachleat]

There are probably some more unholy options here: https://github.com/11ty/eleventy/blob/e1d431d1421f12cde2a614a9819edbd5ce00810e/src/Template.js#L642 (needsCheck might be an escape hatch)

[zachleat]

Well, wait I don’t think you need collections to be JSON serializeable to improve your plugin, no?

https://nodejs.org/docs/latest/api/util.html inspect() handles circular references just fine. format() and %j might work too https://nodejs.org/docs/latest/api/util.html#utilformatformat-args

[dwkns]

inspect() is what I am using now. It works but it's output is a string.

This is fine when the object you are logging is small, you can scan through it pretty easily. However when you have a more complex object like collections.all printing it out as a string becomes very hard to read (sometimes hundreds of lines).

My plan is to allow collapsable output to make reading/navigating complex objects easier. See: https://pgrabovets.github.io/json-view/ for example.

This means whatever you supply to the plugin needs to be JSON serializeable.

The needsCheck hack will probably work but it means writing the code to deal with all sorts of nested arrays/objects etc. I can't use it with any of the existing packages for reccursing over objects/arrays.

[zachleat]

I swapped the content and templateContent properties to be enumerable: false and this greatly helped with JSON stringify (and it didn’t break any tests in the test suite).

Here’s the JSON stringify filter I used that worked with the above internals tweak.

export default function(eleventyConfig) {
	eleventyConfig.addFilter("stringify", (obj) => {
		return JSON.stringify(obj, function replacer(key, value) {
			// skip Template.js instances and circular references in collections
			if(key === "template" || key === "collections") {
				return {};

			}

			return value;
		}, 2);
	})
};

I think I’m open to changing these properties to be unenumerable (and maybe template too?). That would be a bigger breaking change though.

[zachleat]

Going to milestone this to v4

[dwkns]

Don't do extra work on this unless it's needed elsewhere.

I've created my own stringify function. Remembering the goal is to create a better debugging tool (not just produce clean JSON), a couple of big reasons are:

• Dealing with circular references. It's actually useful to show the actual data in a couple of places rather than just a reference to where it first appears. For instance: collections.all[0].data.page and at collections.all[0].data.page.
• Dealing with undefined. JSON.stringify({ a: "a", b: undefined }) produces { "a": "a" } and removes b. When debugging don't want this, you want: `JSON.stringify({ "a": "a"," b": "undefined" }), so you know it is undefined rather than missing.
• Excluding keys. Because ultimately we're rendering out to HTML nodes, being able to exclude a key from the output is valuble (or as I do subsitude a keys' value with a message). collections.all[0].template contains a whole load of stuff you almost certainly don't need.

Once you factor these in, along the needsCheck hack, I quickly realised that rolling my own custom stringify was a better approach. The complicated bit has actaully been turning that JSON in to useful HTML. The libraries I looked at we're OK, but of course not quite what I wanted (espcially in how it's going to be used) so I've been writing a new one.

You can see what it currently outputs here: https://stringify-and-view.netlify.app

And the repo that produces it: https://github.com/dwkns/stringify-and-view

Got a bit more tidying up to do and then I'll roll it in to a new version of the plugin.

Comments welcome.