UI corrupts read-only JSON fields

[FliesLikeABrick]

Deployment Type

Self-hosted

NetBox Version

v4.0.5

Python Version

3.11

Steps to Reproduce

This issue can be reproduced in a couple different workflows. This was originally reported in #16328 after we saw this behavior, which we believe was introduced between 3.6.3 and 3.7.7. We have now reproduced this in a clean install of 4.0.5 running inside docker using netbox-docker 2.9.1.

Edits through the API are not impacted.

Option 1 - corruption during edit:

1. Create a JSON custom field - set "UI Editable" to "No". Leave the default value as null.
2. Create a prefix, it is not necessary to populate the JSON field with a value -- it will default to null
3. View the prefix, observe that the JSON custom field shows a value of "-" in the UI due to it being null
4. Edit the prefix and click "save" without making any changes
5. Observe that the custom value now changes to the quoted string "null"

Option 2 - corruption on creation:

1. Create a JSON custom field - set "UI Editable" to "No". Set the default to JSON such as {"key1":"value1"}
2. Create a prefix
3. View the prefix, observe that the JSON custom field now shows an escaped string value of "{\"key1\": \"value1\"}"
4. Edit the prefix and click "save" without making any changes
5. Observe that the custom value now changes to the quoted string "null"

Expected Behavior

The JSON value should not be changed due to the UI being set to read-only for this field

Observed Behavior

The server is setting a new and incorrect/corrupt value for this field in the database. The native JSON is being converted to an escaped string

[FliesLikeABrick]

We have stepped through releases and found that this was introduced in 3.7.7, the issue was not present in 3.7.6. Both of the "steps to reproduce" workflows described above are triggering the bug in 3.7.7.

[FliesLikeABrick]

"editing" and saving a prefix multiple times in the UI (not actually editing anything, let alone the read-only JSON field) results in the field value being escaped multiple times:

[FliesLikeABrick]

We have written a script that finds and repairs the affected data, which can be found at https://github.com/FliesLikeABrick/netbox_fix_json

[jeffgdotorg]

Thanks for reporting this problem and for providing clear and comprehensive steps to reproduce. I was able to reproduce the option-1 case exactly as described in a fresh 4.0.5 install. The option-2 case reproduces slightly differently from how it's written up in the issue body (the value { "thing1": "value1" } gets escaped and displays as "\"{\\\"thing1\\\": \\\"value1\\\"}\""), but I see your follow-on comment lines up with that. I suggest editing your issue body to match what you (and I) actually see in option 2, but in any case I'm moving your issue out of triage.

Extra thanks for doing the additional archaeology to determine where the problem seems to have been introduced. That's potentially a big help.

[FliesLikeABrick]

Huh, my option2 text was actually correct but github or markdown was eating the backslashes . I have edited it to be closer to what is being observed but my escaping may not 100% match the values seen in a reproduction (especially depending on whether someone is observing the raw database value vs that returned in the UI vs that returned by the API before or after being decoded as JSON)

[FliesLikeABrick]

Hi all, I am the reported of the original issue. This PR appears to fix the issue for null values only. Editing existing values, or creating a prefix with a default value both lead to escaped strings being placed in the field

[FliesLikeABrick]

I have updated Option 2 in the description with better detail, and added code tags around the JSON/field values to prevent github/markdown from hiding the escaping which I think was creating some confusion.

[FliesLikeABrick]

The linked PR has been updated and now does prevent this issue from occurring. As for anyone experiencing this issue who needs to fix corrupted data - see https://github.com/FliesLikeABrick/netbox_fix_json