json decoding does not seem to properly respect character encoding

[oboukili]

Hello, I'm using sops 3.2.0 with gcp kms. I noticed that decrypting using json input type does not respect (decode) the keys and values' text encoding.

It is easy to reproduce for the '&' character (https://www.fileformat.info/info/unicode/char/0026/index.htm):

'testing1' file contents

{
  "key&?abcd":"value?p1=true&p2=false"
}

Encrypt as a json input type file and a binary input file

sops --encrypt --input-type=json --output-type=json --gcp-kms projects/someproject/locations/global/keyRings/sops/cryptoKeys/sops-key testing1 > testing1.json.enc  

sops --encrypt --input-type=raw --output-type=raw --gcp-kms projects/someproject/locations/global/keyRings/sops/cryptoKeys/sops-key testing1 > testing1.raw.enc

Decrypt the raw encrypted version (ok)

sops -d --input-type=raw --output-type=raw testing1.raw.enc

{
  "key&?abcd":"value?p1=true&p2=false"
}

Decrypt the json encrypted version (not ok)

sops -d --input-type=json --output-type=json testing1.json.enc

{
	"key\u0026?abcd": "value?p1=true\u0026p2=false"
}

[autrilla]

Semantically, they are the same:

autrilla@iron:/Users/autrilla/Projects/go/src/go.mozilla.org/sops:master# cat test.json
{
  "key&?abcd":"value?p1=true&p2=false"
}
autrilla@iron:/Users/autrilla/Projects/go/src/go.mozilla.org/sops:master# sops -e test.json > test.enc.json
autrilla@iron:/Users/autrilla/Projects/go/src/go.mozilla.org/sops:master# sops -d test.enc.json | jq
{
  "key&?abcd": "value?p1=true&p2=false"
}

We could turn it off through https://godoc.org/encoding/json#Encoder.SetEscapeHTML, but I'm not really sure we should. There could be people relying on this.

[oboukili]

@autrilla thanks for your answer, how about an optional flag that we could enable through cli parameters or environment variable?

[autrilla]

I'd be okay with that. Although I think making this breaking change would be fine, since we really don't make any guarantees about the style of the outputs in any of our formats.

[xakraz]

Hi guys !

What is the status of this issue ? Thx !

[ifraixedes]

I'd like to know if this feature would require a lot of work. I think it shouldn't be I'm thinking blind because I haven't checked the source code.

If it wouldn't require a lot of work, I may think to work on it, however, I wouldn't commit to any time to finish it.

[Moskovych]

Is there any progress? Somebody is working on it? Might be it is included in some milestone for next releases?

[ifraixedes]

@Moskovych I was waiting for a reply from the owning team to spot some time to work on it because I wanted to make sure that if I spend time on it it isn't going to overlap the work with someone else and the changes will be fine to be merged.

[Moskovych]

@ifraixedes , ok, but I've already started working on it: https://github.com/mozilla/sops/pull/887. As discussed in https://github.com/mozilla/sops/issues/881 it will be "breaking" (without semantic) changes and without the flag, as is. Just waiting for the tests fix, and I'll proceed with adding tests, to cover that case.

[ifraixedes]

@Moskovych oh good, sorry for not responding earlier, normally I do in less than 2 days, but I was on holiday for 3 weeks and mostly away from my workstation.

[vborcea]

Hello, is there any progress on this?

[kvishweshwar]

Any updates?

can we use single quote/ literals (around password string) at the time of encryption and remove that single quote at the time of decryption? does this approach cause an issue with other functionalities?

[felixfontein]

@kvishweshwar are you talking about modifying the content? Or the encoding as JSON (or other formats)?

[bscaleb]

Any updates here?

[RoeyPrat]

Plus one on this issue, would love a flag 🫶