yaml file sequence items spaces after hyphen

[lukeab]

Ubuntu 18.04 and archlinux sops v3.3.1

Have weird yaml thing, encrypted a file with lines that looks like

  ENV:
  - name: "AWS_KEY"
    value: "BLAHBLAHBLAH"
  - name: "AWS_REGION"
    value: "eu-west-1"
... etc..

SO i do `sops --encrypt file.yaml --output > file-sops.yaml then do sops --decrypt file-sops.yaml > file-dec.yaml the decrypted file now has those lines looking like

    ENV:
    -   name: AWS_KEY
        value: BLAHBLAHBLAH
    -   name: AWS_REGION
        value: eu-west-1

So, it's changed the yaml indent to 4 spaces, that's fine, it's in the spec, But it's added 2 spaces to the 1 space that was between the key and the - for each sequence item key yamlint blows up on this. though it doesn't cause issues in the python script that i then pass this config yaml to in my usecase, the CI issues yamllint causes are the problem.

The resulting yamllint output on my actual file in my environment, shows up like:

~$> yamllint myfile-dec.yaml 
myfile-dec.yaml
  1:1       warning  missing document start "---"  (document-start)
  10:8      error    too many spaces after hyphen  (hyphens)
  10:5      error    wrong indentation: expected 8 but found 4  (indentation)
  12:8      error    too many spaces after hyphen  (hyphens)
  14:8      error    too many spaces after hyphen  (hyphens)
  16:8      error    too many spaces after hyphen  (hyphens)
...etc...

[lukeab]

reproduced the issue with a standalone file

example

~$> cat myfile.yaml

ENV:
- name: AWS_KEY
  value: BLAHBLAHBLAH
- name: AWS_REGION
  value: eu-west-1

~$> SOPS_PGP_FP=<<mykey>> sops --encrypt myfile.yaml --output > myfile.sops.yaml ~$> cat myfile.sops.yaml

ENV:
-   name: ENC[AES256_GCM,data:An20+bmHvg==,iv:ea0DqrSFpb9T6wlAClBqRbpg5UD5/lvW/doKVhFsbpc=,tag:bNMPWBuIWg71okuBz2atvw==,type:str]
    value: ENC[AES256_GCM,data:iJzPHPrKZ5N62P2A,iv:7XDXT9FWmIjSQ4W+cPk6jQFp+fiMC72Jt2fEkt3Zdys=,tag:Zk4Dp7T1i/IMCxXRNxGWjg==,type:str]
-   name: ENC[AES256_GCM,data:ivyP16RZQlJXdw==,iv:1wTRa1AyoTYpeOUCHobEr4s2WtVc2wzThUmEQjz3AB8=,tag:vjBVjXgI8U9xoC4K5iiuSA==,type:str]
    value: ENC[AES256_GCM,data:LMmCluV9PU+j,iv:Z7cKzjBqpzFsh6wmMdh8xpHIKYniUXeh5qB4wAAKlM4=,tag:8DIWkROViYsIDvALiNlSGg==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-09-03T11:05:59Z'
    mac: ENC[AES256_GCM,data:1BDKFlREqTIuoa+sfC+swZlJxPcOpXY7qEn8I+NY9zUsPUmH5OShRunbbh15ZCf2Gr0/bnXzfxZVwUZm7ZYeVu1ls3EQkkf2sPY+/7DVr+1VR0U5zRPJuLZdbFNJ295RDqNicQDmgBjUbsbXTEs+zdoNY5yHHAHhymjIc+jbC6g=,iv:x2whgFJktAxNSJjgsJFLnqx4wc6VhEcO/ZCpaLuo5W4=,tag:LagzO+RSj0ATcxfg0vQd/w==,type:str]
    pgp:
    -   created_at: '2019-09-03T11:05:59Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMA6u00URofj69ARAAhbAHTmXEEMvoUip76RX3EklIkJSiSe8eDV1NkSEx1b8v
            2IDF/6HI2DuOaiR53SbfIA1dJKaWIiF/QQNZTxGwAQXFnCdKEByAk1qomSfLHYhv
            siJn/solf94F/YMGmWW+rCxTip3HZP7ZSk62HUyXI/byfEEnmddHmkQiU8HnGX2N
            EcwYZXgsp35Yync0YNrYnavuTvrwtpcnsYDrYVC9IjtNRXGQeZhvITYH8X1eMiIW
            dfam72MoiigBBIIVCqNlvchWNwBrehfsr5IotKM2uOXLU4z3XBrSMNwMbhev0lS0
            3KRW2EMYNcGXkMgZmCBFrlbxotREyCEU3OR7eOS4D5uHcIUAI3NFC6XGSZCnVJIx
            PQrBqnerOZSC1V8+H4zbfmnViwQ8bGtUHGPtg53fPC1BZltx3RsKJQALEHz4a3bN
            3KEAo4iOx9u7tpZWeLZZD0YPxEvqLke1wND6exOUMW0Y0GBxgG5uqusk6tFLiejl
            HXq6envDqAU1rEpcp2QBKJQKodcU6RaWfKOQewfrstt5IgFWctNz69Ig/Fpu85p8
            IP8ZAY7W2ZMwjVkJp1FIxxYU7+sDwhhodUJ0dIF/Gggb0KJsF85aOiI26DcMTtd0
            VPDSKg9B5OZxDgPj4y3PLzWblkDTASylj7jlNe1Ke2hWQBuRxvxWwR6gjcl8WnvS
            XAHbWStg12rkDeRM6OJD54sJYhVQyerCMOwm7yCX2uEuFE9ddUfWMLyGPcdFAB7R
            iTZNDXu9erC0LG7zc/aCipnUXdbHDoNVa/MgRtMyXpP6xs9cZzT1E2qMOzTw
            =Gkhz
            -----END PGP MESSAGE-----
        fp: 83BFC84EFA98B9284D3939220954AFBD84AD4B85
    unencrypted_suffix: _unencrypted
    version: 3.3.1

~$> SOPS_PGP_FP=<<mykey>> sops --decryptt myfile.sops.yaml --output > myfile.dec.yaml ~$> cat myfile.dec.yaml

ENV:
-   name: AWS_KEY
    value: BLAHBLAHBLAH
-   name: AWS_REGION
    value: eu-west-1

~$> yamllint myfile.dec.yaml

myfile.dec.yaml
  1:1       warning  missing document start "---"  (document-start)
  2:4       error    too many spaces after hyphen  (hyphens)
  2:1       error    wrong indentation: expected 2 but found 0  (indentation)
  4:4       error    too many spaces after hyphen  (hyphens)

[lukeab]

just a note, origional bug was encountered while use aws kms for encryption, so doesn't seem to be related to encryption type anyway.

[autrilla]

It seems to me that the output is spec compliant. We don’t really make any guarantees on the styling of the YAML, and we especially do not guarantee the output style will be consistent with the input style. The yaml library we use has now been updated such that we could theoretically support it, but nobody has put in the time yet.

On Tue 3. Sep 2019 at 13:10, Luke Ashe-Browne [email protected] wrote:

just a note, origional bug was encountered while use aws kms for encryption, so doesn't seem to be related to encryption type anywya.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mozilla/sops/issues/514?email_source=notifications&email_token=AARH4V6WP3TLXCI25DSWL3DQHZA2LA5CNFSM4ITE3SG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5X3ADQ#issuecomment-527413262, or mute the thread https://github.com/notifications/unsubscribe-auth/AARH4V3APIYBCGOXNZ4O5RLQHZA2LANCNFSM4ITE3SGQ .

[lukeab]

hmm, seems there is a yamllint flag to control the number of spaces allowed after a hyphen https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.hyphens

I assumed since, by default, yamllint threw an error, it was actually a violation of the spec, not just a style concern, but now you have cast doubt on my assumption. Is there any rules in the spec about spaces after sequence item hyphens?

I thought i might try fix it, but you say there is a new library in use, i'd give it a shot, but i haven't been able to find where in the code this fix could be affected easily yet. A pointer in the right direction would be motivating for me to try again.

[lukeab]

Answering my own uncertainty: had a look at the spec https://yaml.org/spec/1.2/spec.html#id2759963 and https://yaml.org/spec/1.2/spec.html#id2797382

Seems it just says

The “-” indicator must be separated from the node by white space.

So there's no guarantee in the spec that it should be only one space character.

[kunickiaj]

Ran into this as well, formatting inconsistency from the rest of our files is a bit inconvenient. Would also be interesting in understanding where we could add support for control of the formatting.

The more important issue I encountered, is that since keys are reordered, block comments no longer appear next to the keys they're intended for.

valueB: 1234
# comment for valueA
valueA: abcdef

yields something like

# comment for valueA
valueB: 1234
valueA: abcdef

[autrilla]

@kunickiaj we would need to upgrade to https://github.com/go-yaml/yaml/tree/v3, parsing into https://godoc.org/gopkg.in/yaml.v3#Node.

FWIW, SOPS does not reorder keys. https://github.com/mozilla/sops/issues/300#issuecomment-367942757 has more information on why the comment behavior you see happens.

[rbabyuk]

hi all,

I am hitting this issue as well, sops version is 3.6.1

[onedr0p]

I ran into this issue too...

sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2021-02-03T20:45:32Z'
    mac: ENC[AES256_GCM,data:sqFu3XksD/IKE4b6WFW/hJ39ba//PD4YikgmENv2K4rNcR0nZCh5e/sEUPKSDIkUM+fyZeNlwwxodhh5oVMsY+0bQ5ISWbH+dxi+h3j1GrMgfms+EYImik49bjSYbcYBj9BeUWtnzz8Ab/PHUqMrbIMNVaHWAfbuClbooYmyKeQ=,iv:tbDF2y3+86qrKEDv8vJS2zIBB/PzJUe20nLX513+5s8=,tag:3GeY6ZqqSdFJqvzVMzcmhA==,type:str]
    pgp:
    -   created_at: '2021-01-27T18:14:44Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMAyUpShfNkFB/AQf/RxF3NLkoonJP4wQqELkfPtJl5Oar1UH7OBYwzep3uEPE
            MCPFeI/Fdr/8WxT0x6ieUC7odZX1J9tNzUeStyaWCS3CX6+zmEb0uavV7RlewdFc
            noIPyM7Q4R6/L6QyGWx6aPwkUq8qvH+8hk6VdqtAdipY8uBVogO/+BHd502nE+5E
            WqzxRFxFr7/6mjFHrPYcab5QvSoVc3Lrhh08hKXJYpquPrjDI4VYkJhTxmbaZK5w
            e676rq+eob8tQ7Nz2nnw5W51Oy/i2Yt6Q/wHj5dSPc4g+LMVUPwdx72+OB829uZR
            3FNKH2EJ6BCVVOe7JUueeoHjz1/UCooAx9vwAlJ8MNJeAbrEJPL1OUEJfkqJuoQk
            v4LGyJD4Gi+58FqLcOpx1DGqNsMZ8ElGKbBUKC3Resksd4K4uTfMyy0HwDRpvUdG
            M47W5K9K28MePouf7T/lsemwoYB59GSoeqIviyeteQ==
            =DSca
            -----END PGP MESSAGE-----
        fp: 3D16CEE4A27381B4
    unencrypted_suffix: _unencrypted
    version: 3.6.1

and yamllint output:

yamllint secret.enc.yaml
my-secrets.enc.yaml
  1:1       warning  missing document start "---"  (document-start)
  9:8      error    too many spaces after hyphen  (hyphens)
  9:5      error    wrong indentation: expected 8 but found 4  (indentation)

The only workaround I found is to have yamllint ignore filenames matching a certain regex, which works until you need to embed the sops data into an existing yaml file.

[felixfontein]

This should be fixed by the change to yaml.v3 in the current develop branch.

[onedr0p]

I can confirm formatting is still the same, even with the new release.

[felixfontein]

The latest version uses a different YAML library, and lists are definitely serialized differently. So formatting is definitely not the same. If yamllint likes the new style better is another question :-)

[onedr0p]

You are correct @felixfontein apologies, I did not test the right binary :(

Before...

    pgp:
    -   created_at: '2021-01-27T18:14:44Z'
        enc: |

After...

    pgp:
        - created_at: "2021-03-25T00:36:44Z"
          enc: |

[almereyda]

Would also be interesting in understanding where we could add support for control of the formatting.

It appears support for https://editorconfig.org/ would also allow us to make some linters happy.

[Ph0tonic]

Hi, I think that this issue can be closed as the formatting is now different and the indentation can now be configured.