Support mounting yaml and json files as is?

[ljani]

My use case: I'd like to pass a sops encrypted .yaml configuration file to my app after decryption as is, instead of picking a single key from the file. So, sops-nix is a little too clever for my use case and it should focus only on decrypting the file, not parsing it.

In other words, I use this file ./secrets/my-config.yaml in my flake:

my-secret1: ENC[AES256_GCM,data:tkyQPQODC3g=,iv:yHliT2FJ74EtnLIeeQtGbOoqVZnF0q5HiXYMJxYx6HE=,tag:EW5LV4kG4lcENaN2HIFiow==,type:str]
my-secret2: ENC[AES256_GCM,data:tkyQPQODC3g=,iv:yHliT2FJ74EtnLIeeQtGbOoqVZnF0q5HiXYMJxYx6HE=,tag:EW5LV4kG4lcENaN2HIFiow==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
...

It should be mounted as /run/secrets/my-config.yaml:

my-secret1: hello
my-secret2: hello

Then I could pass the /run/secrets/my-config.yaml to my app to read any secrets from.

I could do this using a format=binary file, but then I have to remember to use format=binary everywhere, including when invoking sops. Alternatively, I could use a different file extension, but then I'd lose syntax highlight. Yet another alternative would be to use a literal multi line yaml string, but then again I'd lose highlight.

I wonder if this is already supported, because the README.md mentions home-assistant-secrets.yaml, which is similar to what I'm looking for? But I think that was defined as something like this:

home-assistant-secrets.yaml: |
    my-secret1: hello
    my-secret2: hello

(Notice how the renderer fails to highlight the yaml keys my-secret1 and my-secret2).

Does this make any sense to you?

[Mic92]

Currently I don't see any other way but using either the multi-line variant or encrypting the yaml file as a binary, in which case you would still not be able to use sops as an editor of the file. Personally I use multline yaml values to encode yaml secrets. Notice that this a limitation in sops itself, where sops-nix has little control over.

[ljani]

Notice that this a limitation in sops itself, where sops-nix has little control over.

Does it? To me it seems it's sops-nix doing the unmarshalling.

A hack would be to write decryptSecret to something like this (search for SkipUnmarshal), but I have not tested this:

func decryptSecret(s *secret, sourceFiles map[string]plainData) error {
	sourceFile := sourceFiles[s.SopsFile]
	if sourceFile.data == nil || sourceFile.binary == nil {
		plain, err := decrypt.File(s.SopsFile, string(s.Format))
		if err != nil {
			return fmt.Errorf("Failed to decrypt '%s': %w", s.SopsFile, err)
		}
		if s.Format == Binary || s.SkipUnmarshal {
			sourceFile.binary = plain
		} else {
			if s.Format == Yaml {
				if err := yaml.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse yaml of '%s': %w", s.SopsFile, err)
				}
			} else {
				if err := json.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse json of '%s': %w", s.SopsFile, err)
				}
			}
		}
	}
	if s.Format == Binary || s.SkipUnmarshal {
		s.value = sourceFile.binary
	} else {
		val, ok := sourceFile.data[s.Key]

		if !ok {
			return fmt.Errorf("The key '%s' cannot be found in '%s'", s.Key, s.SopsFile)
		}
		strVal, ok := val.(string)
		if !ok {
			return fmt.Errorf("The value of key '%s' in '%s' is not a string", s.Key, s.SopsFile)
		}
		s.value = []byte(strVal)
	}
	sourceFiles[s.SopsFile] = sourceFile
	return nil
}

[ljani]

Yet another way would be to reserialize nested keys back to yaml/json and set that string into s.value, but I'm not sure if I like that

[Mic92]

I think I would prefer format=binary over adding another option which does the same thing. At the end of the day you would need an option to treat it differently.

[ljani]

Yeah, I understand and thus I'm trying to come up with a good idea. What about supporting setting s.Key to nil? In other words, if the secret key is not set, use the whole file:

func decryptSecret(s *secret, sourceFiles map[string]plainData) error {
	sourceFile := sourceFiles[s.SopsFile]
	if sourceFile.data == nil || sourceFile.binary == nil {
		plain, err := decrypt.File(s.SopsFile, string(s.Format))
		if err != nil {
			return fmt.Errorf("Failed to decrypt '%s': %w", s.SopsFile, err)
		}
		if s.Format == Binary || s.Key == nil {
			sourceFile.binary = plain
		} else {
			if s.Format == Yaml {
				if err := yaml.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse yaml of '%s': %w", s.SopsFile, err)
				}
			} else {
				if err := json.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse json of '%s': %w", s.SopsFile, err)
				}
			}
		}
	}
	if s.Format == Binary || s.Key == nil {
		s.value = sourceFile.binary
	} else {
		val, ok := sourceFile.data[s.Key]

		if !ok {
			return fmt.Errorf("The key '%s' cannot be found in '%s'", s.Key, s.SopsFile)
		}
		strVal, ok := val.(string)
		if !ok {
			return fmt.Errorf("The value of key '%s' in '%s' is not a string", s.Key, s.SopsFile)
		}
		s.value = []byte(strVal)
	}
	sourceFiles[s.SopsFile] = sourceFile
	return nil
}

EDIT: Err, the current way of defining secrets does not really support that. To do this, secrets should be indexed by their file paths, not their keys. EDIT2: I think I probably should investigate systemd's LoadCredential= and LoadCredentialEncrypted= a bit more as their functionality is more closer how I imagine secrets should be handled. As far as I understand, those could be plugged to use sops as well.

[shajra]

I think I would prefer format=binary over adding another option which does the same thing. At the end of the day you would need an option to treat it differently.

I ran into wanting the same request personally. I do agree that functionally, you get some of the same things with a binary encoding. But the biggest thing you don't get with a binary encoding is one of SOPS's selling points, which is a nice textual diffs in source control. So I don't think it's completely fair to say it's the same thing, though it's close.

So that said, is this feature something we think we can get upstreamed? That would inform the effort and approach I take personally in trying to get it to work.

[piyoki]

Hi, I came across with similar use cases. Any updates on this feature request?

[musjj]

Would love to have this too. It's really useful for passing a complex configuration file as-is to a program, while still retaining the benefits of SOPS (diff-able plain text).