Add support for omiting filename in Vault path

[Kamahl19]

Adds a new setting into Vault's destination rule called vault_path_omit_filename which is false by default so there is no breaking change. And if it's set to true it will omit filename in Vault path.

Closes #659

[codecov-io]

Codecov Report

Merging #661 into develop will not change coverage. The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           develop     #661   +/-   ##
========================================
  Coverage    39.16%   39.16%           
========================================
  Files           22       22           
  Lines         3028     3028           
========================================
  Hits          1186     1186           
  Misses        1736     1736           
  Partials       106      106           

Impacted Files	Coverage Δ
config/config.go	71.64% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8b14d4e...35f16b0. Read the comment docs.

[Kamahl19]

@autrilla Hey, could you please also take a look on this one? :)

[autrilla]

First of all, the code LGTM. But I'm reluctant to accept this PR just yet, hopefully you think this is reasonable:

I think most people who've interacted with SOPS for a while would agree that we've added too many features that probably very few people use. For something like this, I want to ask you if you can use a fork for your use case, and when other people have similar use cases, they can explain theirs as well and we can figure out a solution that covers everyone.

I hope you understand I don't mean to say your use case is not good or that we don't care about it. It's just that in the past, we've had to badly hack together new features because the existing ones didn't cover all the bases, and we didn't want to break backwards compatibility for our users. We could come up with some other solution such as "experimental" features that we're allowed to remove or change completely, but that would be a long discussion, I feel.

Does this seem reasonable? If keeping your own fork is a painful experience, I'd like to also understand why.

[Kamahl19]

I understand the stated reasons in general for sure. Although I believe they dont quite apply to this case (I am not saying it just because this is MY case :D not at all). I will explain why I think so.

Publishing to Vault was integrated to SOPS as the last destination target (after S3 and GCS). Both S3 and GCS are "file storage" services/destinations. Therefor it makes sense to publish SOPS secrets as files they are stored in eg s3.aws.com/customizable-pathname/secrets.yml.

However Vault is different kind of destination. It's not file storage but secrets storage. Therefor it does not make sense to publish SOPS secrets as files rather publish secrets as secrets (Am I making sense? :) ).

I suspect the filename is appended to Vault path just because it was done the same way for S3 and GCS. It was just copy/pasted logic with different destination. So when you want to publish secrets to vault your path will be secret/customizable-pathname/secrets.yaml. That means I cant really configure secret path, only part of it.

IMHO the best way would be to remove appending filename to Vault path altogether as it does not make much sense for Vault destination. However that is breaking change and I understand you wont upgrade to v4 just because of this. Therefor I added a backward compatible setting to allow all users publishing to Vault to really configure the path.

[Kamahl19]

@autrilla did you have a time to think about it further? Or is your decision final? :)

[autrilla]

Hey, sorry, I've been busy these past few days. I understand that Vault is different than S3, and it's not that I'm against doing this -- I just want to make sure this is a problem other people care about before we add that maintenance cost to SOPS. I don't want someone showing up next week asking for the ability to fully control the location of the secret, and us having to have two features for essentially the same problem.

Are there reasons why it's inconvenient for you to work from your own branch? We could work on remedying those.

[Kamahl19]

I understand you don't want to increase the maintenance cost and a new option might do that.

Best solution IMHO would be not to add the flag but to remove filename from vault path altogether. This would however be a breaking change and require a major version bump, thats why I introduces a flag.

Btw in the meantime, my use-case changed a bit and I wont be using sops CLI for publishing to vault. Therefor this PR is not important for me personally anymore, just wanted to fix this for everyone. So if you feel like closing this, go ahead :) Just keep in mind that publishing to Vault is not really usable in the current form.

I have 2 questions though and it would really help me if you could explain them to me even though they are a bit OT.

My use-case changed and I need to publish sops to vault programatically, in my own Go library. I suppose I cant use the sops publish command in my Go library programatically, can I?

I guess I need to decrypt sops file myself and then publish it to Vault. For that I wanted to use the "decrypt" library you provide. However I noticed the decrypt library is using very different way to decrypt secrets than publish command does. Decrypt lib: https://github.com/mozilla/sops/blob/master/decrypt/decrypt.go#L49 Publish command: https://github.com/mozilla/sops/blob/master/cmd/sops/subcommand/publish/publish.go#L131

So can I use the decrypt library to get cleartext values and then publish them to vault myself, or do I need to "re-implement" the publish command of sops? Thanks :)

[autrilla]

I understand you don't want to increase the maintenance cost and a new option might do that.

Best solution IMHO would be not to add the flag but to remove filename from vault path altogether. This would however be a breaking change and require a major version bump, thats why I introduces a flag.

Btw in the meantime, my use-case changed a bit and I wont be using sops CLI for publishing to vault. Therefor this PR is not important for me personally anymore, just wanted to fix this for everyone. So if you feel like closing this, go ahead :) Just keep in mind that publishing to Vault is not really usable in the current form.

I see. Breaking this is not an option right now, and I imagine someone is using Vault publishing right now, so it must be usable. Not sure why they would've gone through the trouble of implementing it otherwise.

As for helping other people, I think that's great. I think they should still be able to find this with some searching and use it.

I have 2 questions though and it would really help me if you could explain them to me even though they are a bit OT.

Happy to!

My use-case changed and I need to publish sops to vault programatically, in my own Go library. I suppose I cant use the sops publish command in my Go library programatically, can I?

I don't see why not. The API is going to be a bit clunky, since it was made just for the CLI though. Also, keep in mind we make no guarantees of API stability outside of the decrypt package, so the API there might break when you upgrade your vendored sops.

I guess I need to decrypt sops file myself and then publish it to Vault. For that I wanted to use the "decrypt" library you provide. However I noticed the decrypt library is using very different way to decrypt secrets than publish command does. Decrypt lib: https://github.com/mozilla/sops/blob/master/decrypt/decrypt.go#L49 Publish command: https://github.com/mozilla/sops/blob/master/cmd/sops/subcommand/publish/publish.go#L131

So can I use the decrypt library to get cleartext values and then publish them to vault myself, or do I need to "re-implement" the publish command of sops? Thanks :)

Either way works. There's no difference between the way publish and decrypt decrypt the file (other than the code being a bit different for historical reasons). Publishing through the SOPS code will however be more limited, since SOPS has opinions about e.g. where the files go into Vault.

Hope this helps!

[Kamahl19]

Thanks! In that case I will just use the decrypt package and publish to Vault myself :)

[Kamahl19]

@autrilla I have one more unrelated question :) Is it somehow possible to NOT include the encryption key, in my case AWS KMS key, in the output file?

[blarghmatey]

As someone who is new to SOPS and looking to publish secrets to Vault, what would be great is to have a couple of modes of operation. On one hand, being able to publish secrets to Vault using a customized path (without the file extension), and on another being able to specify that a given set of keys (e.g. in a YAML or JSON file) each get set to different paths (possibly using multi-document YAML files?)

Also, being able to populate the contents of encrypted files in Vault (e.g. RSA keys/certificates) would be immensely helpful.