KV Engine: Recursively list keys

[vfauth]

Is your feature request related to a problem? Please describe. With a KV engine, if I want to list all keys in the directory /foo/, it only returns keys directly under /foo/ For example, if I have the following keys:

/foo/some_key
/foo/bar/some_other_key

A LIST operation on /foo/ returns some_key and bar/, while I would like to have some_key and bar/some_other_key

Describe the solution you'd like Add a parameter to either recursively return ALL keys in the provided path.

Describe alternatives you've considered Another way to do it would be to add a parameter specifying the depth up to which look recursively for keys.

[extrail]

This is a very useful feature, especially for the HTTP API

[agaudreault]

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

[QWERTY92009]

Also interested in this kind of feature.

[kir4h]

For anyone ending up here, I created a small cli to perform recursive kv read/list operations while we wait for the native solution. Not very tested yet, I will be fixing bugs as they show up.

[antonu17]

@agaudreault-jive, thanks! Btw, local readonly path="$1" doesn't work as you might expect. use local -r path="$1" https://stackoverflow.com/a/45409823

[vishalnayak]

Issues that are not reproducible and/or have not had any interaction for a long time are stale issues. Sometimes even the valid issues remain stale lacking traction either by the maintainers or the community. In order to provide faster responses and better engagement with the community, we strive to keep the issue tracker clean and the issue count low. In this regard, our current policy is to close stale issues after 30 days. If a feature request is being closed, it means that it is not on the product roadmap. Closed issues will still be indexed and available for future viewers. If users feel that the issue is still relevant but is wrongly closed, we encourage reopening them.

Please refer to our contributing guidelines for details on issue lifecycle.

[dondrich]

Would love this feature

[rnsc]

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

var secretListPath []string

func isNil(v interface{}) bool {
	return v == nil || (reflect.ValueOf(v).Kind() == reflect.Ptr && reflect.ValueOf(v).IsNil())
}

// ListSecret returns a list of secrets from Vault
func ListSecret(vaultCli *vault.Client, path string) (*vault.Secret, error) {
	secret, err := vaultCli.Logical().List(path)
	if err != nil {
		log.Println("Couldn't list from the Vault.")
	}

	if isNil(secret) {
		log.Printf("Couldn't list %s from the Vault.", path)
	}
	return secret, err
}

// RecursiveListSecret returns a list of secrets paths from Vault
func RecursiveListSecret(vaultCli *vault.Client, path string) []string {
	secretList, err := ListSecret(vaultCli, path)
	if err == nil && secretList != nil {
		for _, secret := range secretList.Data["keys"].([]interface{}) {
			if strings.HasSuffix(secret.(string), "/") {
				RecursiveListSecret(vaultCli, path+secret.(string))
			} else {
				secretListPath = append([]string{strings.Replace(path, "metadata", "data", -1) + secret.(string)}, secretListPath...)
			}
		}
	}
	return secretListPath
}

[kir4h]

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

@rnsc Just in case you didn't notice my comment above and it can help, I wrote a small go based cli for this (https://github.com/hashicorp/vault/issues/5275#issuecomment-662694602)

[colinjcahill]

Adding my vote for this feature please.

[mechaHarry]

Ditto! Essentially looking for some kind of vault-grep to find out any secret/engine/etc. that has some key, value, or pathname that matches a desired string.

[VWRalf]

One more vote for search

[jsalatiel]

please implement this. It has been 4 years!

[tguvdamm]

This is a must have, listing all secrets now is a major issue.

[maxadamo]

if 103 people upvoted this feature request, maybe it's about time to start adding this feature :smile_cat: For instance, Consul lists all the keys recursively, and Redis does it by default (using: KEYS *). And whilst the solution from @kir4h works pretty well, it would be nice to expose this feature through the API, and libraries for any language can make use of this functionality.

[Sharkzeng]

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

[ndlanier]

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough! ./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

This script doesn't work if you have spaces in the path name. Even with passing the argument with double quotes.

[ndlanier]

I replicated that script in python

import os
import subprocess
import json

#
# Note: Before running, you must authenticate via the vault cli or the commands will fail.
#

def traverse (inputPath):
    path = inputPath

    result = subprocess.getoutput('vault kv list -format=json -address="'+vaultAddr+'" "'+inputPath+'"')
    jResult = json.loads(result)

    for secret in jResult:
        if "/" in secret:
            print(inputPath+secret)
            traverse(inputPath+secret)
        elif "/" not in jResult:
            print(inputPath+secret)

parentPath = "Secret Engine/" #change to secret engine name that you are want to scan
vaultAddr = "https://vault.whatever" #change to vault address

vaults = subprocess.getoutput('vault kv list -format=json -address="'+vaultAddr+'" "'+parentPath+'"')
vault = json.loads(vaults)

for secret in vault:
    print(parentPath+secret)
    if "/" in secret:
        traverse(parentPath+secret)

[FalcoSuessgott]

Had the same problem and ended up writing my own little tool which helps me listing KV secrets recursively in various useful formats:

https://github.com/FalcoSuessgott/vkv

[DE110283]

Adding my vote for this feature please.

[NicolasTobias]

me too, seems Hashiguys doesnt want to

[finnstech]

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

• What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?
• What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?
• What if I have LIST or READ access to a secret within a path but not the parents within that path? Thanks!

[vfauth]

Good news! For your question, I would say:

• 404, to give no information about the key existence
• 404, to give no information about the key existence
• 200, because the access to this specific key has been allowed

[ndlanier]

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

• What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?
• What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?
• What if I have LIST or READ access to a secret within a path but not the parents within that path? Thanks!

• I agree with vfauth's thinking, give does not exist or permission denied, best to not let someone assume a secret or path exists if they don't have permission.
• Same as above
• Display the secret as the root path, or don't display any path information at all.

[rnsc]

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

• What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

• What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

• What if I have LIST or READ access to a secret within a path but not the parents within that path?

Thanks!

For me it should silently not return anything that it cannot list or read in the sub tree. Now of course if the given path cannot even be listed, a 404 would be good. If LIST is ok but no secrets nor any sub path can be read a 404 or a 201 empty response would be appropriate imho.

If you want to recurse from a top level path and you only have read access to a secret from two levels down I'd argue returning a 201 would be the correct behaviour. As it wouldn't show up in the UI either would it?

That's just how I would implement this for me though with the predicate that people know that recursively iterating over a path that you don't have the appropriate ACL for your token set to do is a user error and that given how this works we cannot hold the hands of the user too much without implementing insane logic.

[joshsleeper]

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

* What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

* What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

* What if I have LIST or READ access to a secret within a path but not the parents within that path?
  Thanks!

I think I disagree with some of the suggestions above, mainly because the topic of this issue is about recursively LISTing keys, not READing the secret data at those keys, right?

1. keys that you can LIST but can't READ should absolutely present in the recursive list output. a simple label/indicator for keys that you don't have READ access to could be a nice enhancement, but imo the purpose of the recursive list command isn't to precisely enumerate all of your access permissions, but simply get an easy overview of the path/key structure.
2. for a sub-path/key where a user lacks LIST permissions I wouldn't expect anything under that sub-path to be present in the output, full stop. even if the user (strangely) has other permissions on the sub-path/key like READ or WRITE, the focus here is on recursively listing keys, and if LIST is restricted, it's seemingly not something the admins want the user to be able to enumerate. from a user-friendliness standpoint, I would like to see the output inform the user that the unlistable sub-path/key exists but the user lacks LIST permission to enumerate the path further.
3. this is a great question, and imo the answer is an error should be promptly returned by vault a recursive list logically can't start at a root the user doesn't have LIST access to, so attempting to do so at a path where LIST isn't permitted should return a nice simple error informing the user they don't have LIST permissions at that path.

[tysonkamp]

@finnstech This is great news. Is there any low/high confidence guess about when such functionality woudl be released? This quarter, this year?

[staehler]

Is there any progress on this issue? I'm keen on using this feature.

[bitroniq]

Also interested in this feature

[bastian-schlueter]

Same here. We had to super awkwardly implement width-first-search with limited depth in Terraform to be able to retrieve all the paths of secrets of a sub-tree.