Different behaviour between different `ksm secret list` invocations

[kript]

Hi folks,

Apologies if I am driving this wrong, but according to the documentation there doesnt appear to be any difference between

• ksm secret init<TOKEN>; ksm secret list
• KSM_TOKEN="<TOKEN>" ksm secret list

However when I run the former I get output;

$ ksm profile init <TOKEN>
$ ksm secret list

 UID                     Record Type Title
 ----------------------- ----------- -----
 0FC-kP1ggocbyjBdjSusUA  login       igs

but when I run the latter I get an error;

$ KSM_TOKEN="<TOKEN>" ksm secret list
ksm had a problem: Invalid base64-encoded string: number of data characters (45) cannot be 1 more than a multiple of 4

I am running 1.1.1;

$ pip freeze | grep keeper
keeper-secrets-manager-cli==1.1.1
keeper-secrets-manager-core==16.6.3
keeper-secrets-manager-helper==1.0.4
keeper-secrets-manager-storage==1.0.2

[idimov-keeper]

Try setting token without quotes an without surrounding <> chars - tokens are either 43 or 46 (incl. host prefix) characters long and the error says that your token is 45 chars long, seems like either quotes or <> are included in the token string breaking it Try using the --token option (works with both padded and unpadded tokens): ksm profile init --token=US:XXX

[flybyray]

Actually providing KSM_TOKEN via environment variable is not working. But it should! KSM_TOKEN

$ KSM_TOKEN="<TOKEN>" ksm secret list
ksm had a problem: Invalid base64-encoded string: number of data characters (45) cannot be 1 more than a multiple of 4

I used it with KSM_TOKEN=EU:4Y1X....................................LRd0 ksm profile init Same or similar error.

KSM_DEBUG=DEBUG KSM_TOKEN=EU:4Y1XAbwCKgtAZuMPSnKanjHtfwDV-P3bOXHA4iLLRd0 ksm secret list
Traceback (most recent call last):
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/__main__.py", line 1383, in main
    cli(obj={"cli": None}, prog_name=program_name)
  File "~/.local/share/pipx/venvs/ansible/lib64/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/share/pipx/venvs/ansible/lib64/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "~/.local/share/pipx/venvs/ansible/lib64/python3.12/site-packages/click/core.py", line 1685, in invoke
    super().invoke(ctx)
  File "~/.local/share/pipx/venvs/ansible/lib64/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/share/pipx/venvs/ansible/lib64/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.local/share/pipx/venvs/ansible/lib64/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/__main__.py", line 303, in cli
    "cli": _get_cli(
           ^^^^^^^^^
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/__main__.py", line 152, in _get_cli
    return KeeperCli(**kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/__init__.py", line 43, in __init__
    self.profile = Profile(cli=self, ini_file=ini_file, config=global_config)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/profile.py", line 55, in __init__
    Profile.init(
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/profile.py", line 133, in init
    client = KeeperCli.get_client(config=config_storage)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/secrets-manager/integration/keeper_secrets_manager_cli/keeper_secrets_manager_cli/__init__.py", line 32, in get_client
    return SecretsManager(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/secrets-manager/sdk/python/core/keeper_secrets_manager_core/core.py", line 155, in __init__
    self._init()
  File "~/secrets-manager/sdk/python/core/keeper_secrets_manager_core/core.py", line 228, in _init
    existing_secret_key_bytes = url_safe_str_to_bytes(existing_secret_key)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/secrets-manager/sdk/python/core/keeper_secrets_manager_core/utils.py", line 99, in url_safe_str_to_bytes
    b = base64.urlsafe_b64decode(s + '==')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/base64.py", line 134, in urlsafe_b64decode
    return b64decode(s)
           ^^^^^^^^^^^^
  File "/usr/lib64/python3.12/base64.py", line 88, in b64decode
    return binascii.a2b_base64(s, strict_mode=validate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
binascii.Error: Invalid base64-encoded string: number of data characters (45) cannot be 1 more than a multiple of 4

[flybyray]

$ read -s KSM_TOKEN && ( echo  "$KSM_TOKEN" ) && unset KSM_TOKEN || unset KSM_TOKEN
test123
$ echo $KSM_TOKEN 

$ read -s KSM_TOKEN && ( ksm profile init  ) && unset KSM_TOKEN || unset KSM_TOKEN
Error: A one time access token is required either as a command parameter or an argument.
$ echo $KSM_TOKEN 

$ read -s KSM_TOKEN && ( export KSM_TOKEN ; ksm profile init  ) && unset KSM_TOKEN || unset KSM_TOKEN
ksm had a problem: Invalid base64-encoded string: number of data characters (45) cannot be 1 more than a multiple of 4
$ echo $KSM_TOKEN 

$ read -s KSM_TOKEN && ( ksm profile init --token $KSM_TOKEN ) && unset KSM_TOKEN || unset KSM_TOKEN
Added profile _default to INI config file located at ~/ansible/keeper.ini

read lines input: 1.,: test123 5.,9.,13.,: EU:token

[flybyray]

Workarround or maybe this is the not documented correct way todo it.

$ read -s S_KSM_TOKEN && ( KSM_TOKEN=${S_KSM_TOKEN#*:} KSM_HOSTNAME=${S_KSM_TOKEN%%:*} ksm profile init  ) ; unset S_KSM_TOKEN
Added profile _default to INI config file located at ~/ansible/keeper.ini
Error: A one time access token is required either as a command parameter or an argument.

[idimov-keeper]

How are you generating your tokens? Your token is missing a padding character at the end. Try appending equal sign at he end ('=' without quotes) for a quick test. Probably after a recent update Commander CLI started generating unpadded tokens. The next release of the KSM CLI tool should be able to handle both padded/unpadded base64 tokens

[flybyray]

How are you generating your tokens?

directly copied from the keeper desktop application

Your token is missing a padding character at the end.

i did not removed or added something. i expect the copied token should just work

Probably after a recent update Commander CLI started generating unpadded tokens.

Maybe. But if you would read my workarround command in full you would notice that there is some bash shell magic replacment done on the read environment token variable, so that i supply now 2 variables to ksm profile init.

KSM_TOKEN KSM_HOSTNAME

and that works! I just instpected the code base and really if someone with some functional skills would read several occasions where the token is handled pushed further procesed again on testing if it is splitt by hostname part ... ah ... just messy code. should be cleaned up. similar with the keeper_cache_dir and keeper_use_cache variables in different context needs to be cleanup. for that i already wrote an email but not sure if they will respond.

The next release of the KSM CLI tool should be able to handle both padded/unpadded base64 tokens

I will wait for it and test it . Hopefully it fixes it . Otherwise i try to write another mail. Hopefully someone is looking into it.

[idimov-keeper]

We found a bug in KSM CLI tool that's preventing the use of KSM_TOKEN environment variable. We are working on a fix.

As a workaround currently you can use --token option (working with both padded and unpadded tokens) ex. ksm profile init --token=US:XXX

or with custom ini file for testing: ksm profile init --ini-file=ksm.ini --token=US:XXX

[flybyray]

Thanks for the update

As a workaround currently you can use --token option (working with both padded and unpadded tokens) ex. ksm profile init --token=US:XXX

Will wait for the KSM_TOKEN environment variable. Need it to document inital workflows. read -s KSM_TOKEN is still one of the preferred ways to inject data instead of command arguments. But yes it is a workarround.

[idimov-keeper]

In the next release KSM CLI 1.1.6 we introduce new env variable KSM_CLI_TOKEN that works the same way as KSM_TOKEN but allows you to use command line options for the remaining parameters. Currently to use KSM_TOKEN you need to specify all non-default parameters as env vars (KSM_TOKEN, KSM_HOSTNAME, KSM_INI_DIR, KSM_INI_FILE, KSM_CLI_PROFILE) vs. only the token - (ex. with KSM_CLI_TOKEN) ksm profile init --ini-file=/tmp/custom.ini --profile=non_default

Note: Since KSM_TOKEN is consumed (by the SDK) too early during the CLI init phase it has no access to the command line parameters - hence the requirement to use the env vars (the whole lot if necessary)