Logstash keystore / run as service / missing password / error message / misleading.

[jakelandis]

When Logstash is run as service and the password is not available to the running service via /etc/sysconfig/logstash the following error message may occur:

[2018-06-06T16:52:47,582][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::LoadException", :message=>"Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.", :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:265)"

The error message is misleading because the keystore may indeed be a valid Logstash keystore. This can be checked with the following command:

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list

If you can list the keys of the keystore, it is a valid keystore.

When run a service (RPM/DEB) the password as needs to be available to the service via the /etc/sysconfig/logstash file as described in the documentation , not via the normal .bash* environment files.

This error may be fixed by correctly configuring the /etc/sysconfig/logstash file :

sudo mkdir /etc/sysconfig
sudo vim /etc/sysconfig/logstash
add the following (replace with actual password): 
LOGSTASH_KEYSTORE_PASS=mypassword
sudo chmod 600 /etc/sysconfig/logstash
sudo systemctl stop logstash.service (may have to force kill it)
sudo systemctl start logstash.service 

This issue is logged to fix the error message so that it is not so misleading. Also note this is only applicable when run as service via RPM/DEB based installations.

[mellieA]

@jakelandis would this happen if one had configured BOTH env variables and the file?

[jakelandis]

I haven't tested, but I am pretty sure that the the .bash* environment entries are only sourced upon login and is not read by our service (hence the need for the file).

Are you seeing this error with the /etc/sysconfig/logstash set ?

[mellieA]

@jakelandis Yes

[mellieA]

@jakelandis - just posting the additional information I received about sysv:

For Ubuntu 14.04 the file containing the LOGSTASH_KEYSTORE_PASS variable must be /etc/default/logstash. The password must also be in double quotes.

You must also not create the /etc/sysconfig/logstash file because the service script /etc/init/logstash.conf sources both the /etc/default/logstash file but also the /etc/sysconfig/logstash file which will also cause things to fail.

[lucasmogari]

@jakelandis - just posting the additional information I received about sysv:

For Ubuntu 14.04 the file containing the LOGSTASH_KEYSTORE_PASS variable must be /etc/default/logstash. The password must also be in double quotes.

You must also not create the /etc/sysconfig/logstash file because the service script /etc/init/logstash.conf sources both the /etc/default/logstash file but also the /etc/sysconfig/logstash file which will also cause things to fail.

Thanks for pointing that out. It's the same for Centos7.

[pedrosk]

Was wondering if this issue is present in Logstash version 7.7 on a CentOS system.

[roaksoax]

Potential cause and workaround that might be surfacing this misleading error message:

Can you confirm that the /etc/default/logstash file can be read by the logstash user and isn't for root only?

Eg. BAD:

$ sudo ls -lha /etc/default/ | grep logstash
-rw-------  1 root root  304 Jan  8 13:20 logstash
Eg GOOD:

$ sudo ls -lha /etc/default/ | grep logstash
-rw-r--r--  1 root root  304 Jan  8 13:20 logstash

[S3l3ct3d]

I am currently working through this issue as well. I built a new RHEL 7 box for a DEV/QA Logstash server.

Installed logstash through with yum, no issues. My logstash.yml file was initially hard coded with my cloud auth and id credentials. I ran through the documentation that is provided for the keystore. set +o history export LOGSTASH_KEYSTORE_PASS=mypassword set -o history sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create

Gave me a created keystore at /etc/logstash/logstash.keystore

I am then able to add all my keys in the key store, which is four in total. I can then run the command to list which shows my keys listed.

I start logstash as normal: sudo systemctl start logstash.

Logstash service shows as running but in journalctl it shows the following:

logstash[6919]: ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash, exception=Java::OrgLogstashSecretStore::SecretStoreException::AccessException, message=>Can not access Logstash keystore at /etc/logstash/logstash.keystore. Please verify correct file permissions and keystore password.

So, I started looking at permissions. Which led me here. But, per my elastic support technician my permissions are correct.

So the OP stated that the error can be misleading, I I can do all these steps prior to actually trying to start logstash.service. Once I start the service, and it fails.

I then try to run the list command again:

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list

and I get this error now: [ERROR] 2021-07-26 19:04:40.540 [main] secretstorecli - Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore. {:cause=>java.io.IOException: Integrity check failed: java.io.IOException: getSecretKey failed: Password is not ASCII, :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:284)", "org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:59)", "org.logstash.secret.store.SecretStoreFactory.doIt(org/logstash/secret/store/SecretStoreFactory.java:129)", "org.logstash.secret.store.SecretStoreFactory.load(org/logstash/secret/store/SecretStoreFactory.java:115)", "org.logstash.secret.cli.SecretStoreCli.command(org/logstash/secret/cli/SecretStoreCli.java:97)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)", "usr.share.logstash.lib.secretstore.cli.<module:SecretStoreCli>(/usr/share/logstash/lib/secretstore/cli.rb:52)", "usr.share.logstash.lib.secretstore.cli.<main>(/usr/share/logstash/lib/secretstore/cli.rb:33)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:710)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:1205)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1128)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1146)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:958)", "org.jruby.Main.doRunFromMain(org/jruby/Main.java:400)", "org.jruby.Main.internalRun(org/jruby/Main.java:292)", "org.jruby.Main.run(org/jruby/Main.java:234)", "org.jruby.Main.main(org/jruby/Main.java:206)"]}

Any insights would be greatly appreciated.

Thanks,

[mwangiwambui]

I am using RHEL, and i was able to fix it be

1. Defining my LOGSTASH_KEYSTORE_PASS in /etc/default/logstash. I also used double quotes to store the password in the file. e.g. LOGSTASH_KEYSTORE_PASS="thisismypassword"
2. After setting this I created a new keystore and I made sure i first exported the variable with password as with the documentation here