Seyren not picking up alerts from graphite

[siva-sfly]

Hi,

We've a graphite server with seyren installed on the same server. I recently moved seyren to a different server (exported mongodb data from the old server and imported into the new server). After I start seyren, I only see the alerts from the migrated data. I don't see any new alerts? Any idea how to fix it?

Thanks, Siva

[alenkacz]

do you see anything in the log?

[siva-sfly]

Here's a gist of the seyren.log on startup https://gist.github.com/siva-sfly/f483224345a9326835219a53268fb810. I see connections made to mongodb but not to graphite initially. After a long time when a check tried to run, I see some graphite connection errors.

[siva-sfly]

I tried making a sample connection to graphite using SSLPoke. Here's the error. Do I need some kind of ssl certs to make this work?

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138)
        at SSLPoke.main(SSLPoke.java:31)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
        ... 15 more

[siva-sfly]

@alenkacz Any idea how to fix the error I posted in the previous comment?

[alenkacz]

@siva-sfly your graphite runs on https? because the ssl manager is used only for graphite on ssl. Can you post also snippet with how seyren is started (the config)

[siva-sfly]

@alenkacz Yes, our graphite runs on https. Here's the seyren startup config.

GRAPHITE_URL="https://graphite.domain"
export GRAPHITE_URL

MONGO_URL="mongodb://localhost:27017/seyren"
export MONGO_URL

GRAPHITE_USERNAME="graphite_user"
export GRAPHITE_USERNAME

GRAPHITE_PASSWORD='graphite_pass'
export GRAPHITE_PASSWORD

SEYREN_URL="http://localhost:8080/seyren"
export SEYREN_URL

SEYREN_LOG_PATH="/var/log/seyren/"
export SEYREN_LOG_PATH

PIDFILE='/var/run/seyren/seyren.pid'
CHECK_PID_RUNNING=$(ps -ef | grep seyren | grep -v grep | awk '{print $2}')

USER=seyren
GROUP=seyren


start-stop-daemon --start --make-pidfile --chuid seyren --pidfile /var/run/seyren/seyren.pid --exec /usr/bin/java -- -jar /opt/seyren/seyren-web-1.4.0.jar -extractDirectory /opt/seyren/.extract

[siva-sfly]

@alenkacz Any new ideas based on my previous comments? We're still not able to get this working.

[howdoicomputer]

This can probably be closed out. The problem was indeed due to SSL certs. I created a keystore, added in the proper certs that matched with the domain that Seyren was supposed to communicate with, and then passed in the keystore via jvm options.

java -jar /opt/seyren/seyren-1.5.0.jar -Djavax.net.ssl.trustStore=/opt/seyren/ssl/clientkeystore -Djavax.net.ssl.trustStorePassword=superdupersecret

Once that was done, it took a while for Seyren to 'catch up' since there was a lot of old data in mongo.

For anyone coming across this thread, here are the rough steps for making Seyren SSH friendly:

keytool -keystore clientkeystore -genkey -alias client
keytool -import -keystore clientkeystore -file yourservercert.cert -alias yourserverdomain

Then include the path to the store either with environment variables or, like above, jvm options.