fieri ssl errors

[kplimack]

supermarket-2.9.7-1.el6.x86_64

2017-03-06_02:28:10.77163 2017-03-06T02:28:10.771Z 30588 TID-zrkxk WARN: {"class":"MetricsRunner","args":[{"cookbook_name":"dmg","cookbook_version":"2.2.0","cookbook_artifact_url":"https://supermarket.myOrg.com/system/cookbook_versions/tarballs/670/original/dmg.tgz?1488766954","controller":"fieri/jobs","action":"create"}],"retry":true,"queue":"default","jid":"9507a140bf0413f9be0c5079","created_at":1488766954.5832617,"enqueued_at":1488767290.766189,"error_message":"SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol","error_class":"OpenSSL::SSL::SSLError","failed_at":1488766954.5861578,"retry_count":4,"retried_at":1488767290.7709894}
2017-03-06_02:28:10.77177 2017-03-06T02:28:10.771Z 30588 TID-zrkxk WARN: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol
2017-03-06_02:28:10.77179 2017-03-06T02:28:10.771Z 30588 TID-zrkxk WARN: /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock'
2017-03-06_02:28:10.77180 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:933:in `connect'
2017-03-06_02:28:10.77180 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
2017-03-06_02:28:10.77180 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:852:in `start'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:584:in `start'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:479:in `get_response'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/lib/ruby/2.3.0/net/http.rb:456:in `get'
2017-03-06_02:28:10.77181 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/supermarket_api_runner.rb:24:in `get_api_response'
2017-03-06_02:28:10.77182 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/supermarket_api_runner.rb:6:in `cookbook_api_response'
2017-03-06_02:28:10.77182 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/metrics_runner.rb:20:in `cookbook_api_response'
2017-03-06_02:28:10.77183 /opt/supermarket/embedded/service/supermarket/vendor/cache/fieri/app/models/metrics_runner.rb:7:in `perform'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:152:in `execute_job'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:134:in `block (2 levels) in process'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
2017-03-06_02:28:10.77184 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidetiq-0.7.2/lib/sidetiq/middleware/history.rb:8:in `call'
2017-03-06_02:28:10.77186 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2017-03-06_02:28:10.77186 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/logging.rb:32:in `with_context'
2017-03-06_02:28:10.77187 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:133:in `invoke'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:129:in `block in process'
2017-03-06_02:28:10.77188 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:168:in `stats'
2017-03-06_02:28:10.77190 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:128:in `process'
2017-03-06_02:28:10.77190 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:80:in `process_one'
2017-03-06_02:28:10.77190 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:68:in `run'
2017-03-06_02:28:10.77191 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/util.rb:17:in `watchdog'
2017-03-06_02:28:10.77191 /opt/supermarket/embedded/service/supermarket/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.1.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2017-03-06_02:28:10.77256 2017-03-06T02:28:10.772Z 30588 TID-ujnyo MetricsRunner JID-eedf88110b5276f6c179c972 INFO: start
2017-03-06_02:28:10.77547 2017-03-06T02:28:10.775Z 30588 TID-ujnyo MetricsRunner JID-eedf88110b5276f6c179c972 INFO: fail: 0.003 sec

[robbkidd]

A couple questions:

1. Do you want Fieri—the component that runs a growing number of community quality metrics—running in your private Supermarket? If not, omit fieri from the list of features enabled in the features attribute.
2. With HTTPS enabled, are you using a certificate for https://supermarket.myOrg.com signed by a trusted Certificate Authority?

[kplimack]

@robbkidd

1. I think i want it. foodcritic output isnt super helpful since my cookbooks go through CI to make their way to the supermarket and they must pass foodcritic, etc there. But id like to enable it and see for myself, or if any additional metrics make their way in.

2. my chef server and supermarket are sharing the same wildcard cert from an external CA, so i "assume" that it should work.

[nellshamrell]

@kplimack does your Supermarket instance run behind a load balancer? If it does, does SSL termination happen at the load balancer, or on the instance that Supermarket is running on?

[kplimack]

No. It's accessible without an LB

[nellshamrell]

Could you enter the rails console on the instance you are running Supermarket on? You can do this by SSH'ing to the instance and running $ sudo -u supermarket supermarket-ctl console

Could you let me know what it returns when you run this from within the console?

ENV['FIERI_URL']

[kplimack]

@nellshamrell looks like i also need to track down the source of all this spam about port 7777, too.

ENV['FIERI_URL']E, [2017-03-07T20:44:11.903797 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)

=> "http://localhost:13000/fieri/jobs"
irb(main):002:0> E, [2017-03-07T20:44:12.905355 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:13.906924 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:14.908478 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:15.909958 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:16.911600 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:17.913218 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:18.914706 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)
E, [2017-03-07T20:44:19.916183 #2568] ERROR -- : Failed to open TCP connection to localhost:7777 (Connection refused - connect(2) for "localhost" port 7777)

[jplimack@supermarket]~% curl -I http://localhost:13000/fieri/status
HTTP/1.1 200 OK

[robbkidd]

@kplimack Failed to open TCP connection to localhost:7777 is a byproduct of adding a monitoring agent to Supermarket. We have an issue open on that agent and will make a new release when the agent is updated with a fix.

[nellshamrell]

Alright, that FIERI_URL looks correct, and ty for demonstrating that you can curl it successfully. Taking another look at the stack trace.

[nellshamrell]

What do you have for ENV['FIERI_SUPERMARKET_ENDPOINT']? After taking another look at your stack trace, that seems to be what it is choking on.

[kplimack]

@nellshamrell "https://localhost:13000"

[robbkidd]

That's the default set in the omnibus install for FIERI_SUPERMARKET_ENDPOINT and I think it that may be the wrong default to set. The Rails app service running on localhost:13000 does not do SSL; SSL is handled higher up in the stack by nginx.

@kplimack If you were to override the fieri_supermarket_endpoint attribute—e.g. the same way you override fqdn—to set it to http://localhost:13000, I suspect the SSL error would stop and fieri job results would get posted successfully. If you try that, let us know if it succeeds (or doesn't). We'll get a fix in for that.

[kplimack]

@robbkidd thanks. I'll try that tomorrow. Would you happen to have any docs on integrating supermarket into a ci/cd pipeline?

[kplimack]

@robbkidd That seems to have done the trick!
I would love it if you had any docs on CI/CD workflows using a supermarket. In a chef-zero environment, it makes total sense, but in a client/server setup its role is not as well defined.