Heroku CI support

[zharikovpro]

When I run command locally, it works fine. It works fine on CodeShip, too. That same command fails on Heroku, though.

bundle exec rspec && bundle exec codeclimate-test-reporter

70 examples, 0 failures
Randomized with seed 16670
Coverage report generated for RSpec to /app/coverage. 3377 / 7074 LOC (47.74%) covered.
I, [2017-03-26T15:45:30.729786 #1192]  INFO -- : Reporting coverage data to Code Climate.
fatal: Not a git repository: './.git'
fatal: Not a git repository: './.git'
fatal: Not a git repository: './.git'
bundler: failed to load command: codeclimate-test-reporter (/app/vendor/bundle/ruby/2.3.0/bin/codeclimate-test-reporter)
CodeClimate::TestReporter::InvalidPayload: A git commit timestamp was not found in the test report payload
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter/payload_validator.rb:16:in `validate'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter/payload_validator.rb:11:in `validate'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter/formatter.rb:27:in `format'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter.rb:19:in `run'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/bin/codeclimate-test-reporter:16:in `<top (required)>'
  /app/vendor/bundle/ruby/2.3.0/bin/codeclimate-test-reporter:22:in `load'
  /app/vendor/bundle/ruby/2.3.0/bin/codeclimate-test-reporter:22:in `<top (required)>'
-----> test command `bundle exec rake factory_girl:lint && bundle exec rspec && bundle exec codeclimate-test-reporter` failed with exit status 1```

Any clues on how to fix this?

[dblandin]

Hi @zharikovpro,

When submitting test coverage data to Code Climate, we require a couple pieces of git commit information so that we can reliably link your test coverage report to the appropriate code:

• git commit sha
• git commit timestamp

If git is installed and we have access to a git repository, we can grab these values from git itself. Otherwise, we have to rely on environment variables set by the CI provider.

We fetch those environment variables, based on the CI provider, here: https://github.com/codeclimate/ruby-test-reporter/blob/master/lib/code_climate/test_reporter/ci.rb

From the output you shared, it seems like we don't have access to a git repository and we don't currently map values from Heroku CI set environment variables. After briefly looking at the Heroku CI docs I wasn't able to determine if these values are available during a CI run.

If they are, all that should be necessary is adding a new entry to the Ci class.

[zharikovpro]

Will forward this to the Heroku team, they could probably fix it, thanks.

[appleton]

Hello, I'm on the Heroku CI team and I'd love for this to work properly!

There's no git repo present on Heroku CI runs and I don't think we could add that without some significant work but I could expose some more information as environment variables.

We already have HEROKU_TEST_RUN_ID and HEROKU_TEST_RUN_BRANCH, I could add HEROKU_TEST_RUN_COMMIT_SHA if that helps?

[dblandin]

Hey @appleton! Thanks for following up!

A HEROKU_TEST_RUN_COMMIT_SHA environment variable would definitely help!

To completely allow test coverage submission from a Heroku CI build to Code Climate without a git repo, it would also help to have these additional environment variables available:

Name	Value
CI_NAME	"heroku" or "herokuci"
HEROKU_TEST_RUN_TIMESTAMP	Unix timestamp of the build time
HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP	Unix timestamp of the committed at time

The names can vary but these are the remaining bit of information we would need to generate a valid payload within the test reporter.

Thanks so much for your help! Let me know if you have any further questions.

[dwightwatson]

Any update on this one? The same issue is still present, it would be great to have this working with Heroku CI.

[a-b]

Please let us know ETA for this feature.

[a-b]

@appleton any updates on this issue?

[dblandin]

@a-b I haven't heard anything from the Heroku team. I'll try to get in touch with them via another channel and report back.

Thanks for the nudge!

[dblandin]

@a-b Also, as we're winding down development on this ruby-specific reporter, I'd recommend checking out our new test reporter and migrating to it if you're interested!

https://github.com/codeclimate/test-reporter https://codeclimate.com/changelog/591c99570d51f6481f001291

It's decoupled from the coverage tool used to generate the report, distributed as a static binary, and has support for parallelized builds. 🎉

[a-b]

Thanks for update @dblandin! It would be awesome to include couple words in README on how to configure this new test-reporter to work with Heroku CI for the rails app.

[joshwlewis]

Hey y'all -- Sorry for the delay here. @appleton has been on an extended PTO, so he's not around to respond. However, we did add HEROKU_TEST_RUN_COMMIT_VERSION (which reports recently in response to this (and similar issues).

I'll bring up the other suggestions (CI_NAME, HEROKU_TEST_RUN_TIMESTAMP, and HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP) with the team.

[dblandin]

Hey @joshwlewis,

Thank you for the update! Out of those three, HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP would definitely be the most important as it's a required value, either from the git repo itself (which is unavailable during Heroku CI runs) or the environment.

If we had that value available, we could address and close this issue easily. Thanks!

[dja]

If we used this solution: https://github.com/codeclimate/test-reporter/issues/226#issuecomment-335218428, what other ENV variables can we send to the ruby test reporter to ensure things run smoothly? (and how do we format them)

[dblandin]

Hey @dja,

That workaround is specific to the new test-reporter. We'd have to support it differently within this reporter.

I'd recommend opening a PR on the new test-reporter repo to support the Heroku env variables that are available and generate the committed at timestamp until Heroku CI supports it.

Here's a PR that recently added Drone CI support: https://github.com/codeclimate/test-reporter/pull/215

[dja]

Hey @joshwlewis, just following up here :) Any progress on making those variables available?

[pmaderamitlasf1]

@appleton , I am currently using HEROKU CI , unfortunately I can not run functional test which are part of my source code, because the review app url is dynamically changing , is it possible to have HEROKU_APP_NAME/ WEB URL accessible for HEROKU CI example :HEROKU_TEST_RUN_ID is available for my test script. Appreciate you help on this. OR do you suggest any workaround for this?

[appleton]

Hey @pmaderamitlasf1, I don’t work at Heroku any more but I’ll cc @joshwlewis ☝️

[pmaderamitlasf1]

Thanks @appleton . @joshwlewis can you please help me on this?