api-benchmarks
api-benchmarks copied to clipboard
Published
20 hours ago •
davidcelis
davidcelis
Heroku benchmarks sometimes error out
Sometimes running benchmarks on Heroku will result in this error:
rake aborted!
undefined method `[]' for nil:NilClass
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark/table.rb:33:in `parse_results'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark/table.rb:12:in `add_row'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark/framework.rb:18:in `benchmark!'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark.rb:22:in `block (2 levels) in run!'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark.rb:21:in `each'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark.rb:21:in `block in run!'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark.rb:16:in `each'
/Users/david/Documents/Code/api-benchmarks/lib/api/benchmark.rb:16:in `run!'
/Users/david/Documents/Code/api-benchmarks/lib/tasks/benchmark.rake:28:in `block (3 levels) in <top (required)>'
Tasks: TOP => benchmark:heroku:all
(See full trace by running task with --trace)
It doesn't appear to have anything to do with a particular framework. I suspect that some sort of timeout occurs, which changes the output structure of wrk.
Have you noticed any differences between testing against a free vs hobby vs standard vs professional vs performance dyno?
