Benchmark for performance and memory usage

[FooBarWidget]

Issue fullstaq-labs/fullstaq-ruby-server-edition#7 described tuning additional settings for server-friendliness. I think that's a good idea, but one potential caveat is that we tune things wrongly. Of course it's impossible to tune things in a way that it's better for all workloads, but at the very least I think we should have some kind of performance and memory usage benchmark. However imperfect such a benchmark may be, having something is better than doing guesswork with no benchmark.

A standardized benchmark suite will also help us compare, when we implement new improvements (like https://github.com/fullstaq-labs/fullstaq-ruby-server-edition/issues/57), or when we research new memory allocators (like #4 and #6).

There are currently third party benchmark reports, but they are erratic and are hard to reproduce, so we can't rely on them.

How would such a benchmark look like? Can we reuse existing benchmarks or do we have to write a new one? Help on building or integrating a benchmark would be much appreciated.

The first goal is to have a benchmark suite. Once we have it, a second goal (which will be codified in a separate issue) is to integrate the benchmark in a CI and monitoring system (https://github.com/fullstaq-labs/fullstaq-ruby-server-edition/issues/58), so that we can automatically see the impact of changes over time.

[noahgibbs]

That makes sense. Have you thought about any workloads you especially want to tune this for? And how heavily you want to emphasize Rails and/or high concurrency?

[FooBarWidget]

I think we need both a Rails-centric and a non-Rails centric benchmark. Rails is popular so we should definitely take it into account. But Rails is also notoriously hard to performance-optimize for, so I'm contend with "not making Rails slower".

We will want to emphasize high concurrency, because that's what Ruby apps are moving towards (with Puma being the default for Ruby), and because the memory bloating issues mainly affect multithreaded apps.

I wonder how useful @schneems's derailed benchmarks are for this purpose. Maybe optcarrot is also useful.

[FooBarWidget]

And the memory benchmark I used in my research is basically a bunch of threads that all perform this:

def stress_thread_main
  ary = []
  File.open('/dev/zero', 'rb') do |f|
    while !quit?  # becomes true after 5 min
      ary << f.read(1024)
      if ary.size == 50
        ary.clear
      end
    end
  end
end

I don't know how much help this is, but it's better than nothing.