improved steps to run locally

[danmayer]

hey this is a great project and I am looking for a way to test the impact of some gems and patches to Ruby and how they impact app performance. I have found tiny microbenchmarks not that accurate and think this kind of project could be a good approach... By running the benchmark on my system then patching Ruby and running again. I guess first is that a good use case for your project, do you intend for folks to be able to use it this way?

After that, I have had some issues to get it up and running on my new OSX system and ran into some issues.

• it appears discourse migrations have changed to run the db seeds, which means the DB always has data so the script exits saying it expects an empty DB. (I resolved that by disabling the seeds)
• issues with the gem / bundler environment between the host project benchmarking and the discourse app. * I have ChRuby installed and working * when this line runs it always bombs out not being able to find gems require File.expand_path(File.join(File.dirname(__FILE__), "work/discourse/config/environment")) * I can cd into the discount project and run any rake tasks or start the server without issue.

I was hoping the README could be updated with a bit more detailed instructions on getting it running locally, and possibly which tools you have tested / know it works with (IE I am happy to switch to RbEnv or the like if that is known to work)

[noahgibbs]

I definitely do intend folks to use it that way. I usually install multiple Rubies, patched and unpatched, instead of running-patching-running.

There are definitely some issues with OS/X, and with newer Discourse. I've fixed some, but not all, of the newer Discourse issues and I'm working on the others. You can find a SHA of Discourse that works in packer/ami.json.

I primarily use the benchmark via AWS, basically because running locally is always hard for exact benchmarking - you can never be sure what else is running on your system. Using a dedicated instance with no UI is always better for that.

[danmayer]

great, yeah makes sense about installing multiple rubies... I actually have started to do that from my fork on Ruby trunk. I will take a look at the specific SHA and perhaps take a look at just spinning up the AWS server to run things on.

[noahgibbs]

Sounds good. There are AMIs listed in the README if you're looking for known-good AWS configuration. You could also (manually or with Packer) install other Rubies on top of the known-good AMIs if you want an apples-to-apples comparison with specific Ruby patches.

[danmayer]

OK as you saw in my post, I ended up doing some benchmarking with a simpler setup for Ruby patch... As I finished up some of that work I am returning to getting Rails Ruby Bench running so I can get a far more robust process / flow for testing some gem and possibly Ruby patches.

This time going through setup on AWS opposed to running locally. I still think a bit of effort to make it easier to get the benchmark up and running would help with adoption.

From the readme there are two suggested public images, but not specifying the required region...

 aws ec2 run-instances --image-id ami-f678218d --count 1 --instance-type m4.2xlarge --key-name my_key_name --placement Tenancy=dedicated

An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id '[ami-f678218d]' does not exist

I normally run in the EU, but from the packer readme guessed this should be in US-EAST, so changed my config and got authorization errors, for both of the images on the readme.

aws ec2 run-instances --image-id ami-f678218d --count 1 --instance-type m4.2xlarge --key-name my_key_name --placement Tenancy=dedicated

An error occurred (AuthFailure) when calling the RunInstances operation: Not authorized for images: [ami-f678218d]

aws ec2 run-instances --image-id ami-554a4543 --count 1 --instance-type m4.2xlarge --key-name my_key_name --placement Tenancy=dedicated

An error occurred (AuthFailure) when calling the RunInstances operation: Not authorized for images: [ami-554a4543]

back to the packer file to see if I can find newer ones... I search the AWS console community images for rails-ruby-benchmark finding ami-36cb5820

Then tried with that, but had an error about a subnet

 aws ec2 run-instances --image-id ami-36cb5820 --count 1 --instance-type m4.2xlarge --key-name my_key_name --placement Tenancy=dedicated

An error occurred (VPCResourceNotSpecified) when calling the RunInstances operation: The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.

I resolved that by looking up my default

aws ec2 run-instances --image-id ami-36cb5820 --count 1 --instance-type m4.2xlarge --key-name my_key_name --placement Tenancy=dedicated --subnet-id subnet-MY_DEFAULT
...
SUCCESS

Thinking perhaps linking to something from the readme would be better than specific AMIs, which seem to go away. Making clear the region could be helpful.

I am happy to send some PRs related to the readme, but now with a up and running AMI, I still have issues...

after SSHing into the box, I cd into the project and originally didn't see the file referenced in the readme

ubuntu@ip-10-0-1-226:~/rails_ruby_bench$ ls
alice.txt  Gemfile.lock        packer           start.rb           work
COPYING    INSTALL.md          README.md        TODO
Gemfile    multi_ruby_test.sh  seed_db_data.rb  user_simulator.rb
ubuntu@ip-10-0-1-226:~/rails_ruby_bench$ git branch
* latest-development
ubuntu@ip-10-0-1-226:~/rails_ruby_bench$ git checkout master
Branch master set up to track remote branch master from origin.
Switched to a new branch 'master'
ubuntu@ip-10-0-1-226:~/rails_ruby_bench$ git pull
Already up-to-date.
ubuntu@ip-10-0-1-226:~/rails_ruby_bench$ ls
alice.txt     graph            only_startup.rb  seed_db_data.rb      user_simulator.rb
COPYING       in_each_ruby.rb  packer           start.rb             work
Gemfile       INSTALL.md       process.rb       TODO
Gemfile.lock  multi_start.rb   README.md        update_discourse.sh
ubuntu@ip-10-0-1-226:~/rails_ruby_bench$ ./in_each_ruby.rb "for i in {1..2}; do ./start.rb -i 3000 -w 100 -s 0; done"
./in_each_ruby.rb:5:in `read': No such file or directory @ rb_sysopen - /home/ubuntu/benchmark_ruby_versions.txt (Errno::ENOENT)
	from ./in_each_ruby.rb:5:in `<main>'

I started to debug a bit but was hitting similar issues to running locally on OSX where bundler seemed unhappy...

# checked out master to get the missing in_each_ruby file...
# bundled installed
# then hit this
 bundle exec ./start.rb
bundler: failed to load command: ./start.rb (./start.rb)
LoadError: cannot load such file -- rails/all

I don't know if you have time or interest, but if you wanted to pair up some time to try to have someone else go through setting it up while documenting all the steps, I would be happy to set up some remote pairing time where we could try to get it running either on OSX or AWS, and document all the steps needed for someone that doesn't have your current setup.

[noahgibbs]

Hey, Dan! Two things come to mind here.

One is that you may want to use Packer to build your own images. You found mine (well done!) and it's wonderful that you got things working as far as you did. But you may have an easier time that way. I'm not sure why you're not finding ~ubuntu/benchmark_ruby_versions.txt -- those should be included in every image as it's built, including the public ones. I could try building you a more recent one since it's been a bit since my last public build...

The other thing that seems relevant is that, over time, I'm coming to agree with the Docker fans that that's going to be the right approach. It's annoying in a lot of ways (e.g. hard to "reach into" the container to mess with the Discourse config files, hard to pass data around, some benchmarking concerns with filesystem performance.) But especially for the Mac and local usage stories, it's hard to imagine any other approach working long-term.

I don't have a Dockerized version yet, though I'll probably be starting from the Docker version I put together for Ruby-bench.org awhile back.

That is to say: it would be very useful to have you work through the steps, as it has been for AWS. But I'm worried about asking you to do that with an approach that's about to change :-/

[danmayer]

OK I am going to work on some other related things for awhile but will keep watching this let me know when you have another setup you want me to give a go.

[joel]

This initiative is really awesome, I think it would be great to be able to bench any rails project.

I wanted to dig a bit, but making work locally it kind of not working.

I ran into missing dependency cannot load such file -- gabbler (LoadError)

Not loading file No such file to load -- lib/discourse.rb (LoadError)

I fixed like

diff --git a/config/application.rb b/config/application.rb
index 3eb998adca..93c876661c 100644
--- a/config/application.rb
+++ b/config/application.rb
@@ -75,11 +75,14 @@ module Discourse
     # this pattern is somewhat odd but the reloader gets very
     # confused here if we load the deps without `lib` it thinks
     # discourse.rb is under the discourse folder incorrectly
-    require_dependency 'lib/discourse'
-    require_dependency 'lib/js_locale_helper' 
+    require_dependency Rails.root.join('lib/discourse')
+    require_dependency Rails.root.join('lib/js_locale_helper')
 
     # tiny file needed by site settings
-    require_dependency 'lib/highlight_js/highlight_js'
+    require_dependency Rails.root.join('lib/highlight_js/highlight_js')
 
     # mocha hates us, active_support/testing/mochaing.rb line 2 is requiring the wrong
     #  require, patched in source, on upgrade remove this

missing database

FATAL: database "discourse_profile" does not exist (ActiveRecord::NoDatabaseError)

into missing Redis

Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)

And abandoned here 😁

PG::UndefinedTable: ERROR:  relation "theme_fields" does not exist (ActiveRecord::StatementInvalid)
LINE 8:  WHERE a.attrelid = '"theme_fields"'::regclass

[noahgibbs]

As far as benching any Rails app, keep in mind that at minimum you'll need to come up with a set of test routes for any Rails app. An awful lot of the interesting code for Rails Ruby Bench is, by its nature, specific to Discourse - simulating users reading forum comments and posting, for instance.

[noahgibbs]

For discourse_profile, you'll probably need to "rails db:create:all db:migrate:all" or similar. Though you could just for the profile environment if you'd rather.