Multiple engines

[waghanza]

Hi,

This PR allows to test each frameworks on various setups.

Actual

All frameworks are tested once. For example, ruby framework are tested with puma (since widely spread in ruby communities)

Purpose

Test framework with multiple engines (for the sake of this feature we keep focus on engines, not compilation options, vms, ...). The idea the is to test frameworks with various engines, when accurate indeed.

---

List of engines

---

• Ruby
  • https://puma.io
  • https://github.com/socketry/falcon
  • https://github.com/boazsegev/iodine
• Javascript
  • https://github.com/nodejs/node
  • https://deno.land
  • https://github.com/uNetworking/uWebSockets.js
• Python
  • https://gitlab.com/pgjones/hypercorn
  • https://github.com/django/daphne
  • https://github.com/encode/uvicorn
  • https://github.com/benoitc/gunicorn
  • https://github.com/jonashaag/bjoern
  • https://github.com/Pylons/waitress
• PHP
  • https://github.com/php/php-src, aka php-fpm
  • https://github.com/swoole/swoole-src
  • https://github.com/walkor/Workerman
  • https://github.com/reactphp/reactphp
  • https://github.com/amphp/amp
  • https://roadrunner.dev/

[ahopkins]

Great idea. You can also add uvicorn for Python (ASGI).

[waghanza]

Is uvicorn suitable for production now ? I mean we are using gunicorn (with uvicorn workers), but perhaps this is outdated @ahopkins

[ahopkins]

That pattern is valid and mainly used because gunicorn has a rich feature set. For the ASGI frameworks, I would suggest variants:

• hypercorn
• Daphne
• uvicorn
• gunicorn + uvicorn

As you can see, I imagine having all these variants across all the languages is likely to lead to a long list. Sanic (for example) can be run directly by gunicorn, by any of the above methods, and also using its own internal server.

Have you had any thoughts of how to present all this data without being too verbose? Maybe the variants list is displayed only by language?

[waghanza]

Maybe a discussion should be a great place to collect ideas

[dominikzogg]

@waghanza would be cool if those variants are structured like php/chubbyhp/php-fpm, php/chubbyphp/swoole... would be easier to maintain

[waghanza]

@waghanza would be cool if those variants are structured like php/chubbyhp/php-fpm, php/chubbyphp/swoole... would be easier to maintain

Yep. This is what I opted for

Example, for rails

|
-- ruby
           |
           --- rails
                        |
                        --- all files for implementation
                        |
                        .falcon <- specific file for `falcon` variant
                        |
                        .iodine <- specific file for `iodine` variant

The idea is too keep the basics, and change file if required (composer.json, as long as source code, extensions could differs)

And also, some variant requires custom docker images (graalvm in java)

[waghanza]

@the-benchmarker/web-frameworks I have successfully enable variants for ruby. The implementation is not ideal, but can work as a MVP.

If you have time to try it, I'll be glad to review any PR (and have some feedbacks) ❤️

The idea is :

• define glob patterns to list files at language level
• define variants at framework level You can see any config.yaml im ruby folder

[waghanza]

@dominikzogg Any idea about context https://github.com/the-benchmarker/web-frameworks/pull/3958/checks?check_run_id=3899021842#step:6:58 ?

[dominikzogg]

@waghanza it seems the npm install is missing

[waghanza]

ah of course, npm install should be made before compilation

[ariaieboy]

you can add roadrunner for php

[ahopkins]

Passing CI!! :tada: Does that mean this gets merged sometime soon? What is the plan for finalizing this?

[waghanza]

Will try to stabilize first (some frameworks are still failing), and need to think how to display results (if I merge this as-is it will break UI) @ahopkins

[ahopkins]

Will try to stabilize first (some frameworks are still failing), and need to think how to display results (if I merge this as-is it will break UI) @ahopkins

Spot checking some of the Python failures, seems like some missing wheels maybe? I feel like this might be a recurring problem with this strategy as requirements are constantly shifting. Can we just display some message about build failures on those combinations?

[waghanza]

For now, I prefer to show failures by hand (dig into the log myself), but having a feature that can expose it automate is a great plus (and I plan to add it at some point).

Which framework are you refereing to @ahopkins ?

[ahopkins]

Which framework are you refereing to @ahopkins ?

I don't remember specifically which, but I looked at a few of the Python builds and they all seemed to die with some failure to install dependencies: usually multidict and sometimes yarl.

[waghanza]

I think a lot of python failures are due to uwsgi @ahopkins. Let's see if non-LTS version will fix smth

[waghanza]

@adamluzsi rack-app seems to timeout, but only with puma (camping also)

[waghanza]

@adamluzsi

/usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli/fetcher/server.rb:1:in `require': cannot load such file -- rack/server (LoadError)
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli/fetcher/server.rb:1:in `<top (required)>'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli/fetcher.rb:3:in `require'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli/fetcher.rb:3:in `<module:Fetcher>'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli/fetcher.rb:1:in `<top (required)>'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli.rb:4:in `require'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli.rb:4:in `<class:CLI>'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app/cli.rb:2:in `<top (required)>'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app.rb:11:in `require'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app.rb:11:in `<class:App>'
	from /usr/local/bundle/gems/rack-app-7.7.0/lib/rack/app.rb:5:in `<top (required)>'
	from config.ru:3:in `require'
	from config.ru:3:in `block in <main>'
	from /usr/local/bundle/gems/rack-3.0.0/lib/rack/builder.rb:106:in `eval'
	from /usr/local/bundle/gems/rack-3.0.0/lib/rack/builder.rb:106:in `new_from_string'
	from /usr/local/bundle/gems/rack-3.0.0/lib/rack/builder.rb:97:in `load_file'
	from /usr/local/bundle/gems/rack-3.0.0/lib/rack/builder.rb:67:in `parse_file'
	from /usr/local/bundle/gems/puma-6.0.0/lib/puma/configuration.rb:364:in `load_rackup'
	from /usr/local/bundle/gems/puma-6.0.0/lib/puma/configuration.rb:286:in `app'
	from /usr/local/bundle/gems/puma-6.0.0/lib/puma/runner.rb:158:in `load_and_bind'
	from /usr/local/bundle/gems/puma-6.0.0/lib/puma/single.rb:44:in `run'
	from /usr/local/bundle/gems/puma-6.0.0/lib/puma/launcher.rb:186:in `run'
	from /usr/local/bundle/gems/puma-6.0.0/lib/puma/cli.rb:75:in `run'
	from /usr/local/bundle/gems/puma-6.0.0/bin/puma:10:in `<top (required)>'
	from /usr/local/bundle/bin/puma:25:in `load'
	from /usr/local/bundle/bin/puma:25:in `<top (required)>'
	from /usr/local/lib/ruby/3.1.0/bundler/cli/exec.rb:58:in `load'
	from /usr/local/lib/ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'
	from /usr/local/lib/ruby/3.1.0/bundler/cli/exec.rb:23:in `run'
	from /usr/local/lib/ruby/3.1.0/bundler/cli.rb:484:in `exec'
	from /usr/local/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /usr/local/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /usr/local/lib/ruby/3.1.0/bundler/cli.rb:31:in `dispatch'
	from /usr/local/lib/ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /usr/local/lib/ruby/3.1.0/bundler/cli.rb:25:in `start'
	from /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.3.7/libexec/bundle:48:in `block in <top (required)>'
	from /usr/local/lib/ruby/3.1.0/bundler/friendly_errors.rb:103:in `with_friendly_errors'
	from /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.3.7/libexec/bundle:36:in `<top (required)>'
	from /usr/local/bin/bundle:25:in `load'
	from /usr/local/bin/bundle:25:in `<main>'

[adamluzsi]

As long as the application code is loaded once, there shouldn't be an issue. rack-app has no web server-specific logic. The only thing I can think of is the initial application routing building that happens on the first call.

[waghanza]

and seems to be with the last major of puma @adamluzsi https://github.com/puma/puma/blob/master/6.0-Upgrade.md

[jeremyevans]

@waghanza The rack/server issue is because that code was extracted to a gem named rackup when rack 3 was released. So if you add the rackup gem, it should continue to work.

[adamluzsi]

Can you give it a try to not use the CLI? Maybe it's a client issue only; try using puma directly from the command line.

[adamluzsi]

Okay, adding rackup, or deprecating the cli altogether sounds like the options.

[waghanza]

Up to you then @adamluzsi. Maybe add this on rack-app side is accurate 😛

[adamluzsi]

Okay, I go with deprecating the CLI because the puma cli should work out of the box with the framework. However, I need time to delve into these changes.

Could you replace the rack-app CLI usage with the puma CLI?

[waghanza]

Not sure to understand @adamluzsi. The app run with bundle exec puma

[adamluzsi]

Okay, I included the rackup gem in the lib to avoid doing work for you. Try to upgrade to the latest version (v8).

[waghanza]

@jeremyevans seems that also camping suffer for puma 6 migration (but I'm not even sure if keeping it here is valuable)

[jeremyevans]

Camping says it supports rack 1+, but it doesn't look compatible with rack 3 (https://github.com/camping/camping/blob/master/lib/camping-unabridged.rb#L16 shows uppercase header, when all headers must be lowercase in rack 3). This is known upstream, but not yet fixed: https://github.com/camping/camping/issues/130

Maybe leaving camping at puma 5, or temporarily disabling benchmarking until it fixed, seems like the best choice of action.