Extend API to support Sidekiq

[mperham]

I'm trying to use Garnet from Ruby (I'd love to see Sidekiq running on Garnet!) and find myself blocked by case sensitivity.

% irb
irb(main):001> require "redis-client"
=> true
irb(main):002> r = RedisClient.new
=> #<RedisClient redis://localhost:6379>
irb(main):003> r.call("llen", "foo")
/Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client/connection_mixin.rb:71:in `call_pipelined': ERR unknown command (redis://localhost:6379) (RedisClient::CommandError)
	from /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client.rb:771:in `block in connect'
	from /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client/middlewares.rb:16:in `call'
	from /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client.rb:770:in `connect'
	from /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client.rb:732:in `raw_connection'
	from /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client.rb:697:in `ensure_connected'
	from /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.21.1/lib/redis_client.rb:277:in `call'
	from (irb):3:in `<main>'
	from /Users/mperham/.gem/ruby/3.2.0/gems/irb-1.12.0/exe/irb:9:in `<top (required)>'
	from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `load'
	from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `<main>'
irb(main):004> r.call("LLEN", "foo")
=> 0

Using v1.0.2.

07::46::01 info: GarnetServer[0] Garnet 1.0.2 64 bit; standalone mode; Port: 6379
07::46::01 info: ArgParser[0] Configuration import from embedded resource succeeded. Path: defaults.conf.
07::46::01 info: ArgParser[0] Configuration file path not specified. Using default values with command-line switches.
07::46::01 info: Options[0] [Store] Using page size of 32m
07::46::01 info: Options[0] [Store] Using log memory size of 16g

[lmaas]

Thank you for bringing this issue to our attention. What is happening here is that redis-client is issuing the RESP3 handshake (HELLO 3) during its initial connection to Garnet. However, because Garnet currently does not support RESP3, this results in an error response of ERR unknown command.

You should be able to circumvent this issue by using the redis gem instead, which should work with Garnet:

irb(main):001:0> require 'redis'
=> true
irb(main):002:0>  r = Redis.new
=> #<Redis client v5.1.0 for redis://localhost:6379>
irb(main):003:0> r.call("llen", "foo")
=> 0

[mperham]

Sidekiq 7 requires RESP3 because of the better type support. Does Garnet plan to support it? I don't see any mention on the public roadmap:

https://microsoft.github.io/garnet/docs/welcome/roadmap

[mperham]

...and the Redis 6.2 command set, things like BITFIELD_RO are used for queries. I'm not clear on which commands Garnet might be missing vs Redis 6.2.

[mperham]

Ah, it looks like Garnet doesn't support blocking list operations so it's a non-starter for job systems. I hope this will change in the future.

[badrishc]

Which specific blocking list APIs do you need to implement a job system? We can prioritize accordingly, this isn't very hard to add - we just didn't receive a use case for it yet.

[badrishc]

For Sidekiq to work correctly - if you can provide us a list of missing APIs that will be quite useful.

[mperham]

brpop, blmove and these:

https://github.com/sidekiq/sidekiq/blob/cee6f0eb36fe71a4300ccd715c0cf5bd2ab1d9d0/lib/sidekiq/redis_client_adapter.rb#L27-L32

[mardukbp]

The Python-cousin of sidekiq, Celery, also does not work because BRPOP is not supported. With Garnet one could finally be able to use Celery on Windows Server. This matters because Django (the Python-cousin of Rails) typically uses Celery for background jobs. Thank you for this awesome project!

[TalZaccai]

Hey @mperham! Quick update - we have blocking list operations added in PR #356. I was wondering if you might have time to run a quick test and see if Sidekiq should be supported now or if we're missing anything else. @mardukbp - LMK if you'd like to give Celery a try as well. Appreciate both your help!

[mperham]

Thank you, the other big item needed by packages like Sidekiq is the noeviction maxmemory policy from Redis. Jobs are transactional data and cannot be discarded like a cache. Does Garnet support this policy already?

[mperham]

@TalZaccai Unfortunately I'm on macOS and can't build a custom binary since dotnet only supports windows and linux according to your getting started page. If you can point me to a Docker image for your PR, I can use that.

https://microsoft.github.io/garnet/docs/getting-started

[TalZaccai]

Hey @mperham! You should still be able to build and run Garnet on macOS, we just haven't tested it on macOS.

[mardukbp]

@TalZaccai Thank you for the quick implementation :) I built the branch for #356 on Windows and executed the Celery hello world. It failed because Garnet has not implemented ZREVRANGEBYSCORE. I searched the source code of Celery and found that it also depends on EVALSHA, which Garnet has not implemented. I think with those two Celery is all set with Garnet :)

[TalZaccai]

Thanks for the quick reply @mardukbp! ZREVRANGEBYSCORE should be an easy addition, unfortunately EVALSHA might take longer as we don't currently support scripting in Garnet.

[mardukbp]

On second look, apparently EVALSHA is only a key in a dictionary, but otherwise it is not mentioned. So I guess it is not really necessary. Looking forward to trying the PR with ZREVRANGEBYSCORE!

[mperham]

Alas, Sidekiq migrated to RESP3 in v7 for better datatype support. That's a showstopper.

EVALSHA is not critical to Sidekiq so we can live without it (although Sidekiq Pro and Sidekiq Enterprise will not work without Lua).

[badrishc]

We do have basic RESP3 support in the latest. Would like to know what specific operator is not working correctly.

[mperham]

I mistook an error with flushall as an error with RESP3 in general.

The Sidekiq test suite is now running with about 90% passing.

With stock Redis 6.2:

src/sidekiq (main) % bundle exec rake
/Users/mperham/.gem/ruby/3.2.0/gems/mail-2.8.1/lib/mail/parsers/date_time_parser.rb:837: warning: statement not reached
/Users/mperham/.gem/ruby/3.2.0/gems/mail-2.8.1/lib/mail/parsers/date_time_parser.rb:691: warning: assigned but unused variable - testEof
Run options: --seed 43943

# Running:

................................./Users/mperham/src/sidekiq/test/api_test.rb:640: warning: Direct access to `Sidekiq::Work` attributes is deprecated, please use `#payload`, `#queue`, `#run_at` or `#job` instead
/Users/mperham/src/sidekiq/test/api_test.rb:641: warning: Direct access to `Sidekiq::Work` attributes is deprecated, please use `#payload`, `#queue`, `#run_at` or `#job` instead
/Users/mperham/src/sidekiq/test/api_test.rb:642: warning: Direct access to `Sidekiq::Work` attributes is deprecated, please use `#payload`, `#queue`, `#run_at` or `#job` instead
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Fabulous run in 2.693802s, 185.9825 runs/s, 667.4581 assertions/s.
501 runs, 1798 assertions, 0 failures, 0 errors, 0 skips

With Garnet 1.0.13:

src/sidekiq (main) % REDIS_URL=redis://localhost:3278/0 bundle exec rake
/Users/mperham/.gem/ruby/3.2.0/gems/mail-2.8.1/lib/mail/parsers/date_time_parser.rb:837: warning: statement not reached
/Users/mperham/.gem/ruby/3.2.0/gems/mail-2.8.1/lib/mail/parsers/date_time_parser.rb:691: warning: assigned but unused variable - testEof
Run options: --seed 14534

# Running:

..............................................E

Error:
Sidekiq::Web#test_0032_can delete scheduled:
TypeError: can't convert nil into Float
    <internal:kernel>:173:in `Float'
    lib/sidekiq/api.rb:511:in `initialize'
    lib/sidekiq/api.rb:713:in `new'
    lib/sidekiq/api.rb:713:in `block in fetch'
    lib/sidekiq/api.rb:711:in `each'
    lib/sidekiq/api.rb:711:in `each_with_object'
    lib/sidekiq/api.rb:711:in `fetch'
    lib/sidekiq/web/application.rb:286:in `block (2 levels) in <class:WebApplication>'
    lib/sidekiq/web/application.rb:285:in `each'
    lib/sidekiq/web/application.rb:285:in `block in <class:WebApplication>'
    lib/sidekiq/web/application.rb:416:in `instance_exec'
    lib/sidekiq/web/application.rb:416:in `block in call'
    lib/sidekiq/web/application.rb:414:in `catch'
    lib/sidekiq/web/application.rb:414:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-session-2.0.0/lib/rack/session/abstract/id.rb:272:in `context'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-session-2.0.0/lib/rack/session/abstract/id.rb:266:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-3.1.0/lib/rack/static.rb:161:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-3.1.0/lib/rack/builder.rb:277:in `call'
    lib/sidekiq/web.rb:118:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test.rb:360:in `process_request'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test.rb:163:in `custom_request'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test.rb:112:in `post'
    /Users/mperham/.rubies/ruby-3.2.0/lib/ruby/3.2.0/forwardable.rb:240:in `post'
    test/web_test.rb:517:in `block (3 levels) in <top (required)>'
    lib/sidekiq/config.rb:167:in `block in redis'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
    lib/sidekiq/config.rb:164:in `redis'
    test/web_test.rb:515:in `block (2 levels) in <top (required)>'

bin/rails test /Users/mperham/src/sidekiq/test/web_test.rb:513

E

Error:
Sidekiq::Web#test_0034_can retry all retries:
RedisClient::ReadTimeoutError: Waited 3 seconds
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/buffered_io.rb:214:in `block in fill_buffer'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/buffered_io.rb:197:in `loop'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/buffered_io.rb:197:in `fill_buffer'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/buffered_io.rb:187:in `ensure_remaining'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/buffered_io.rb:152:in `getbyte'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:191:in `block in parse_sequence'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:190:in `times'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:190:in `parse_sequence'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:167:in `parse_array'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:133:in `parse'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/ruby_connection.rb:96:in `read'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/connection_mixin.rb:31:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client.rb:279:in `block (2 levels) in call'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/middlewares.rb:16:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client.rb:278:in `block in call'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client.rb:699:in `ensure_connected'
    /Users/mperham/.gem/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client.rb:277:in `call'
    lib/sidekiq/redis_client_adapter.rb:36:in `block (2 levels) in <module:CompatMethods>'
    lib/sidekiq/api.rb:681:in `block (2 levels) in each'
    lib/sidekiq/config.rb:167:in `block in redis'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
    /Users/mperham/.gem/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
    lib/sidekiq/config.rb:164:in `redis'
    lib/sidekiq.rb:74:in `redis'
    lib/sidekiq/api.rb:680:in `block in each'
    lib/sidekiq/api.rb:677:in `loop'
    lib/sidekiq/api.rb:677:in `each'
    lib/sidekiq/api.rb:795:in `retry_all'
    lib/sidekiq/web/application.rb:245:in `block in <class:WebApplication>'
    lib/sidekiq/web/application.rb:416:in `instance_exec'
    lib/sidekiq/web/application.rb:416:in `block in call'
    lib/sidekiq/web/application.rb:414:in `catch'
    lib/sidekiq/web/application.rb:414:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-session-2.0.0/lib/rack/session/abstract/id.rb:272:in `context'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-session-2.0.0/lib/rack/session/abstract/id.rb:266:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-3.1.0/lib/rack/static.rb:161:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-3.1.0/lib/rack/builder.rb:277:in `call'
    lib/sidekiq/web.rb:118:in `call'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test.rb:360:in `process_request'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test.rb:163:in `custom_request'
    /Users/mperham/.gem/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test.rb:112:in `post'
    /Users/mperham/.rubies/ruby-3.2.0/lib/ruby/3.2.0/forwardable.rb:240:in `post'
    test/web_test.rb:545:in `block (2 levels) in <top (required)>'

bin/rails test /Users/mperham/src/sidekiq/test/web_test.rb:541

[...snip...]

........

Fabulous run in 126.974693s, 3.9457 runs/s, 11.5692 assertions/s.
501 runs, 1469 assertions, 7 failures, 60 errors, 0 skips

Looks like maybe there's an issue with the return value from hmget and returning scores with zrange withscores?

If you install Ruby 3.x, check out the Sidekiq source ([email protected]:sidekiq/sidekiq.git), run bundle and then REDIS_URL=redis://localhost:3278/0 bundle exec rake, you may be able to reproduce the failures locally.

[TalZaccai]

Ok yeah I'm able to reproduce locally. Getting slightly less failures with the blocking ops branch, but still:

Fabulous run in 103.937969s, 4.8202 runs/s, 14.3162 assertions/s.
501 runs, 1488 assertions, 5 failures, 54 errors, 0 skips

[badrishc]

@mperham - do we have everything we need for Sidekiq to work at this point?

[mperham]

I will need to test but won't be in the office until 8/26.

[mperham]

Kudos, I'm seeing very few errors now:

1. Sharding test errors due to missing support for database indices (the tests use db indexes as a proxy for different Redis shards)
2. An error due to differences in info output.
3. Errors due to the one Lua script which Sidekiq uses (the schedule poller).

The first two are incidental, the third would require me to craft a fallback, which is doable.

I think it's possible to get Sidekiq working but since Sidekiq Pro/Ent require Lua, it's doubtful I can officially support Garnet long term until Lua is on the roadmap.