redis-rb
redis-rb copied to clipboard
Redis::ConnectionError: Connection lost (EPIPE) when write large data to sorted set in a transaction
$redis.multi do |multi| multi.zadd 'my_key', my_arr end
'my_arr' contains about 1 million records. When reduce 'my_arr' to 0.5 million records, it worked well. All records can be zadded successfully one by one, all stored data costs 130M memory.
The backtrace is as following:
rake aborted! Redis::ConnectionError: Connection lost (EPIPE) /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:257:in
rescue in io'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:250:in io' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:269:in
write'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:228:in block (3 levels) in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:222:in
each'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:222:in block (2 levels) in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:367:in
ensure_connected'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:221:in block in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:306:in
logging'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:220:in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:187:in
call_pipelined'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/inst/nosql/redis.rb:66:in block in call_pipelined_with_oneapm_trace' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/agent/datastore.rb:49:in
block in wrap'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/support/method_tracer.rb:67:in block in trace_execution_scoped' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/support/method_tracer/helpers.rb:83:in
trace_execution_scoped'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/support/method_tracer.rb:65:in trace_execution_scoped' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/agent/datastore.rb:46:in
wrap'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/inst/nosql/redis.rb:65:in call_pipelined_with_oneapm_trace' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:157:in
block in call_pipeline'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:293:in with_reconnect' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:155:in
call_pipeline'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:2304:in block in multi' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:58:in
block in synchronize'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:58:in synchronize' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:2296:in
multi'
/Users/ryang/work/my_code/lib/tasks/redis.rake:19:in block (2 levels) in <top (required)>' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/rake-12.1.0/exe/rake:27:in
<top (required)>'
/Users/ryang/.rvm/gems/ruby-2.3.0@my_code/bin/ruby_executable_hooks:15:in eval' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/bin/ruby_executable_hooks:15:in '<main>'Errno::EPIPE: Broken pipe <internal:prelude>:134:in
__write_nonblock'
internal:prelude:134:in write_nonblock'
I have set time_out, connect_timeout, write_timeout, but it doesn't work. And, sometimes, the thrown errors are like that :
Errno::EPROTOTYPE: Protocol wrong type for socket <internal:prelude>:134:in
__write_nonblock'
internal:prelude:134:in write_nonblock'
Does it work outside of a multi? Redis takes special care to make MULTI transactional. Do you have any errors in the redis server log?
The entire network code was replaced, so I think we can close this unless someone can reproduce in 5.0.