Redis::ConnectionError: Connection lost (EPIPE) when write large data to sorted set in a transaction

[gokusleepy2]

$redis.multi do |multi| multi.zadd 'my_key', my_arr end

'my_arr' contains about 1 million records. When reduce 'my_arr' to 0.5 million records, it worked well. All records can be zadded successfully one by one, all stored data costs 130M memory.

The backtrace is as following:

rake aborted! Redis::ConnectionError: Connection lost (EPIPE) /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:257:in rescue in io' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:250:in io' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:269:in write' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:228:in block (3 levels) in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:222:in each' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:222:in block (2 levels) in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:367:in ensure_connected' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:221:in block in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:306:in logging' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:220:in process' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:187:in call_pipelined' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/inst/nosql/redis.rb:66:in block in call_pipelined_with_oneapm_trace' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/agent/datastore.rb:49:in block in wrap' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/support/method_tracer.rb:67:in block in trace_execution_scoped' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/support/method_tracer/helpers.rb:83:in trace_execution_scoped' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/support/method_tracer.rb:65:in trace_execution_scoped' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/agent/datastore.rb:46:in wrap' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/oneapm_rpm-1.4.2/lib/one_apm/inst/nosql/redis.rb:65:in call_pipelined_with_oneapm_trace' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:157:in block in call_pipeline' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:293:in with_reconnect' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis/client.rb:155:in call_pipeline' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:2304:in block in multi' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:58:in block in synchronize' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:58:in synchronize' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/redis-3.3.3/lib/redis.rb:2296:in multi' /Users/ryang/work/my_code/lib/tasks/redis.rake:19:in block (2 levels) in <top (required)>' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/gems/rake-12.1.0/exe/rake:27:in <top (required)>' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/bin/ruby_executable_hooks:15:in eval' /Users/ryang/.rvm/gems/ruby-2.3.0@my_code/bin/ruby_executable_hooks:15:in '<main>'Errno::EPIPE: Broken pipe <internal:prelude>:134:in __write_nonblock' internal:prelude:134:in write_nonblock'

I have set time_out, connect_timeout, write_timeout, but it doesn't work. And, sometimes, the thrown errors are like that :

Errno::EPROTOTYPE: Protocol wrong type for socket <internal:prelude>:134:in __write_nonblock' internal:prelude:134:in write_nonblock'

[mperham]

Does it work outside of a multi? Redis takes special care to make MULTI transactional. Do you have any errors in the redis server log?

[byroot]

The entire network code was replaced, so I think we can close this unless someone can reproduce in 5.0.