Resolv::DNS::DecodeError: limit exceeded

[anthonator]

Has anybody seen this before? I thought this might be a rate limiting issue but I'm not having issues with the web client. I also deleted my Slack app and recreated it but I'm still having this issue.

Resolv::DNS::DecodeError: limit exceeded
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1583:in `get_string'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1631:in `get_label'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1625:in `get_labels'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1598:in `get_name'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1641:in `get_rr'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1515:in `block (2 levels) in decode'
    from /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/core_ext/range/each.rb:5:in `each'
    from /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/core_ext/range/each.rb:5:in `each'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1514:in `block in decode'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1531:in `initialize'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1495:in `new'
    from /usr/local/lib/ruby/2.3.0/resolv.rb:1495:in `decode'
    from /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/dns_resolver.rb:46:in `resolve'
    from /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_socket.rb:101:in `create_socket'
    from /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_socket.rb:53:in `initialize'
    from /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/socket.rb:39:in `new'
... 37 levels...
    from /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/dependencies.rb:296:in `load'
    from /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/dependencies.rb:296:in `block in load'
    from /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/dependencies.rb:268:in `load_dependency'
    from /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/dependencies.rb:296:in `load'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/commands/rails.rb:6:in `call'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/command_wrapper.rb:38:in `call'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application.rb:185:in `block in serve'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application.rb:156:in `fork'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application.rb:156:in `serve'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application.rb:131:in `block in run'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application.rb:125:in `loop'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application.rb:125:in `run'
    from /usr/src/bundle/gems/spring-1.6.4/lib/spring/application/boot.rb:18:in `<top (required)>'
    from /usr/local/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/local/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from -e:1:in `<main>'irb(main):046:0> E, [2016-02-29T15:33:05.306451 #29] ERROR -- : Actor crashed!
Resolv::DNS::DecodeError: limit exceeded
    /usr/local/lib/ruby/2.3.0/resolv.rb:1583:in `get_string'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1631:in `get_label'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1625:in `get_labels'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1598:in `get_name'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1641:in `get_rr'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1515:in `block (2 levels) in decode'
    /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/core_ext/range/each.rb:5:in `each'
    /usr/src/bundle/gems/activesupport-5.0.0.beta3/lib/active_support/core_ext/range/each.rb:5:in `each'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1514:in `block in decode'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1531:in `initialize'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1495:in `new'
    /usr/local/lib/ruby/2.3.0/resolv.rb:1495:in `decode'
    /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/dns_resolver.rb:46:in `resolve'
    /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_socket.rb:101:in `create_socket'
    /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_socket.rb:53:in `initialize'
    /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/socket.rb:39:in `new'
    /usr/src/bundle/gems/celluloid-io-0.17.3/lib/celluloid/io/socket.rb:39:in `new'
    /usr/src/bundle/gems/slack-ruby-client-0.6.0/lib/slack/real_time/concurrency/celluloid.rb:72:in `build_socket'
    /usr/src/bundle/gems/slack-ruby-client-0.6.0/lib/slack/real_time/concurrency/celluloid.rb:86:in `connect'
    /usr/src/bundle/gems/slack-ruby-client-0.6.0/lib/slack/real_time/socket.rb:26:in `connect!'
    /usr/src/bundle/gems/slack-ruby-client-0.6.0/lib/slack/real_time/concurrency/celluloid.rb:30:in `connect!'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /usr/src/bundle/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'

[dblock]

Interesting. This happens every time?

You can try to switch to EventMachine as a workaround, but of course it's not a "solution".

[anthonator]

Yeah, it's consistent.

If I get time I'll try switching over.

[dblock]

Can you post your code somewhere?

[skull-squadron]

It's likely an infrastructure configuration issue related to EDNS0 and legacy DNS where the answer QDCOUNT > 1 || packet payload > 512 bytes, and the resulting payload is incomplete. Some old packet sanitizers / firewalls also truncate or drop DNS packets > 512 bytes and prevent EDNS0 from working properly. I'm considering put together a patch for resolv.rb which $DEBUGs malformed packets for production troubleshooting.