Sensu Server Out of Memory

[Simiil]

After running for some time the Sensu Server crashes with a NoMemoryError. The server is dead until it is restarted. The error is:

/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.0/lib/sensu/json/oj.rb:25:in `dump': failed to allocate memory (NoMemoryError)
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.0/lib/sensu/json/oj.rb:25:in `dump'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.0/lib/sensu/json.rb:43:in `dump'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-logger-1.2.1/lib/sensu/logger/stream.rb:125:in `create_log_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-logger-1.2.1/lib/sensu/logger/stream.rb:49:in `add'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-logger-1.2.1/lib/sensu/logger/stream.rb:29:in `block (2 levels) in create_level_methods'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/filter.rb:207:in `filter_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/process.rb:264:in `block in process_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/process.rb:262:in `each'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/process.rb:262:in `process_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/process.rb:706:in `block (4 levels) in process_check_result'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/process.rb:520:in `block in update_event_registry'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.2.0/lib/sensu/redis/client.rb:337:in `dispatch_response'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.2.0/lib/sensu/redis/client.rb:365:in `parse_response_line'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.2.0/lib/sensu/redis/client.rb:387:in `receive_data'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run_machine'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/server/process.rb:33:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/exe/sensu-server:10:in `<top (required)>'
        from /opt/sensu/bin/sensu-server:22:in `load'
        from /opt/sensu/bin/sensu-server:22:in `<main>'

The Log level is on info, the embedded ruby is used. Logs are written correctly up to the crash. The machine that runs sensu-server has 2GB of RAM and normally doesn't come close to using that much

Environment:

Sensu-Server Version: 1.2 OS: Ubuntu Linux (trusty)

[cwjohnston]

Hi @Simiil, thanks for raising this issue. Because Ruby dynamically allocates memory as needed, out of memory conditions aren't something we see very often. If you will provide some of the following details, they might help us better understand the problem:

Is this condition reproducible in your environment?

Are you able to determine how much memory is in use by Sensu's Ruby process leading up to the crash?

Do you have any extensions installed in your Sensu instance?

[vbichov]

Hi, Similar issue:

{"timestamp":"2018-03-13T17:12:13.087162+0000","level":"error","message":"unexpected api error","error":"400 Bad request","backtrace":"/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/em-http-server-0.1.8/lib/em-http-server/server.rb:61:in `bad_parsing'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/em-http-server-0.1.8/lib/em-http-server/server.rb:48:in `parse_first_header'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/em-http-server-0.1.8/lib/em-http-server/server.rb:23:in `receive_request'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/em/protocols/header_and_content.rb:99:in `dispatch_request'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/em/protocols/header_and_content.rb:73:in `receive_line'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/em/protocols/linetext2.rb:73:in `receive_data'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run_machine'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/api/process.rb:17:in `run'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/exe/sensu-api:10:in `<top (required)>'\n/opt/sensu/bin/sensu-api:22:in `load'\n/opt/sensu/bin/sensu-api:22:in `<main>'"}
/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.0/lib/sensu/json/oj.rb:25:in `dump': failed to allocate memory (NoMemoryError)
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.0/lib/sensu/json/oj.rb:25:in `dump'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.0/lib/sensu/json.rb:43:in `dump'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/api/http_handler.rb:205:in `respond'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/api/routes/events.rb:24:in `block (3 levels) in get_events'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.2.0/lib/sensu/redis/client.rb:337:in `dispatch_response'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.2.0/lib/sensu/redis/client.rb:369:in `parse_response_line'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.2.0/lib/sensu/redis/client.rb:387:in `receive_data'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run_machine'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/lib/sensu/api/process.rb:17:in `run'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.2.0/exe/sensu-api:10:in `<top (required)>'
    from /opt/sensu/bin/sensu-api:22:in `load'
    from /opt/sensu/bin/sensu-api:22:in `<main>'

Both sensu-extensions-occurrences and sensu-extensions-check-dependencies are used

Non reproducible (at least for now)

[portertech]

This could be an Oj JSON parser C extension leak (doesn't use Ruby heap). We use an older release of Oj (https://github.com/ohler55/oj/blob/master/CHANGELOG.md#2181---2017-01-09) to avoid a known leak which has probably been addressed. We can try upgrading to the latest release (https://github.com/ohler55/oj/blob/master/CHANGELOG.md#360---2018-05-01).

As we have not yet seen this reproduced and upgrading Oj could introduce other problems, I'm adding this to the 1.5 milestone.

[robertely]

This just hit us as well. I was also unable to reproduce it, and can't run at a log level that would show what caused the parser to melt.

This didn't just kill a single sensu-server instance it nearly simultaneously killed every instance in the cluster. We noticed it pretty quickly because it was so dramatic but still pretty bad.

Stack with versions:

/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.1/lib/sensu/json/oj.rb:25:in `dump': failed to allocate memory (NoMemoryError)
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.1/lib/sensu/json/oj.rb:25:in `dump'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-json-2.1.1/lib/sensu/json.rb:43:in `dump'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-logger-1.2.2/lib/sensu/logger/stream.rb:125:in `create_log_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-logger-1.2.2/lib/sensu/logger/stream.rb:49:in `add'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-logger-1.2.2/lib/sensu/logger/stream.rb:29:in `block (2 levels) in create_level_methods'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/filter.rb:207:in `filter_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/process.rb:265:in `block in process_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/process.rb:263:in `each'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/process.rb:263:in `process_event'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/process.rb:711:in `block (4 levels) in process_check_result'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/process.rb:529:in `block in update_event_registry'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.3.0/lib/sensu/redis/client.rb:354:in `dispatch_response'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.3.0/lib/sensu/redis/client.rb:382:in `parse_response_line'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-redis-2.3.0/lib/sensu/redis/client.rb:404:in `receive_data'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run_machine'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:194:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/lib/sensu/server/process.rb:34:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.4.2/exe/sensu-server:10:in `<top (required)>'
        from /opt/sensu/bin/sensu-server:22:in `load'
        from /opt/sensu/bin/sensu-server:22:in `<main>'