logdna-agent on Ubuntu 20.04 stops sending logs

[0xCiBeR]

logdna-agent on Ubuntu 20.04 stops sending logs randomly after some time(can be a couple of hours or a day)

Steps to reproduce:

• install logdna-agent from APT on Ubuntu 20.04
• point log dir to some directory
• generate a couple of log files with a lot of lines and new names every couple of minutes

[smusali]

@ciber96, can you please tell us which version of the agent you are using? We have recently released LogDNA Agent v2.2.1 today.

Thanks!

[0xCiBeR]

# logdna-agent --version
2.2.1

[ModdyLP]

Can Confirm on Ubuntu 18.04 same agent Version

systemctl status logdna-agent.service
● logdna-agent.service - LSB: Starts LogDNA Agent at boot
   Loaded: loaded (/etc/init.d/logdna-agent; generated)
   Active: active (exited) since Sat 2021-03-13 19:19:10 CET; 2 days ago
     Docs: man:systemd-sysv-generator(8)

Mar 13 19:19:10 *.*.de systemd[1]: Stopped LSB: Starts LogDNA Agent at boot.
Mar 13 19:19:10 *.*.de systemd[1]: Starting LSB: Starts LogDNA Agent at boot...
Mar 13 19:19:10 *.*.de logdna-agent[6082]: Starting logdna-agent...
Mar 13 19:19:10 *.*.de systemd[1]: Started LSB: Starts LogDNA Agent at boot.

[lonacluka]

Our agent also crashes very frequently with Java Heap OOM errors.

Ubuntu 18.04 version 2.2.1

Any other logs I can provide to help tshoot?

2021-11-09 07:46:39 -0600 [info] streaming /var/log: 81 file(s)

<--- Last few GCs --->

[55900:0x3a4fdd0] 20555768 ms: Mark-sweep 1917.5 (2087.4) -> 1917.5 (2056.2) MB, 595.9 / 0.0 ms  (average mu = 0.529, current mu = 0.000) last resort GC in old space requested
[55900:0x3a4fdd0] 20556304 ms: Mark-sweep 1917.5 (2056.2) -> 1917.5 (2055.7) MB, 536.3 / 0.0 ms  (average mu = 0.372, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x20ad149408d1 <JSObject>
    0: builtin exit frame: byteLength(aka byteLengthUtf8)(this=0x176391599989 <Object map = 0x372a8d4650a9>,0x01244fcbadc9 <Very long string[1969751]>,0x176391599989 <Object map = 0x372a8d4650a9>)

    1: fromString(aka fromString) [0x176391598341] [buffer.js:~438] [pc=0x1b4085a7d99a](this=0x21497f2004b1 <undefined>,0x01244fcbadc9 <Very long string[1969751]>,0x2b6a614e8201 <String[#4]: utf8>)
  ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0xa37f40 node::Abort() [logdna-agent]
 2: 0xa38381 node::OnFatalError(char const*, char const*) [logdna-agent]
 3: 0xbb4a4e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [logdna-agent]
 4: 0xbb4ca2 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [logdna-agent]
 5: 0xd73935  [logdna-agent]
 6: 0xd851c9 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [logdna-agent]
 7: 0xd51acc v8::internal::factory::NewRawOneByteString(int, v8::internal::AllocationType) [logdna-agent]
 8: 0xf9b166 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [logdna-agent]
 9: 0xbb3f8b v8::internal::String::Flatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, v8::internal::AllocationType) [logdna-agent]
10: 0xbc8c88 v8::String::Utf8Length(v8::Isolate*) const [logdna-agent]
11: 0xa0fef1  [logdna-agent]
12: 0xc26e62  [logdna-agent]
13: 0xc28c54 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [logdna-agent]
14: 0x141ae99  [logdna-agent]