out_es: update off_prev using log decoder

[nokute78]

off of out_es is not updated from v2.1.0 and it will log [debug] [out_es] converted_size is 0. e.g. https://github.com/fluent/fluent-bit/issues/7942 This patch is to update off.

off is updated by msgpack_unpack_next in v2.0.14. https://github.com/fluent/fluent-bit/blob/v2.0.14/plugins/out_es/es.c#L375

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [X] Debug log output from testing the change

• [X] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [N/A] Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

Configuration

ruby -e "print 'a' * 10000" > a.log
ruby -e 'print "\n"' >> a.log
ruby -e "print 'b' * 10000" >> a.log

[INPUT]
    Name tail
    Path a.log
    Read_From_Head on

[OUTPUT]
    name  es
    match *

Debug/Valgrind output

$ valgrind --leak-check=full bin/fluent-bit -c a.conf  -v
==40718== Memcheck, a memory error detector
==40718== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==40718== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==40718== Command: bin/fluent-bit -c a.conf -v
==40718== 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/24 07:53:12] [ info] Configuration:
[2023/09/24 07:53:12] [ info]  flush time     | 1.000000 seconds
[2023/09/24 07:53:12] [ info]  grace          | 5 seconds
[2023/09/24 07:53:12] [ info]  daemon         | 0
[2023/09/24 07:53:12] [ info] ___________
[2023/09/24 07:53:12] [ info]  inputs:
[2023/09/24 07:53:12] [ info]      tail
[2023/09/24 07:53:12] [ info] ___________
[2023/09/24 07:53:12] [ info]  filters:
[2023/09/24 07:53:12] [ info] ___________
[2023/09/24 07:53:12] [ info]  outputs:
[2023/09/24 07:53:12] [ info]      es.0
[2023/09/24 07:53:12] [ info] ___________
[2023/09/24 07:53:12] [ info]  collectors:
[2023/09/24 07:53:13] [ info] [fluent bit] version=2.1.10, commit=2631ca199e, pid=40718
[2023/09/24 07:53:13] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/09/24 07:53:13] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/24 07:53:13] [ info] [cmetrics] version=0.6.3
[2023/09/24 07:53:13] [ info] [ctraces ] version=0.3.1
[2023/09/24 07:53:13] [ info] [input:tail:tail.0] initializing
[2023/09/24 07:53:13] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/09/24 07:53:13] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/09/24 07:53:13] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/09/24 07:53:13] [debug] [input:tail:tail.0] inotify watch fd=27
[2023/09/24 07:53:13] [debug] [input:tail:tail.0] scanning path a.log
[2023/09/24 07:53:13] [debug] [input:tail:tail.0] inode=4228864 with offset=0 appended as a.log
[2023/09/24 07:53:13] [debug] [input:tail:tail.0] scan_glob add(): a.log, inode 4228864
[2023/09/24 07:53:13] [debug] [input:tail:tail.0] 1 new files found on path 'a.log'
[2023/09/24 07:53:13] [debug] [es:es.0] created event channels: read=29 write=30
[2023/09/24 07:53:13] [debug] [output:es:es.0] host=127.0.0.1 port=9200 uri=/_bulk index=fluent-bit type=_doc
[2023/09/24 07:53:14] [ info] [output:es:es.0] worker #1 started
[2023/09/24 07:53:14] [ info] [output:es:es.0] worker #0 started
[2023/09/24 07:53:14] [ info] [sp] stream processor started
[2023/09/24 07:53:14] [debug] [input chunk] update output instances with new chunk size diff=20074, records=2, input=tail.0
[2023/09/24 07:53:14] [debug] [input:tail:tail.0] [static files] processed 19.5K
[2023/09/24 07:53:14] [debug] [input:tail:tail.0] inode=4228864 file=a.log promote to TAIL_EVENT
[2023/09/24 07:53:14] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4228864 watch_fd=1 name=a.log
[2023/09/24 07:53:14] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/09/24 07:53:14] [debug] [task] created task=0x54f8440 id=0 OK
[2023/09/24 07:53:14] [debug] [output:es:es.0] task_id=0 assigned to thread #0
[2023/09/24 07:53:15] [debug] [upstream] KA connection #59 to 127.0.0.1:9200 is connected
[2023/09/24 07:53:15] [debug] [http_client] not using http_proxy for header
[2023/09/24 07:53:15] [debug] [output:es:es.0] HTTP Status=200 URI=/_bulk
[2023/09/24 07:53:15] [debug] [output:es:es.0] Elasticsearch response
{"errors":false,"items":[{"create":{"status":201,"result":"created"}},{"create":{"status":201,"result":"created"}}]}
[2023/09/24 07:53:15] [debug] [upstream] KA connection #59 to 127.0.0.1:9200 is now available
[2023/09/24 07:53:15] [debug] [out flush] cb_destroy coro_id=0
[2023/09/24 07:53:15] [debug] [task] destroy task=0x54f8440 (task_id=0)
^C[2023/09/24 07:53:33] [engine] caught signal (SIGINT)
[2023/09/24 07:53:33] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/24 07:53:33] [ info] [input] pausing tail.0
[2023/09/24 07:53:33] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/24 07:53:33] [ info] [input] pausing tail.0
[2023/09/24 07:53:33] [ info] [output:es:es.0] thread worker #0 stopping...
[2023/09/24 07:53:33] [ info] [output:es:es.0] thread worker #0 stopped
[2023/09/24 07:53:34] [ info] [output:es:es.0] thread worker #1 stopping...
[2023/09/24 07:53:34] [ info] [output:es:es.0] thread worker #1 stopped
[2023/09/24 07:53:34] [debug] [input:tail:tail.0] inode=4228864 removing file name a.log
[2023/09/24 07:53:34] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=4228864 watch_fd=1
==40718== 
==40718== HEAP SUMMARY:
==40718==     in use at exit: 0 bytes in 0 blocks
==40718==   total heap usage: 2,015 allocs, 2,015 frees, 1,619,029 bytes allocated
==40718== 
==40718== All heap blocks were freed -- no leaks are possible
==40718== 
==40718== For lists of detected and suppressed errors, rerun with: -s
==40718== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[nokute78]

I merged current master into this PR branch.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[nokute78]

Ping

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.