Add support for Workers

[kishaningithub]

Fluent bit added support for workers in v.1.7.0

https://fluentbit.io/announcements/v1.7.0/

It would be really valuble go plugins can also have worker support

Impact

Because of the single threaded nature if one output plugin is "stuck", then the subsequent outputs / pipelines are not getting executed https://github.com/fluent/fluent-bit/issues/3088

I believe workers can solve this problem. Basically setting workers to 1 should ensure there is a separate thread for processing

[kishaningithub]

From what i observed setting

Workers 1

did not spawn a new thread in the engine for data delivery.

IMO, full workers need not be supported as go lang has goroutines. But what is needed in an ability for the go plugins to run in a separate delivery thread.

[nokute78]

I merged the patch https://github.com/fluent/fluent-bit/pull/3908 @gautampunhani Thank you.

[l2dy]

Should we add the RWMutex mentioned in https://github.com/fluent/fluent-bit-go/commit/b191f58f228c29d1df67ee75ecbee7eb413854ef to protect the contexts map?

[nokute78]

I think it is better to add RWMutex for prevention.

In my understanding, a lock is taken care on C side. https://fluentbit.io/announcements/v1.7.0/

The multithread implementation is lock-free at runtime.

On the other hand, the API FLBPluginSetContext and FLBPluginGetContext are for golang side API. We can't know when they are called.

[l2dy]

I think "lock-free" means that there is no lock on the C side, so contexts in Go must be protected with a mutex or sync.Map.