fluent-logger-python
fluent-logger-python copied to clipboard
fluent
using fluent-logger-python with eventlet
Hello,
I saw a surprising memory usage growth (see this issue). But when I remove monkey_patch looks like memory usage back to normal. Does fluent-logger-python compatible with eventlet?
I don't know about eventlet, but I have been using it with gevent in production for about 4 years now with no problems.
I saw fluent-logger uses threading.Local to store last errors, but does it make sense when it is running in an eventlet? I found, although it could be wrong observation, by removing threading.local, memory usage could be surprisingly reduced.
The question is how does eventlet monkey-patch the threading.local if at all? You'll also see that we're using threading.Lock.
If eventlet is leaking memory through threading.local because it didn't monkey it correctly or if you're leaking eventlets that are not being garbage collected and thus retain local state, then there is not much Fluent logger can do about it, sorry.
I understand, but I am still interested why last_error is stored and how can I take advantage of it. Because I am afraid this may not be easy to solve for eventlet, so if last_error is not used, I am thinking to fork the repo and maintaining a custom version...
The reason for a thread local last error is because the same sender is global, can be shared by multiple threads as it's stateless, except for last error. Furthermore, the same behavior in AsyncSender allows the last error to be attributed to the invoking thread. It's not the best design but it's been there for a long time. Judging by eventlet/eventlet#741, it has been confirmed to be an eventlet bug.
I am thinking to fork the repo and maintaining a custom version...
https://www.youtube.com/watch?v=BvK6KsLkPUs