meteor icon indicating copy to clipboard operation
meteor copied to clipboard

fix: shutdown of connection to sink before publishing last record

Open GrayFlash opened this issue 3 years ago • 0 comments

Describe the bug The last stream of records is not published in the sink if we use the Close() function in the sink to close the connection to the sink. In agent.go the sinks are closed when the stream closes, but the stream closes once all records are emitted from extractor, hence while publishing the last record we get an error that the connection has been closed. This error has not caused any issue with sinks yet, because most sinks do not use the Close() in the sink for anything.

To Reproduce Steps to reproduce the behavior:

  1. Go to file sink
  2. In close function, uncomment the return s.File.Close()
  3. Run a recipe with sink as File Sink.

Expected behavior With proper config in the recipe, all records must be published in the sink before Close() is called.

Screenshots

Screenshot 2022-06-24 at 6 54 43 PM

Additional context The file sink was allowed for now, because not closing the File was fine and golang's garbage collector anyway closes it after the job ends. But it will be great to rethink how we close the stream and sinks altogether in the agent to make Close() for sink useful.

GrayFlash avatar Jun 24 '22 13:06 GrayFlash