elastic
Improve logging around 'queue is full' error
This was first considered with https://github.com/elastic/apm-server/issues/990. At the time, it was closed, but it might be worth revisiting now that some time has passed.
The 'queue is full' error can indicate a number of different issues. It would be very helpful if we could improve the logging around the 'queue is full' event to include some more information, such as what is being returned from Elasticsearch or Logstash, and what the current state of the queue itself is. It would also be helpful to allow an option to log the current state of the queue and how many events it contains, even if it is a debug level setting. Exposing it so metricbeat can collect queue statistics would be even better.
improve the logging around the 'queue is full' event to include some more information, such as what is being returned from Elasticsearch or Logstash
Only the logstash and kafka output are still based on libbeat output, and can therefore experience queue is full. Since 8.0, the elasticsearch output is no longer based on this queue based libbeat output implementation, but uses an output that does not require manual user configuration and tuning.
Given the limited scope of where the libbeat based output is still used, I don't think we'll be able to prioritize this request any time soon. The request around logging number of events from the queue would require substantial changes in libbeat, which is outside of the apm-server team's ownership.
