Data Loss When Logger is Destroyed

[fastbike]

We have some data being logged via the new database log appender. Obviously this is not as quick as writing entries to a log file so items can build up in the queue.

If the Logger is destroyed while there are still LogItems in the queue, they will be discarded thanks to this code

procedure TThreadSafeQueue<T>.DoShutDown;
begin
  fShutDown := True;
  fCriticalSection.Enter;
  try
    while fQueue.Count > 0 do
    begin
      fQueue.Extract.Free;
    end;
  finally
    fCriticalSection.Leave;
  end;
end

Many of our applications are deployed under IIS which periodically recycles the application pool - effectively terminating the instance of the app while spooling up a new instance. If there is anything in the queue we lose it. We could put a delay in the main thread to allow the queue some time to continue writing, however we have no idea of the size of the delay. If there are no items there is no need for a delay. If there are multiple items a short delay will be insufficient.

There is no ability to find out how many items are in the queue at any stage, not any way to persist the contents of the queue to disk when terminating and reload when the application restarts.

I'm looking for some guidance/discussion on possible solutions.

[danieleteti]

This is a very long standing discussions about log systems: "If I shutdown the application while the log system is dequeueing, should I discard the items in the queue and shutdown or should force the user to wait untile the queue is empty?". You case is very clear. Usually having an application "waits" to empty the queue is not nice, but lost the log item isn't as well.