Remove failed jobs after a certain amount of time

[holgersindbaek]

I like how failed jobs are still in the queue, so I can debug those failed jobs. I have a good deal of them though, so it would be nice if the failed jobs deleted themselves after 48 hours or so. Is there any way to do this?

[katowulf]

Something like this?

var ONE_HOUR = 1000 * 60 * 60;
var FOURTY_EIGHT_HOURS = ONE_HOUR * 48;

setInterval(processFailedJobs, ONE_HOUR );

function processFailedJobs() {
  var ref = firebase.database().ref('queue/tasks');

  // Assumes all failed tasks will fit in memory.
  // Depending on resources, frequency of tasks,
  // and likelihood they could fail, may want to
  // decrease setInterval above or the threshold (i.e. 48 hours)
  // to smaller incerements
  ref.orderByChild('_state').equalTo('error_state').once('value', function(valueSnap) {
    valueSnap.forEach(function(ss) {
      var data = ss.val();
      // if we haven't visited this failed task before, mark it with a timestamp
      if( !data.timeMarked ) {
        ss.ref.child('timeMarked').set(firebase.database.ServerValue.TIMESTAMP);
      }
      // if the timestamp is older than 48 hours, then delete it
      else if( data.timeMarked > FOURTY_EIGHT_HOURS ) {
        ss.ref.remove();
      }
   }
 });
}

[holgersindbaek]

I was looking for something build in to the system, but I guess that would work. Thanks!

[katowulf]

Leaving this opened so engineers can review and decide if it's something appropriate to consider.

[cbraynor]

This is an interesting feature request, and in general scheduled operations come up a lot. They are, however, hard in a distributed system with only peers.

I'll leave this task around as a feature request