FileQueue exploding hard disk in a video streaming application

[Ali-Parandeh]

I'm using this filequeue in a video streaming application that puts frames of a video onto the queue and then process them real-time. The issue is the filequeue is not getting rid of tmp files as they are taken from the queue.

How can I cleanup queue file on disk straight after getting my item from filequeue?

[GP89]

Hey Ali!

I think only newer filesystems support truncating the start of a file, which is what would need to happen to clean up after itself. The issue is new items are written to the end of the file, and old items are read from the start of the file, then as items are read off, the start of the file would need to be truncated each time which I'm not aware how to do.

The LifoFileQueue does truncate, as it's always taking from the end of the file. But that might not be useful to you if you need the items in a fifo order.

Another option might be to dispose the queue (and delete the temp file) when it's empty, and create a new one to clean up that way. Although I admit that's not very pretty. If your queue doesn't empty often, you could either block puts to allow it to empty, so you can clean up and then continue puts. Or better create a new second FileQueue, with new puts going into that while you wait for the first queue to empty, then you can clean it up and swap gets over the to new second queue after the first one is cleaned up