Deleting entries in 'source' should delete in 'destination' as well

[kfricke-hacon]

I have stale images and albums in my destination directory, which have been deleted from the source directory. There is no reasonable workaround for purging these entries. An empty intermediate directory leads to always complete generation of the whole gallery. 'find'ing older entries using an cronjob is not feasible, because then all files and directories (even unchanged) must at least be touched while updating a gallery.

Markdown files (index.md) seem to get touched with each gallery build, but there is still no way to find "to-be-deleted" images in the destination directory.

I currently see no good way to tackle this problem. Any hints regarding this and I will try to implement a solution.

[saimn]

This is indeed an issue, but the only solution I see is to iterate on all images of the destination directory to see if they are still in the source directory. It can be time expensive, so until now I didn't consider this as a big issue: these stale images will just take some disk size. If you're motivated, we could add this as an optional step, after the list of images in the source dir has been built, and before the actual processing.

[kfricke-hacon]

Ok, understood. Can you give me a hint on how to implement this in a proper way? Would it really be sufficient to iterate over file names in the destination directory? Or should we try to generate a list of media objects based on the destination directory? That way we could compare the source and destination "media" directly. Without deeper knowledge of the gallery generation code I would assume that the later approach would make it possible to compare the lists of media in an existing step.

[saimn]

IMO the first aproach is sufficient, the second one will not help you. I would do this at the end of Gallery.__init__: https://github.com/saimn/sigal/blob/master/sigal/gallery.py#L509 It could even be a plugin as it may not be useful for everybody (at least not activated by default). Currently there no mean to check if an album contain an image with its path, it is something that must be added.

[saimn]

As discussed in #276 (cc @reagle) it would be useful in some cases to have this cleanup mode, it could be an optional feature activated with --clean or --purge flag.

[reagle]

I do delete, but it is a nuisance to remember which directory to delete (or accidentally delete the source directory) and it is very slow if there are videos to re-transcode.

[transplier]

Since it's been a few years, I'll add my voice in support of this feature. My workflow is to run Sigal and use aws s3 sync to place the gallery onto s3 for hosting. I often delete (or worse, rename) entire folders of the gallery and individual files. If I forget to manually clean up the output file, it's effectively there forever. I can't reasonably regenerate the gallery from scratch, as I have days worth of video in total. Even if I did regenerate everything, the upload to s3 would be prohibitive on my residential connection, and cost actual money (due to aws transit costs). I could try a terrible workaround where I configure ffmpeg to always generate only one frame of the video, instruct sigal to generate the gallery from scratch into a temporary directory, get a complete list of files from that temporary gallery as well as the real gallery, then delete any files which only appear in the real gallery. But this is a little bonkers, a --clean flag would be amazing.