[Twitter] Postprocessor to convert Twitter mp4 to gif using ImageMagick

[KonoVitoDa]

Is there a way to set it? It would be done running the following command with the downloaded mp4:

magick convert INPUT.mp4 -loop 0 OUTPUT.gif

I'm just looking for an easy and good way to download Twitter gifs as actual gifs, and not mp4. If you know a better option to do so, please tell me.

[Hrxn]

Well, there's the exec postprocessor

{
    "name"   : "exec",
    "async"  : false,
    "command": "...",
    "event"  : "after"
}

Just use exec.command like in the example you gave... So, what you describe here is actually possible, but the real question is: Why?

GIFs are horridly inefficient, every transformation is (usually) a loss.. I'd consider it a good thing that you get MP4 and would expect the usual conversions to be in the other direction?!

[KonoVitoDa]

I added this to the end of my config file:

    "postprocessor":
    {
        "mp4 to gif":
        {
            "name": "exec",
            "async"  : false,
            "command": "magick convert {} -loop 0 {}.gif",
            "event"  : "after",
            "filter": "duration==0"
        }
    }

And this to the extractor options:

        "twitter":
        {
            "postprocessors": ["mp4 to gif"]
        }

It almost gave me the desired result, except that the new filename became filename.mp4.gif. How could I make it not keep the old extension in the name? And how to delete mp4 after conversion?

GIFs are horridly inefficient, every transformation is (usually) a loss.. I'd consider it a good thing that you get MP4 and would expect the usual conversions to be in the other direction?!

So what do you suggest? Having a low resolution mp4 would be useless as I want the gifs to be placed and played along with other images. And despite its limited quality, gif is still the dominant format for posting and sharing animations with others, etc.

EDIT 1: If you read the previous message, ignore all the mess I made before. I already fixed it. I'm a programming noob, sorry.

EDIT 2: Magick also has this command to convert all files in the directory it is running from:

magick mogrify -format gif *.mp4

But since it needs to be run directly from the target directory, I first need a way to run "cd DIRECTORY". How to do this?

[KonoVitoDa]

It almost gave me the desired result, except that the new filename became filename.mp4.gif. How could I make it not keep the old extension in the name? And how to delete mp4 after conversion?

@Hrxn This would be possible with these Magick options:

magick convert {} -set filename:f '%d/%t' %[filename:f].gif

Where: %d = directory component of path %t = filename without directory or extension (suffix)

It works fine when I run it directly from the terminal, but when I place it in "command" in the config file, it results in this:

convert: unable to open image ''\\?\D:\Downloads\twitter/NAME'.gif': Invalid argument @ error/blob.c/OpenBlob/3527.
[postprocessor.exec][warning] 'magick convert "\\?\D:\Downloads\twitter\NAME.mp4" -set filename:f '%d/%t' %[filename:f].gif' returned with non-zero exit status (1)

This should be caused by the prsence of [ and ], some way to fix?

Reference: https://imagemagick.org/script/command-line-processing.php https://gist.github.com/fedir/9918685

[mikf]

It seems that Magick does not like the \\?\ prefix that gallery-dl uses for every Windows path. You can work around that by putting all command-line arguments in a list and using {_path[4:]} to strip the first for characters:

        {
            "name": "exec",
            "event": "after",
            "filter": "extension == 'mp4'",
            "command": ["magick", "convert", "{_path[4:]}", "-set", "filename:f", "%d/%t", "%[filename:f].gif"]
        }

This actually works and converts every .mp4 to a much, much larger .gif, but it does not delete the original file. Maybe there is an imagemagick option for that, but I couldn't find one.

Maybe use a separate script that does both conversion and deletion, and call that from gallery-dl? ("command": ["my_script.bat", "{_path[4:]}"])

[KonoVitoDa]

but it does not delete the original file. Maybe there is an imagemagick option for that, but I couldn't find one

It wouldn't be possible to use a native CMD/Powershell command to remove the input file?

This actually works and converts every .mp4 to a much, much larger .gif

Also, add 'duration == 0' to filter if you just want to convert GIFs, (afaik, gallery-dl recognizes Twitter GIFs as having duration of 0.0s)

[KonoVitoDa]

@mikf why is the duration metadata for Twitter videos/GIFs gone? It was the way I used to differentiate GIFs from normal videos. :(

[mikf]

It is? This must have happened during the transition from REST to GraphQL API in v1.20.2 (cb43f773), but this was unintended.

The GIFs from https://twitter.com/gifs still have a duration, so I don't know.

[KonoVitoDa]

The GIFs from https://twitter.com/gifs still have a duration, so I don't know.

For me it doesn't. And I noticed later that even the extension for videos/GIFs now is always "None".

[earthyrock]

So looking at the above and not being clear, I hand-jammed the above into my config. Tell me how to clean this up, would ya?

    "postprocessors": [
        {
            "name": "ugoira",
            "whitelist": ["pixiv", "danbooru"],
            "extension": "mp4",
            "keep-files": false,
            "ffmpeg-twopass": false,
            "ffmpeg-args": ["-hide_banner", "-loglevel", "quiet", "-c:v", "libx264", "-b:v", "2M", "-preset", "veryfast", "-pix_fmt", "yuv420p"],
            "ffmpeg-location": "G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe"
        },
{
    "gif to mp4":
    {
        "name": "exec",
        "async"  : false,
        "command": "ffmpeg -i {} {}.mp4 && del {}",
        "event"  : "after",
        "filter": "duration==0"
    }
},
        {
            "name": "metadata",
            "whitelist": ["danbooru", "yandere", "sankaku"],
            "mode": "tags"
        },
    ],





    "exhentai":
    {
        "directory": ["e-hentai", "{title}"],
        "username": "?",
        "password": "?",
        "postprocessors": ["gif to mp4"]
        "filename": "{num:>04}_{name}.{extension}"
    },

[Hrxn]

@earthyrock

{
    "postprocessor":
    {
        "gif to mp4":
        {
            "name": "exec",
            "async"  : false,
            "command": "ffmpeg -i {} {}.mp4 && del {}",
            "event"  : "after",
            "filter": "duration==0"
        }
    },

    "extractor":
    {

    "postprocessors": [
        {
            "name": "ugoira",
            "whitelist": ["pixiv", "danbooru"],
            "extension": "mp4",
            "keep-files": false,
            "ffmpeg-twopass": false,
            "ffmpeg-args": ["-hide_banner", "-loglevel", "quiet", "-c:v", "libx264", "-b:v", "2M", "-preset", "veryfast", "-pix_fmt", "yuv420p"],
            "ffmpeg-location": "G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe"
        },

        {
            "name": "metadata",
            "whitelist": ["danbooru", "yandere", "sankaku"],
            "mode": "tags"
        }
    ],

    "exhentai":
    {
        "directory": ["e-hentai", "{title}"],
        "username": "?",
        "password": "?",
        "postprocessors": ["gif to mp4"],
        "filename": "{num:>04}_{name}.{extension}"
    }

    }
}

[mikf]

( a small correction from my side: postprocessor needs to be at the top level, not inside extractor )

[Hrxn]

You're right, should've taken a look at my own config file beforehand..

[KonoVitoDa]

For me it doesn't. And I noticed later that even the extension for videos/GIFs now is always "None".

UPDATE: It seems to be working again. Maybe the problem was caused because I was still using the user-agent from the old config file (early 2021), I don't know.

[earthyrock]

See attached batch file and gallery-dl.conf. Copied and pasted the above Hxrn comment into my gallery-dl.conf, as listed; no modifications. Getting the following error when running batch file. Again, goal is GIF to MP4. Btw, it does download the 1st GIF file, no conversion, and then the error.

Gallery-dl.bat.txt

gallery-dl.conf.txt

[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.17763
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.9
[gallery-dl][debug] Starting DownloadJob for 'https://exhentai.org/g/1480774/26b6b4a48d/'
[exhentai][debug] Using ExhentaiGalleryExtractor for 'https://exhentai.org/g/1480774/26b6b4a48d/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): exhentai.org:443
[urllib3.connectionpool][debug] https://exhentai.org:443 "GET /g/1480774/26b6b4a48d/ HTTP/1.1" 200 4382
[exhentai][debug] Sleeping for 5.0 seconds
[urllib3.connectionpool][debug] https://exhentai.org:443 "GET /s/9e9c230ac4/1480774-1/1480774-1 HTTP/1.1" 200 1356
[exhentai][debug] Using download archive 'X:\12X\11Web\gallery-dl\E-Hentai\GDB.sqlite3'
[exhentai][debug] Active postprocessor modules: [ExecPP]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): shxpiyu.dylbjdelkrou.hath.network:443
[urllib3.connectionpool][debug] https://shxpiyu.dylbjdelkrou.hath.network:443 "GET /h/9e9c230ac41d826b29157fc25a1189225e9e0892-6030939-320-180-gif/keystamp=1657680600-4fa66bdc9f;fileindex=72876253;xres=org/Autopsy_Day10_01.gif HTTP/1.1" 200 6030939
[exhentai][error] An unexpected error occurred: NameError - name 'duration' is not defined. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[exhentai][debug]
Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 77, in run
  File "gallery_dl\job.pyc", line 121, in dispatch
  File "gallery_dl\job.pyc", line 273, in handle_url
  File "gallery_dl\job.pyc", line 496, in _call_hook
  File "<expr>", line 1, in <module>
NameError: name 'duration' is not defined
Press any key to continue . . .

[mikf]

@earthyrock replace line 10 in your config ("filter": "duration==0") with "filter": "extension == 'gif'". exh doesn't have a duration.

[earthyrock]

@mikf Modified with the above and transcoded great (Thanks mikf)!!!

Two issues came out of it, though:

1. Like @vitormathews, the transcoded filenames result in *.gif.mp4.

Quoted from the far above discussion, not sure what combination of the below should be put where (obviously not "magick"):

            "command": ["magick", "convert", "{_path[4:]}", "-set", "filename:f", "%d/%t", "%[filename:f].gif"]

2. Unless I have ffmpeg in the same folder as the gallery-dl.bat, I get the below error.

'ffmpeg'` is not recognized as an internal or external command,
operable program or batch file.
[postprocessor.exec][warning] 'ffmpeg -i "\\?\X:\12X\11Web\gallery-dl\e-hentai\[Motkeyz] Timestamps, Unconditional Love (Animation) [v1.0 PE]\0003_None.gif" "\\?\X:\12X\11Web\gallery-dl\e-hentai\[Motkeyz] Timestamps, Unconditional Love (Animation) [v1.0 PE]\0003_None.gif".mp4 && del "\\?\X:\12X\11Web\gallery-dl\e-hentai\[Motkeyz] Timestamps, Unconditional Love (Animation) [v1.0 PE]\0003_None.gif"' returned with non-zero exit status (1)

Is there a method to declare an ffmpeg path?

        "gif to mp4":
        {
            "name": "exec",
            "async"  : false,
            "command": "ffmpeg -i {} {}.mp4 && del {}",
            "event"  : "after",
            "filter": "extension == 'gif'"
        }

[Hrxn]

Yeah, just using a fully-qualified path should work, for example...

        "gif to mp4":
        {
            "name": "exec",
            "async"  : false,
            "command": "D:\\Path\\to\\my\\ffmpeg.exe -i {} {}.mp4 && del {}",
            "event"  : "after",
            "filter": "extension == 'gif'"
        }

[earthyrock]

"command": "G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe -i {} {}.mp4 && del {}",

In the above, due to spaces in the path name, it's seeing the command as G:\00OSz\12win10b

Can't use double quotes within double quotes.

"command": ""G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe" -i {} {}.mp4 && del {}",

Errors out.

I read thru the following trying to work around the spaces. https://www.howtogeek.com/694949/how-to-escape-spaces-in-file-paths-on-the-windows-command-line/

Can't use ^ before each space to escape the spaces.

"command": "G:\\00OSz\\12win10b^ zEnt-LTSC^ 1809^ x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe -i {} {}.mp4 && del {}",

I get the following error:

[NULL @ 00000278e43cd500] Unable to find a suitable output format for 'zEnt-LTSC'
zEnt-LTSC: Invalid argument
[postprocessor.exec][warning] 'G:\00OSz\12win10b^ zEnt-LTSC^ 1809^ x64\05Apps\Multimedia\Video\Installed\ffmpeg\bin\ffmpeg.exe -i "\\?\X:\12X\11Web\gallery-dl\e-hentai\[Motkeyz] Timestamps, Unconditional Love (Animation) [v1.0 PE]\0007_None.gif" "\\?\X:\12X\11Web\gallery-dl\e-hentai\[Motkeyz] Timestamps, Unconditional Love (Animation) [v1.0 PE]\0007_None.gif".mp4 && del "\\?\X:\12X\11Web\gallery-dl\e-hentai\[Motkeyz] Timestamps, Unconditional Love (Animation) [v1.0 PE]\0007_None.gif"' returned with non-zero exit status (1)

Ideas? I know the below can be in "postprocessors" but doesn't appear to work if I place it as well under "postprocessor". Errors out.

"ffmpeg-location": "G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe"

    "postprocessor":
    {
        "gif to mp4":
        {
            "name": "exec",
            "async"  : false,
            "ffmpeg-location": "G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe"
            "command": "ffmpeg.exe -i {} {}.mp4 && del {}",
            "event"  : "after",
            "filter": "extension == 'gif'"
        }
    },

    "extractor":
    {

    "postprocessors": [
        {
            "name": "ugoira",
            "whitelist": ["pixiv", "danbooru"],
            "extension": "mp4",
            "keep-files": false,
            "ffmpeg-twopass": false,
            "ffmpeg-args": ["-hide_banner", "-loglevel", "quiet", "-c:v", "libx264", "-b:v", "2M", "-preset", "veryfast", "-pix_fmt", "yuv420p"],
            "ffmpeg-location": "G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe"
        },

[Hrxn]

Yeah, two options..

1. Put the ffmpeg.exe binary file into a path without any spaces
2. To escape double quotes in a JSON string, use the JSON escape character \ directly in front of the inner double quotes. i.e. "command": "\"G:\\00OSz\\12win10b zEnt-LTSC 1809 x64\\05Apps\\Multimedia\\Video\\Installed\\ffmpeg\\bin\\ffmpeg.exe\" -i {} {}.mp4 && del {}"

[earthyrock]

WOOT!!! That's the money!! I used the JSON escape character. Thank you so much @Hrxn

@mikf Any idea now how to correct previous issue 1. Like @vitormathews, the transcoded filenames result in *.gif.mp4.

gallery-dl.conf.txt Gallery-dl.bat.txt