[Facebook] Downloading Without Specifying Video Format (ie. Default Best Quality) results in Merged MKV Output

[hairycactus]

Checklist:

• [x] I'm reporting a bug
• [x] I've tested the following youtube-dlc x64 versions on Win 10 x64:
  • v2020.11.11-3 -- Latest available version, file properties state v2020.11.11.4
  • v2020.11.10 -- No binaries supplied
  • v2020.11.07
• [x] I've checked that all provided URLs are alive and playable in a browser
• [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [x] I've searched the bugtracker for similar bug reports including closed ones
• [x] I've read bugs section in FAQ

Expected Behaviour:

> youtube-dlc https://www.facebook.com/<PageName or PageID>/videos/<IDString>

If no parameter options & video format are specified (as above), Youtube-dlc is expected to default to downloading the best-available format.

From past experience, this is typically HD MP4 for Facebook "non-livestream" videos, & SD MP4 for Facebook "livestream" videos -- as listed by the "-F" parameter.

Observed Behaviour:

Unlike in the past, Youtube-dlc now downloads the audio & video streams separately, before merging them into MKV video instead, with the warning "Requested formats are incompatible for merge and will be merged into mkv."

Impact: The process is slower than the previous default of directly downloading the (best) HD MP4 video, & the output MKV video is of a poorer quality (albeit smaller filesize) than the previous default of HD MP4 format.

In order to get the (best) HD MP4 video, one now has to explicitly specify -f dash_hd_src.

Verbose Log:

Listing Available Video Formats

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-v', 'https://www.facebook.com/228735667216/videos/2464080690554475']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.11-3
[debug] Python version 3.8.6 (CPython) - Windows-10-10.0.18362-SP0
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1
[debug] Proxy map: {}

[facebook] 2464080690554475: Downloading webpage

[info] Available formats for 2464080690554475:
format code               extension  resolution note
2042113065921699a         m4a        audio only [eng] DASH audio   65k , m4a_dash container, mp4a.40.5 (48000Hz)
392682168593861v          webm       240x426    DASH video   95k , webm_dash container, vp9, video only
303462437382642v          webm       240x426    DASH video  180k , webm_dash container, vp9, video only
381371343103379v          webm       360x640    DASH video  356k , webm_dash container, vp9, video only
748609266053074v          webm       480x854    DASH video  635k , webm_dash container, vp9, video only
834417070728744v          webm       720x1280   DASH video 1328k , webm_dash container, vp9, video only
dash_sd_src               mp4        unknown
dash_sd_src_no_ratelimit  mp4        unknown
dash_hd_src               mp4        unknown    (best)

Downloading Best Video (No Format Specified)

[debug] Command-line args: ['-v', 'https://www.facebook.com/228735667216/videos/2464080690554475']

[facebook] 2464080690554475: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.

[debug] Invoking downloader on 'https://video.fsin9-2.fna.fbcdn.net/v/t39.25447-2/10000000_3272004002928574_2467381211617686634_n.webm?_nc_cat=1&ccb=2&_nc_sid=5aebc0&efg=eyJ2ZW5jb2RlX3RhZyI6ImRhc2hfdnA5XzVzZWNnb3BfbWlucmVzX2hhbG9fMTUwMGtfZnJhZ18yX3ZpZGVvIn0=&_nc_ohc=LtrxVf_WIC8AX8nfwBT&_nc_ht=video.fsin9-2.fna&oh=7b20ce30cc3c0f425c7e184836c18525&oe=5FD78D22'

[download] Destination: 'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.f834417070728744v.webm
[download] 100% of 43.41MiB in 00:02

[debug] Invoking downloader on 'https://video.fsin9-2.fna.fbcdn.net/v/t42.1790-2/125487720_2042113075921698_205332372020228618_n.mp4?_nc_cat=1&ccb=2&_nc_sid=5aebc0&efg=eyJ2ZW5jb2RlX3RhZyI6ImRhc2hfYXVkaW9fYWFjcF82NF9mcmFnXzJfYXVkaW8ifQ==&_nc_ohc=DbP1m5gxHu4AX9FWEEG&_nc_ht=video.fsin9-2.fna&oh=f95cd7d58e1cc56e50acd8db4d2a93c5&oe=5FB35FEA'

[download] Destination: 'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.f2042113065921699a.m4a
[download] 100% of 2.14MiB in 00:02

[ffmpeg] Merging formats into "'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.mkv"

[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.f834417070728744v.webm" -i "file:'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.f2042113065921699a.m4a" -c copy -map "0:v:0" -map "1:a:0" "file:'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.temp.mkv"

Deleting original file 'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.f834417070728744v.webm (pass -k to keep)

Downloading Best Video (Best Format: dash_hd_src)

[debug] Command-line args: ['-f', 'dash_hd_src', '-v', 'https://www.facebook.com/228735667216/videos/2464080690554475']

[facebook] 2464080690554475: Downloading webpage

[debug] Invoking downloader on 'https://scontent.fsin9-2.fna.fbcdn.net/v/t66.36240-6/10000000_403623501020206_4116103325859823556_n.mp4?_nc_cat=109&ccb=2&_nc_sid=985c63&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9oZCJ9&_nc_ohc=YiJIah-Q4sUAX9l-yau&_nc_ht=scontent.fsin9-2.fna&oh=4f15ac73c022b874e2f56d25dd8ad278&oe=5FD79E8D'

[download] Destination: 'Outsiders have come in and cleared them, it's a deep wound.'  A Korean palm ...-2464080690554475.mp4
[download] 100% of 57.45MiB in 00:03

[pukkandan]

This is the expected behaviour. Are you sure this isn't how it used to work? To download the best combined format, you can use -f best

Also check out #23 which allows automatically selecting the combined format if it is better than the separate formats.

[hairycactus]

This is the expected behaviour. Are you sure this isn't how it used to work?

Yes, I had done this numerous times in the past, so I am very sure that not specifying any format for Facebook.com (or any domain), Youtube-dlc defaults to the "(best)" quality as listed by -F parameter.

And for Facebook.com, the "(best)" format as listed has always been MP4 (be it HD or SD).

The behaviour of Youtube-dlc merging audio-only & video-only streams into MKV output only started happening for Facebook videos uploaded within the last 1-2 weeks or so.

Other than the obviously different output video extension, the other main reason I noticed the above is because the process is noticeably longer, especially if the video is large & the network slow.

Once in a while, I do specify -f "bestvideo[height<=400]+bestaudio/best[height<=400]" for certain video-streaming sites. However, I try to avoid such sites because even though I deliberately opt for a lower-resolution video (<=400p), the download & merging process can take 20-30 minutes for a mere 20 MB MP4 video output.

Sample Older Facebook Videos

Youtube-dlc automatically & quickly downloads both of the below videos as "(best)" MP4 without user having to specify any format quality -f.

In older Facebook videos, it appears that Youtube-dlc detects the video streams as MP4, instead of WebM ?

▶ 14 Jun 2016: https://www.facebook.com/alicespringsreptilecentre/videos/1225923104085501

[debug] Command-line args: ['-F', '-v', 'https://www.facebook.com/alicespringsreptilecentre/videos/1225923104085501']

[facebook] 1225923104085501: Downloading webpage

[info] Available formats for 1225923104085501:
format code               extension  resolution note
1225924957418649a         m4a        audio only DASH audio   49k , m4a_dash container, mp4a.40.5 (44100Hz)
1225924907418654v         mp4        256x142    DASH video  125k , mp4_dash container, avc1.4d401e, video only
1225924937418651v         mp4        380x210    DASH video  139k , mp4_dash container, avc1.42c01e, video only
1225924934085318v         mp4        426x236    DASH video  323k , mp4_dash container, avc1.4d401e, video only
1225924954085316v         mp4        576x320    DASH video  516k , mp4_dash container, avc1.4d401e, video only
dash_sd_src               mp4        unknown
dash_sd_src_no_ratelimit  mp4        unknown
dash_hd_src               mp4        unknown
dash_hd_src_no_ratelimit  mp4        unknown    (best)

▶ 07 Aug 2019: https://www.facebook.com/ahh.pp/videos/352741199009678

[debug] Command-line args: ['-F', '-v', 'https://www.facebook.com/ahh.pp/videos/352741199009678']

[facebook] 352741199009678: Downloading webpage

[info] Available formats for 352741199009678:
format code               extension  resolution note
643425262828131a          m4a        audio only DASH audio   65k , m4a_dash container, mp4a.40.5 (44100Hz)
380034553178711v          mp4        256x144    DASH video   41k , mp4_dash container, avc1.42C01E, 30fps, video only
347595503171344v          mp4        426x240    DASH video   91k , mp4_dash container, avc1.42C01E, 30fps, video only
373968017075651v          mp4        640x360    DASH video  182k , mp4_dash container, avc1.4D401E, 30fps, video only
664014814504981v          mp4        960x540    DASH video  712k , mp4_dash container, avc1.4D401F, 30fps, video only
dash_sd_src               mp4        unknown
dash_sd_src_no_ratelimit  mp4        unknown
dash_hd_src               mp4        unknown    (best)

[pukkandan]

Yes, I had done this numerous times in the past, so I am very sure that not specifying any format for Facebook.com (or any domain), Youtube-dlc defaults to the "(best)" quality as listed by -F parameter.

youtube-dlc defaults to bestvideo+bestaudio/best, not best Perhaps you didnt have ffmpeg back then?

[M1ndo]

This is because because it merges audio/video together into one file (mkv) , when specifying a format id in your case (dash_hd_src) it's already mp4 with audio and there's no need to download audio separately and merge it into the mp4 .

[hairycactus]

youtube-dlc defaults to bestvideo+bestaudio/best, not best

I'd used every youtube-dlc version ever since its first release (as forked from youtube-dl). The default (ie. no format quality specified) has always been best, not bestvideo+bestaudio/best.

Was this changed sometime in the past 1 or 2 weeks ? There is no mention of this in the changelogs. And since the changelogs for some versions are a nondescript "Placeholder", I browsed through the commits headers as best as I can, but did not see anything of relevance.

Perhaps you didnt have ffmpeg back then?

I had FFmpeg before I even knew about youtube-dl & youtube-dlc. I'm using the latest stable FFmpeg v4.3.1 before & after this Facebook merged-MKV output issue started occurring, so it's not a FFmpeg issue.

when specifying a format id in your case (dash_hd_src) it's already mp4 with audio and there's no need to download audio separately and merge it into the mp4 .

Previously, there was no need to specify any format quality to get the MP4 directly.

In summary, if no quality format is specified:

PAST:

• Output: MP4 download, higher quality video, faster process

PRESENT: Affects recently-uploaded Facebook videos

• Output: Merged MKV output, lower quality video, slower process
• User needs to specify a quality format to get a fast MP4 download without additional merging

[pukkandan]

From Readme

Since the end of April 2015 and version 2015.04.26, youtube-dlc uses -f bestvideo+bestaudio/best as the default format selection (see # 5447, # 5456). If ffmpeg or avconv are installed this results in downloading bestvideo and bestaudio separately and muxing them together into a single file giving the best overall quality available. Otherwise it falls back to best and results in downloading the best available quality served as a single file.

[hairycactus]

Since the end of April 2015 and version 2015.04.26, youtube-dlc uses -f bestvideo+bestaudio/best as the default format

Thanks for the Readme info.

However, I have been using every youtube-dl & youtube-dlc version before & after v2015.04.26 -- together with every FFmpeg stable release version.

And I am very sure that the default is definitely not a merged MKV for Facebook videos. (As mentioned, I conscientiously avoid specifying any format, because the separate stream download & merge process is tediously slow here for some sites.)

As comparison, the default for Youtube videos (years-old, as well as recent 2-day-old) is still not a merged format -- or at least on my side. Not specifying any format quality continues to default to best (eg. fast direct MP4 720p) output without any merging. It is not giving a merged 'bestvideo+bestaudio/best' non-MP4 output.

[pukkandan]

As comparison, the default for Youtube videos (years-old, as well as recent 2-day-old) is still not a merged format -- or at least on my side. Not specifying any format quality continues to default to best (eg. fast direct MP4 720p) output without any merging. It is not giving a merged 'bestvideo+bestaudio/best' non-MP4 output.

hm.. that is weird. Can you give verbose log of such a video?