waifu2x-caffe icon indicating copy to clipboard operation
waifu2x-caffe copied to clipboard

Lossless YUV input, is it possible?

Open Dioxaz opened this issue 7 years ago • 7 comments

First of all, greetings for making something like waifu2x and watfu2x-caffe possible, as they clearly are the future in upscaling, especially low-resolution sources as they're starting to look very dated and uncomfortable to work with or look at.

I recently did some tests with CUDA acceleration and am really impressed by the results. Problem: what interests me isn't upscaling still images but low-resolution videos which you can still find on streaming platforms like Youtube and Nico Nico Douga.

It is indeed possible to convert such videos to an image sequence and then pass the folder into waifu2x-caffe. Second problem: waifu2x seems to operate better in native YUV than with YUV-converted-to-RGB sources and create some nasty upscaled jaggies on chroma with contrasted solid colour areas.

I'll be using this frame (512x384) example from this video to illustrate my point: sm24627927_0473

Below is PNG input converted from video (watch the green ribbon next to the hair, on the left notably): sm246279270473 y scale width 1440

One solution I found is passing JPEGs converted from video. This way, the yuv420p colorspace is kept. waifu2x indeed treats this as YUV and by using the "2-D illust (Y Model)" model, nasty chroma jaggies are gone but! Third problem: I just can't seem to be able to produce proper lossless JPEGs from ffmpeg so my resulting pictures always have a tiny amount of degradation, which is problematic if the source video resolution is 512x384 or 640x360 (and I simply failed to encode anything in ljpeg or lossless jpeg as I'm always getting an error with ffmpeg).

Example of passed JPEG to waifu2x. Notice the slight degradation, even at max settings with ffmpeg (qmin and qmax at 1 and -q:v at 0). Also, no jaggies around the green ribbon: sm246279270473 y scale width 1440 jpg

So far my only working solution is creating a lossless WebP image sequence from my videos and only this way I'm able to keep waifu2x working in YUV without conversion to RGB. But, as it would be too easy, WebM encoding is painfully slow (2fps average on my ageing Q6600): sm246279270473 y scale width 1440 webp

I also tried JPEG2000 and TIFF image sequences, in order to keep the yuv420p colorspace but unfortunately, waifu2x converts them to RGB prior to treatment and the nasty chroma jaggies are back. sm246279270473 y scale width 1440 jp2

So this leads me to my question (I hope I wasn't boring). Is it possible to feed lossless YUV still pictures to waifu2x, other than the really slow WebP format? Is there some particular manipulation I forgot or is it simply not possible at the moment?

Regards, and keep up the good work.

Dioxaz avatar Dec 07 '16 19:12 Dioxaz