Better result for PixelArt scaling

[RyanBram]

Hi, Developers,

For common image, currently Waifu2x is still the best compared to other algorithm. But, if we want to scale a pixel art picture, Waifu2x tends to resulting jagged picture for the result. The following is the example:

The picture above is the original version. When I scaled it 3 times, the result is like this:

The left picture is Waifu2x and the right picture is xBR. We can see that Waifu2x produce sharper result, but unfortunately it cannot detect the edge well, so we see jagged lines at the edge of the picture. In this case xBR show better result for edge detection.

I hope there will be solution for this.

Thanks.

[nagadomi]

waifu2x is a machine learning based upscaler. It requires input and ideal output pairs for training. In the case of pixel art, it is difficult to make it. I can make it from pixel art upscalers like xBR or https://johanneskopf.de/publications/pixelart/ , but it would be better to use them directly.

[RyanBram]

I can use pixel art scaling algorithm like xBR directly, but from the results above, I learnt that Waifu2x still produce better result as seen in the picture's line on pants and line on white shirt. The only thing that it lacks is edge detection to avoid jagged lines. I try to combine the result manually and the result seems good enough:

[luigio]

While not tailored for pixel art, I find that waifu2x still produces pretty good results with it. In your example, it seems the problem is simply related to lack of support for alpha channel. If you fill the background with white, it detects outer edges correctly. Here's the (amazing) result that I got that way:

[RyanBram]

Here's the (amazing) result that I got that way:

Wow, it just so amazing. It seems your result is what I am looking for. I tried your method using http://waifu2x.udp.jp/, but the result still similar with mine above nor as good as you.

How you got that result?

[nagadomi]

It seems like using denoising. It will be more smooth edges but it is a side effect.

[luigio]

Yes, I selected artwork style and highest noise reduction every time.

It's not perfect, as you can see some details are lost (like the leftmost vertical line in each leg), but its intuition in the details that it kept was very good.

I'm sure it could produce even better results if the network were trained specifically for pixel art with proper input-output pairs.

My suggestion would be to make the output picture in each pair by taking a high-resolution manga picture, reducing it to 16 colors without dithering and shrinking the resulting picture by a factor of 8 or more without color constraints to get natural anti-aliasing. For the input picture, just reduce the input picture to the same set of 16 colors as before, without dithering. I wish I could do it myself, but my programming skills are very limited. :(

[nagadomi]

I think it is very difficult to make pixel art from illustrations or photo. The generated input image must have the same rules as actual pixel art. That is not possible with a generic image resizing algorithm.

[luigio]

It's not quite ideal, but I'm pretty sure it's close enough. At the very least, it will always detect rigged edges and make them smooth, which is the main point.

Also, the rules for true pixel art have to be very similar to the ones I mentioned, or else our eyes wouldn't be able to interpret such images correctly.

[nagadomi]

hmm, here is my current opinion.

art/photo have antialiasing and color gradient on its own, so it is difficult to reduce color properly. For two-color manga(no screen tone), good results may be obtained, but that data can not be used to convert colored pixel art. One possibility is to convert from 3DCG to pixel art. In the 3D model with simple materials and no antialiasing, we do not have to consider color reduction algorithm and we can draw proper edges. For example, mozuya's work: https://www.youtube.com/watch?v=XU_3ZxkDO7U However, it is a hard work to generate various types of training data in that way. Another possibility is to choose training data from danbooru's "oekaki" tag. These images are not antialiased, It might be little easier to convert to pixel art, I think.

well, anyway, that is a troublesome task. if someone release a library or dataset, I will use it.

[luigio]

Here's an example of what I mean. I'll use this picture as a starting point:

First I reduce color without dithering. I think anything over 8 colors will usually be too much, so let's say I use just five basic colors this time: red, blue, green, white and black. Here is the result:

Next I convert the picture back to RGB (i.e. no color limit) and shrink it by a factor of 8:

This will be used as the output picture in a training pair. Note that it's perfectly anti-aliased.

For the input picture, I reduce color again without dithering to get an aliased picture. I use the same set of five colors as before. This is what I get:

So there you have it. You can have Photoshop automatically generate input-output pairs this way from any image database. The one that you used for waifu2x should be fine.

With enough training, the neural network will learn how to turn an aliased picture (pixel art) into an anti-aliased one. Perfect or not, I'm pretty sure it will produce much better results than xBR, and you can always feed the result into waifu2x for upscaling.

Also note that you can get multiple pairs from a single initial picture just by reducing to different sets of colors each time. Here's a different pair that I got from the same picture by letting Photoshop choose the 5 colors that it wanted:

I think a tool like this would be great for (among others) indie adventure game developers, which are fond of using 320x200 resolutions for their games. They could still draw in that resolution, which is faster and cheaper for them, and then make their drawings bigger and painting-like in a couple of clicks.

[luigio]

This is almost exactly what I meant: https://imgur.com/gallery/YDbBZ

[bloc97]

This is an artifact of the pre-upscaling filter used in Waifu2x, if I'm not mistaken. Using a transposed convolution instead of a fixed bilinear/bicubic filter could make extremely-low resolution cases easier for the network.

[bloc97]

In this example you can see that the effective "blur" for the 4x4 pixel image upscaled to 8x8 occupies a large portion of the resulting image, thus making the NN believe that it is part of the image, while the 64x64 upscaled to 128x128 is much more easily resolvable.

[bloc97]

As a workaround, it is possible to pass the original image through a xBRZ filter, downscaling it using bicubic/bilinear, and then applying Waifu2x. This is equivalent of doing a pixel-art pre-smoothing step to make the image less jagged. This is what I obtained. Very promising!

[luigio]

I'm kind of disappointed that nobody has trained a neural net for pixel art upscaling yet. No reason why it shouldn't work, and it would be very useful for indie adventure game developers.

[RyanBram]

I think the problem is because nobody provide proper training materials yet.

I don't know how useful it will be, but maybe something like this DevinatArt gallery can be used for starting point, because it contains low res and hi res pixel art to be compared: https://omegachaino.deviantart.com/art/Street-of-Rage-2-Axel-Pixel-upgrade-634629192

Or maybe we can use sprite from old version of King of Fighters game and compare it with latest version of King of Fighters game. Most sprites and fighting styles don't change much, the only different just the resolution.

[luigio]

I think the problem is because nobody provide proper training materials yet.

As I mentioned before, input-output pairs for training are very easy to create from any image database.

I don't know how useful it will be, but maybe something like this DevinatArt gallery can be used for starting point, because it contains low res and hi res pixel art to be compared: https://omegachaino.deviantart.com/art/Street-of-Rage-2-Axel-Pixel-upgrade-634629192

It will be very hard to get enough such pairs, and it's not what I'm looking for anyway. I like the lack of detail of pixel art, I just don't like seeing the pixels. So I'd love to have a tool for generating bigger, anti-aliased versions of pixel art images without adding extra detail. Something like xBR but more accurate.

[bloc97]

The problem is not that training such a network is hard. There's new network structures that are extremely simple, fast to train and are better than what Waifu2x was originally based upon. You could basically implement one yourself in a few dozen lines in Torch or Caffe and train it in a few hours on a modern GPU. But making an portable version that does not require Linux or the deep learning libraries is somewhat complicated.

[bloc97]

Until there's a good amount of demand for what you need, or if you are willing to give your own time and develop a NN-based pixel art upscaler, I am afraid that we will not see one in the near future.

[nagadomi]

mitaki28 has released a pixel art upscaler. https://github.com/mitaki28/pixel-art-upscaler It seems to be using Nearest Neighbor to create training data.

[luigio]

Thanks, looking good! (Although the demo is quite buggy.)

[luigio]

This is the result I got from it:

[SenhorFlibble]

Hopefully not too late to post, but it is possible to accumulate a certain amount of authentic 8-bit video game graphics that came in both high and low resolution. Off the top of my head:

• in C&C: Red Alert, there's a number of cinematic stills (actually 3D renders) that are shown during the game credits, which are both in 320x200 and 640x400 resolutions
• the early versions of In Pursuit of Greed use lower resolution character sprites (again, of the pre-rendered variety) than the final version, apparently made from the same models; both sprite sheets can be found in the source code release of the game
• the Macintosh release of Wolfenstein 3-D uses high-resolution versions of the textures and decorations (enemy sprites are probably too different from their DOS counterparts to be useful). Some of these were used in Doom II's secret levels
• the data files of The Elder Scrolls: Daggerfall contain a set of higher-resolution duplicates of certain NPC flats that are used in the game, but in some cases there are different details. These are hand-drawn sprites
• also with Daggerfall, back in 1996 some of the NPC animations (more precisely, two dancing females) were published as GIFs in higher resolution in the NUKE review than the same characters used in the actual game. These are pre-rendered
• Rowan Software's flight sims (Dawn Patrol, Air Power etc.) contain high and low resolution cockpit images, the high-res ones being 640x400 matching the 320x200 low-res versions (I'm supposing that high-res stuff with proper aspect ratio, 640x480, would not be useful for low-res 320x200 images). Also there are some static screens in both resolutions
• Interplay's website had GIFs of some of Descent's weapon and other pickups at their old website, retrievable via the Wayback Machine More stuff can be probably obtained from other PC titles that supported both high and low resolution back then.

[DonaldTsang]

Here is an idea: make an ML-based pixel framer and downscaler, and then feed the results into Waifu2x, and then reverse the procedures using the same pixel grid to upscale

[nagadomi]

https://github.com/WuZongWei6/Pixelization can convert images into natural looking pixel art. I think this model has the ability to generate better paired data than before. However, it is non-commercial license.