image-super-resolution
image-super-resolution copied to clipboard
idealo
Inference of large images
I think it would be a great feature to have some option for prediction of larger images, by applying sr to tiles of the main image and stitching the results, this is after enabling gpu inference of course. Thanks!
Unfortunately this does not work. https://colab.research.google.com/drive/1L8Q6_MkTqBDL3NCYi6bAdGq_lq7I_vs-
@cfrancesco Do you mean by calculation time gains? Actually, my concern was on ability to run inference on images larger then GPU memory, I am not expecting a speed gain, but ability to use GPU accelerated functionality on very large images (40-50 megapixels).
@ontheway16 sorry, I misunderstood. It's a very valid point - I added it in the todo list. Thanks!
seems it can change the backbone to mobilenet to reduce the parameters of network. this change can make the large image inference more easier.
@whitelok If it will come without an accuracy drop, then might be. But speed is not ciritical, speaking for myself. it can take minutes, instead of seconds, if better accuracy obtained.
I tested similar things with CycleGAN and pix2pix (different use, but same theory). The algorithms don’t care about how images are processed before and after they are executed, but there’s the issue of consistency between the tiles and at the borders of the tiles, I did not know how to manage that, as the content of each tile changes what the model detects
@victorca25 do you mean seam lines occur ? I made some tests in another project, named EDSR, and there was no problem with their solution (as a feature its already available in there; --chop parameter)
@ontheway16 in my case it did, because cycleGAN’s results are not necessarily consistent between two pieces of the same image, but maybe the SR is much more consistent and this won’t happen?
@victorca25 In my tests joints were seamless, so you may be right, some models might be more suitable for this.
Just in case it's useful, this simple script handles the cropping of images: https://github.com/xinntao/BasicSR/blob/master/codes/scripts/extract_subimgs_single.py
@ontheway16 Any suggestions on which architectures lead to more seamless results? My process here definitely has artefacts along the tiling lines
https://colab.research.google.com/drive/1pwSlxnbmKYHxwA-s6mf1aMFl8saWc6Ii
Fullsize output is too big to display in Colab without crashing it! in this example my tiles were 400x300 which is smaller than necessary - but the same thing happens no matter how big they are - I just haven't figured out how to calculate the max size I can fit in memory at one time. https://i.imgur.com/8eWduaj.jpg
The seams issue can easily be solved by padding. I worked around this exact problem for Prosr image upscaling here: https://github.com/fperazzi/proSR/issues/21
This approach should be easy to apply here as well. I'll probably go and do it myself if I find time, but in the meantime anyone can feel free to do anything they want with my code.
@jongold, had no seam lines with edsr, although it was some other code. Made no seam tests with this one.
The seams issue can easily be solved by padding. I worked around this exact problem for Prosr image upscaling here: fperazzi/proSR#21
This approach should be easy to apply here as well. I'll probably go and do it myself if I find time, but in the meantime anyone can feel free to do anything they want with my code.
Cool idea @Shadetail . Atm I'm working on a model variant that should allow faster inference and could not require special treatment for large images. Regardless, in two weeks I'll be back from conferences and vacation and I'll implement an option for larger images with padding.
Obviously, feel free to create a PR. I would say it should be implemented as an option in the predict function of the ImageModel class.
Thanks for you contribution.
Hi, I added a first (fairly inefficient) implementation of an option in the model predict function to process the image by patch. The usage is the following, using for instance a RRDN model
model = RRDN(arch_params={'C':4, 'D':3, 'G':32, 'G0':32, 'T':10, 'x':4})
model.model.load_weights(weights)
sr_img = model.predict(image, by_patch_of_size=30)
Note, this is not a Keras function, so you need to use the predict function from ISR ImageModel class, NOT the predict function that belongs to the Keras model, so
sr_img = model.predict(image, by_patch_of_size=30)
instead of
sr_img = model.model.predict(image, by_patch_of_size=30)
By default by_patch_of_size is None. If it's not none, than the model will try to compute by patch. Padding can also be specified (by default is 2) and it's the padding for each of the patches. Batch size (how many patches are inferred in one go) is also a parameter (default is 10). It has no tests yet and it's in the dev branch for now, but it should work fine. Let me know if you find any problem with it.
Hi, is it possible to port the patch prediction model to tensorflowjs? If yes, how would it work for inference? I ported the normal model over to js, and used it with success, although even on a gtx 1070 8gb of ram i can't go from 128x128 to 512x512. I can do it with CPU but it takes around 1 hour!!! This option would be a great place to start!
