DLSS
DLSS copied to clipboard
Deep Learning Super Sampling with Deep Convolutional Generative Adversarial Networks.
Deep Learning Super Sampling
Using Deep Convolutional GANS to super sample images and increase their resolution.
How To Use This Repository
-
Tutorial
-
Requirements
- Python 3
- Keras (I use
2.3.1
) - Tensorflow (I use
1.14.0
) - Sklearn
- Skimage
- Numpy
- Matplotlib
- PIL
-
Documentation
-
DLSS GAN Training
-
This script is used to define the DCGAN class, train the Generative Adversarial Network, generate samples, and save the model at every epoch interval.
-
The Generator and Discriminator models were designed to be trained on an 8 GB GPU. If you have a less powerful GPU then decrease the
conv_filter
andkernel
parameters accordingly. -
User Specified Parameters:
-
input_path
: File path pointing to the folder containing the low resolution dataset. -
output_path
: File path pointing to the folder containing the high resolution dataset. -
input_dimensions
: Dimensions of the images inside the low resolution dataset. The image sizes must be compatible meaningoutput_dimensions / input_dimensions
is a multiple of2
. -
output_dimensions
: Dimensions of the images inside the high resolution dataset. The image sizes must be compatible meaningoutput_dimensions / input_dimensions
is a multiple of2
. -
super_sampling_ratio
: Integer representing the ratio of the difference in size between the two image resolutions. This integer specifies how many times theUpsampling2D
andMaxPooling2D
layers are used in the models. -
model_path
: File path pointing to the folder where you want to save to model as well as generated samples. -
interval
: Integer representing how many epochs between saving your model. -
epochs
: Integer representing how many epochs to train the model. -
batch
: Integer representing how many images to train at one time. -
conv_filters
: Integer representing how many convolutional filters are used in each convolutional layer of the Generator and the Discriminator. -
kernel
: Tuple representing the size of the kernels used in the convolutional layers. -
png
: Boolean flag, set to True if the data has PNGs to remove alpha layer from images.
-
-
DCGAN Class:
-
__init__(self)
: The class is initialized by defining the dimensions of the input image as well as the output image. The Generator and Discriminator models get initialized usingbuild_generator()
andbuild_discriminator()
. -
build_generator(self)
: Defines Generator model. TheConvolutional
andUpSampling2D
layers increase the resolution of the image by a factor ofsuper_sampling_ratio * 2
. Gets called when the DCGAN class is initialized. -
build_discriminator(self)
: Defines Discriminator model. TheConvolutional
andMaxPooling2D
layers downsample fromoutput_dimensions
to1
scalar prediction. Gets called when the DCGAN class is initialized. -
load_data(self)
: Loads data from user specified file path,data_path
. Reshapes images frominput_path
to haveinput_dimensions
. Reshapes images fromoutput_path
to haveoutput_dimensions
. Gets called in thetrain()
method. -
train(self, epochs, batch_size, save_interval)
: Trains the Generative Adversarial Network. Each epoch trains the model using the entire dataset split up into chunks defined bybatch_size
. If epoch is atsave_interval
, then the method callssave_imgs()
to generate samples and saves the model at the current epoch. -
save_imgs(self, epoch, gen_imgs, interpolated)
: Saves the model and generates prediction samples for a given epoch at the user specified path,model_path
. Each sample contains 8 interpolated images and Deep Learned Super Sampled images for comparison.
-
-
-
Load Model and Analyze Results
- This script is used to super sample images using the Generator model trained by the
DLSS GAN Training
script. - The script will perform DLSS on all images inside the folder specified in
dataset_path
. You can insert frames of a video in here to create GIFs such as the one in the results section of this document. -
User Specified Parameters:
-
input_dimensions
: Dimensions of the image resolution the model takes as input. -
output_dimensions
: Dimensions of the image resolution the model takes as output. -
super_sampling_ratio
: Integer representing the ratio of the difference in size between the two image resolutions. Used for setting ratio of image subplots. -
model_path
: File path pointing to the folder where you want to save to model as well as generated samples. -
dataset_path
: File path pointing to the folder containing dataset you want to perform DLSS on. -
save_path
: File path pointing to the folder where you want to save generated predictions of the trained model. -
png
: Boolean flag, set to True if the data has PNGs to remove alpha layer from images.
-
- This script is used to super sample images using the Generator model trained by the
-
-
Results
- The results of Nearest Neighbor Interpolation are compared to the DLSS algorithm.
- The results of Nearest Neighbor Interpolation are compared to the DLSS algorithm.
-
Generated Training Sample
-
Generator Model Architecture
- Using
(5,5)
Convolutional Kernels with64
filters. -
input_dimensions = (128,128,3)
andoutput_dimensions = (256,256,3)
- Using
-
Discriminator Model Architecture
- Using
(5,5)
Convolutional Kernels with64
filters. -
input_dimensions = (128,128,3)
andoutput_dimensions = (256,256,3)
- Using