How to generate new images?

[NinePeAkSword]

Hi, I've trained a model and only got the pictures of the training process. The name of the model is 'model-10.pt'. I want to know how to use it to generate new images. How can I load it and use it? The training code is as follows:

if __name__ == '__main__':
    model = Unet(
        dim=64,
        dim_mults=(1, 2, 4, 8),
        channels=1
    ).cuda()

    diffusion = GaussianDiffusion(
        model,
        image_size=200,
        timesteps=1000,
        sampling_timesteps=250,
        loss_type='l1'
    ).cuda()

    trainer = Trainer(
        diffusion,
        r'F:\PyCharm_Files\Diffusion\lucidrains\denoising-diffusion-pytorch\Images_demo',
        train_batch_size=4,
        train_lr=8e-5,
        train_num_steps=10000,
        gradient_accumulate_every=2,
        ema_decay=0.995,
        amp=True
    )

    trainer.train()

[Echo-jyt]

Hello, I also want to know, have you found a solution

[modantailleur]

Once you've trained your network, you can use the sample method of diffusion. For example the following code will generate 1 image:

sampled_images = diffusion.sample(batch_size = 1)

def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)),
                            interpolation='nearest')
    plt.show()

show(sampled_images[0])

[PabloVD]

@NinePeAkSword To load a model previously trained, you can just load it with trainer.load("10"), which loads the 10-th checkpoint in the results folder.

[Shiyi-D]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

[Shiyi-D]

@PabloVD Hi thanks for your answer. I tried but found that train.load('10') is a Nonetype. Appreciated if you could help me figure it out.

[PabloVD]

@Infinity2118 can you check if you have model-10.pt inside the folder results? It automatically saves there the checkpoints by default.

[Adrian744]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

I do not understand your approach. A diffusion Model is learning the underlying the distribution of your training data. There is no such thing as a "test" set in this context, since the model itself is starting from random noise. If you want to have different images, you have to put those into your training data.

[modantailleur]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

This library is for generating images from white noise. You give the algorithm some random noise, and it generates an image that looks like whatever you used during training (a dataset of dogs will give you a random dog, a dataset of faces a random face etc...). It does not support denoising or super-resolution. I get that it can be a bit confusing given the name of the repo, but "denoising diffusion" only refers to the technique of stable diffusion, it doesn't mean that you'll be able to denoise anything from the trained algorithm. One easy way to transform this library into a super-resolution library, is to change a little bit the U-Net and to give on different channels (channels 4, 5, 6 for colorful images) the noisy image (it is explained in the paper "Image Super-Resolution via Iterative Refinement" by Saharia et al.). That's what I'm trying to do right now, so I don't know if it works.

[Shiyi-D]

@Infinity2118 can you check if you have model-10.pt inside the folder results? It automatically saves there the checkpoints by default.

Thx. I've figured it out. Thx again~

[Shiyi-D]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

I do not understand your approach. A diffusion Model is learning the underlying the distribution of your training data. There is no such thing as a "test" set in this context, since the model itself is starting from random noise. If you want to have different images, you have to put those into your training data.

Hi thx for your answer. I'm still wondering how to 'train' my another image set using the pth I've got (like model10.pth). That means I don't need to train again but use the trained model to my own data. Kind of pre-trained thing.

[Shiyi-D]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

This library is for generating images from white noise. You give the algorithm some random noise, and it generates an image that looks like whatever you used during training (a dataset of dogs will give you a random dog, a dataset of faces a random face etc...). It does not support denoising or super-resolution. I get that it can be a bit confusing given the name of the repo, but "denoising diffusion" only refers to the technique of stable diffusion, it doesn't mean that you'll be able to denoise anything from the trained algorithm. One easy way to transform this library into a super-resolution library, is to change a little bit the U-Net and to give on different channels (channels 4, 5, 6 for colorful images) the noisy image (it is explained in the paper "Image Super-Resolution via Iterative Refinement" by Saharia et al.). That's what I'm trying to do right now, so I don't know if it works.

Thx for your specific reply. Maybe I can read this paper and learn more about it. You've got the point that I want to use this model to do super-resolution since I saw "denoising". Anyway thx again. You're sooo nice.

[modantailleur]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

This library is for generating images from white noise. You give the algorithm some random noise, and it generates an image that looks like whatever you used during training (a dataset of dogs will give you a random dog, a dataset of faces a random face etc...). It does not support denoising or super-resolution. I get that it can be a bit confusing given the name of the repo, but "denoising diffusion" only refers to the technique of stable diffusion, it doesn't mean that you'll be able to denoise anything from the trained algorithm. One easy way to transform this library into a super-resolution library, is to change a little bit the U-Net and to give on different channels (channels 4, 5, 6 for colorful images) the noisy image (it is explained in the paper "Image Super-Resolution via Iterative Refinement" by Saharia et al.). That's what I'm trying to do right now, so I don't know if it works.

Thx for your specific reply. Maybe I can read this paper and learn more about it. You've got the point that I want to use this model to do super-resolution since I saw "denoising". Anyway thx again. You're sooo nice.

You're welcome ! Actually I think that the way of doing things that I mentioned in my message is a bit outdated (yes a 2021 paper is outdated for deep learning ahah), now people seem to be using more DDRMs with pre-trained models. The concept is that you train a model in an unsupservised way with a library like this one, or you download a pre-trained model that is relevant for your work. You now have a model that is able to generate an image from random noise. And then, you can condition the model during inference to generate an image that looks like the "noisy" image you give it as input. It will ultimately generate a denoised image. I suggest to take a look at the DDRM library (https://github.com/bahjat-kawar/ddrm) which is used only for inference, and to read their paper, as their method is not trivial at all.

The method I mentioned in my previous message works perfectly well, though ! I've tried it and it is fine for my study. The only thing is that training a diffusion model from scratch can be really tedious depending on your project (for my project it's fine because I'm working with small matrices). The DDRM library, on the contrary, allows you to use pre-trained diffusion models for basically any kind of denoising/restoration problem, which is amazing.

[Shiyi-D]

@modantailleur Hi thanks for your answer. I wonder what I can do instead of sampling if I wanna transport a new image(not consisted in the train set). That means I have a test set to be denoised using my trained trainer. Thanks in advance.

This library is for generating images from white noise. You give the algorithm some random noise, and it generates an image that looks like whatever you used during training (a dataset of dogs will give you a random dog, a dataset of faces a random face etc...). It does not support denoising or super-resolution. I get that it can be a bit confusing given the name of the repo, but "denoising diffusion" only refers to the technique of stable diffusion, it doesn't mean that you'll be able to denoise anything from the trained algorithm. One easy way to transform this library into a super-resolution library, is to change a little bit the U-Net and to give on different channels (channels 4, 5, 6 for colorful images) the noisy image (it is explained in the paper "Image Super-Resolution via Iterative Refinement" by Saharia et al.). That's what I'm trying to do right now, so I don't know if it works.

Thx for your specific reply. Maybe I can read this paper and learn more about it. You've got the point that I want to use this model to do super-resolution since I saw "denoising". Anyway thx again. You're sooo nice.

You're welcome ! Actually I think that the way of doing things that I mentioned in my message is a bit outdated (yes a 2021 paper is outdated for deep learning ahah), now people seem to be using more DDRMs with pre-trained models. The concept is that you train a model in an unsupservised way with a library like this one, or you download a pre-trained model that is relevant for your work. You now have a model that is able to generate an image from random noise. And then, you can condition the model during inference to generate an image that looks like the "noisy" image you give it as input. It will ultimately generate a denoised image. I suggest to take a look at the DDRM library (https://github.com/bahjat-kawar/ddrm) which is used only for inference, and to read their paper, as their method is not trivial at all.

The method I mentioned in my previous message works perfectly well, though ! I've tried it and it is fine for my study. The only thing is that training a diffusion model from scratch can be really tedious depending on your project (for my project it's fine because I'm working with small matrices). The DDRM library, on the contrary, allows you to use pre-trained diffusion models for basically any kind of denoising/restoration problem, which is amazing.

Sooooo happy to receive your reply again!!! Actually DDRM is the first model I learned in the diffusion models, which is more straight forward to denoising. I'd like to go back to it and take more attention of its architecture. Maybe I could strike my mind while learning. Thx for your recommendation! You're one of the best partners of my Github study! Hope you everything goes well~

[lchunleo]

@NinePeAkSword To load a model previously trained, you can just load it with trainer.load("10"), which loads the 10-th checkpoint in the results folder.

Can I check if I use the Trainer class (defined with the training data) to do the

Once you've trained your network, you can use the sample method of diffusion. For example the following code will generate 1 image:

sampled_images = diffusion.sample(batch_size = 1)

def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)),
                            interpolation='nearest')
    plt.show()

show(sampled_images[0])

Can I check the diffusion.sample will it take from the training set( I presume no) when it do the sample ? My understanding is that the sample is a noise ? I also puzzle where is the generative part after loading the checkpoint ? Sorry am new to diffusion and trying to figure things out.

[lypliuyouping]

Thx. I've figured it out. Thx again~

Hello, may I ask how to load a trained model's own image to obtain the generated image? Can you take a screenshot of the code