Stability of 'seed'

[enricoros]

I've tried specifying Imagine(seed = SOME_CONST) a few times and I get different results all the time. Is the same happening for you, @lucidrains?

I don't know where other sources of random are; I was thinking it would be good to start many different random generations and save the seeds, and then select the most promising output and restart the generation of higher quality outputs with a larger number of iterations.

[lucidrains]

@enricoros Hi Enrico! Want to try the latest version? :)

[enricoros]

Impressive! It's not fully reproducible yet but pictures converge to similar styles! Like, with seed=1234 I get all pictures made of Leaves :D

[enricoros]

I've tried adding np.random.seed(seed), and prefer without it.

Then tried torch.set_deterministic(True) and gotten this explanation at runtime: RuntimeError: upsample_bilinear2d_backward_cuda does not have a deterministic implementation, but you set 'torch.set_deterministic(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

Then I disabled bilinear=False and now it is pixel perfect :D

[enricoros]

Removed line by line to get to the minimum set that is deterministic (md5sum of the png files are identical).

Minimal code: if exists(seed): torch.set_deterministic(True) torch.manual_seed(seed) Submitting patch! :)

[enricoros]

Opened on pull request https://github.com/lucidrains/big-sleep/pull/12. I hope it's pixel-equal on other versions of PyTorch/CUDA/OSs.

[enricoros]

Note after further testing: this will work well on a single execution, but there's a strange issue remaining when looping code such as the following:

while True:
    imagine = Imagine(
        ...
        seed=1234,
        ...
    )
    imagine()

A strange pattern emerges: the first generation "A" is always the same. Then a "B" version of an image follows, then a "C" version and the full sequence will look like: [A, B, C, B, C, C, C, C, C, ...]

Tried sprinkling the code everywhere with:

    torch.set_deterministic(True)
    torch.manual_seed(0)
    random.seed(0)
    np.random.seed(0)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

but this doesn't seem to help, it's always A, B, C, B, C C C C C...

The following shows what I mean:

[htoyryla]

Was just looking into this also when wondering why results were so different compared to the original Colab.

[enricoros]

Update for the looping case. If after forward() finishes, we delete the model (before its recreation in a new Imagine()), the results become more predictable, but there's still some dangling randomness somewhere: ... del self.model In which case the sequence generated becomes: A, B, B, B, B...

Still, we're getting closer!

[enricoros]

@htoyryla What kind of difference do you see? I love the results here, some are really outstanding

[htoyryla]

It was actually a colleague of mine who was using the Colab who commented that some results from this implementation were not as good. I started looking at it, compared the code and and told him it might somehow depend on initialisation. After using manual seed and adding some torch settings before I went to sleep, I was already getting similar but not identical results as from the original Colab.

[enricoros]

@htoyryla can you try with torch.set_deterministic(True) and see if you get reproducible results? Run that line as early as possible. Then it could depend on the status of memory/CUDA initialization... would be nice to narrow it down to a full reproducibility. I see that the original colab uses 0 as the seed.

[htoyryla]

I will check. Meanwhile I enjoy this "a cityscape in the style of Van Gogh" using manual seed 0 and torch.cudnn.deterministic NOT set to true.

[htoyryla]

OK. Visually identical results on two subsequent runs using cuda 10.1 and the following settings:

torch.set_deterministic(True)
torch.backends.cudnn.enabled = True
torch.backends.cudnn.deterministic = True
torch.manual_seed(0)
random.seed(0)
#np.random.seed(0)
torch.backends.cudnn.benchmark = False

My previous runs were on cuda 10.2.

[enricoros]

@htoyryla Are the results also the same as the original notebook or different?

[htoyryla]

@htoyryla Are the results also the same as the original notebook or different?

Different. It is unclear to me what torch settings are in use there, I only found that it used torch.manual_seed(0).

[enricoros]

Yeah, that's not enough. Dunno how it worked. I'm trying the same to see if the image output is similar. Can you post your output?

[htoyryla]

I noticed I had a copy&paste error in the prompt. Testing again to see how the results turns out now.

Here's my output at epoch 20 for "a cityscape in the style of Van Gogh":

Now I think I need to take my time and check all my settings though.

[enricoros]

This is mine at Iteration 700:

I've only added the line torch.set_deterministic(True) before torch.manual_seed(0) and every time I run the notebook I get this output. Possibly if you run it further, you'll get to the same result!

[htoyryla]

I am not sure we have the same problem even if we are focusing on the same settings :) E.g. my colleague liked the results he was seeing in the Colab and I tried to show similar enough results from the command line, therefore the solution was not to modify the Colab (though I think I'll try it too)

I think I have tools enough now to get results like we want (more a question of aesthetics than exact reproducibility here.

[lucidrains]

would it be more convenient if I offered a imagine.set_text(new_text) and imagine.reset() commands? the reset would just reinstantiate the Latent parameters

[enricoros]

@lucidrains do you think it will fix the reproducibility issue for good? if it does, it's a wonderful addition. Otherwise my workaround will be to use the tool from the command line, so every time it's restarted from scratch (but will take a bit longer). The use case is to be able to generate many runs at around 50 iterations, and then select the most pleasing seed to run a few thousand iterations on that. It's "human-in-the-loop" creativity.

[lucidrains]

@enricoros https://github.com/lucidrains/big-sleep/commit/7ed17c249e14f5f70b5171f145802b2b8ad88d0c give it a shot!

[htoyryla]

This is mine at Iteration 700:

I've only added the line torch.set_deterministic(True) before torch.manual_seed(0) and every time I run the notebook I get this output. Possibly if you run it further, you'll get to the same result!

Yes, modifying the Colab I get the same image.

[walmsley]

I'm finding that on Colab, the "A B B B B B ..." pattern found by @enricoros holds true.

Here's how I'm achieving full reproducibility using Colab:

At first I was painfully manually restarting the notebook every time to ensure reproducibility (and try to always get an "A" result)... BUT then I realized that the easier thing to do is: at the start of the notebook just run a few iterations of a dummy model to "warm up" the GPU, before re-creating the model for real usage. This way, I manage to ignore the "warm-up" and happily only ever get consistent "B" as a result on every loop.

[LtqxWYEG]

I've observed some similarities between images made with certain seeds, regardless of input text:

#Seed values and styles
# 528 = only white or brown dog
# 519 = beehive in tree?
# 5092785 = dark maybe, maybe not?
# 5092387857 = indoor scene with people
# 50923887857 = flowers, plants, (colorful)
# 50921568718573 = buildings in green scenery, clouds, or blue thing on concrete
# 12 = picture of a women
# 11 = people holding things??
# 10 = leaves, insects, garden?
# 9 = dogs, orange oval?
# 8 = text, canvas?
# 7 = pets in cages?
# 6 = indoors human interaction
# 5 = 
# 4 = black and white, people
# 3 = street pictures?
# 2 = family pictures
# 1 = backyards, parks?
# 0 = oval horse-thing, dunes, sattelite images??

I was under the impression that the seed would set a starting point for the RNG and not an actual set of images the NN was trained on? (But then, we're training it...?) I mean that's generally what a "seed" implies. However, using longer numbers, up to 18446744073709551615, as seed works just as well, so how can it realistically produce similar images with different input text, if seed would set a picture and not a number? Was it trained on 18446744073709551615 images?? And it seems like every number has two images attached Can someone write a short explanation, please? :)