UNet Example

[tanmayb123]

I'd love to contribute an example of using UNet for image segmentation - would this be useful?

[rxwei]

Sure thing!

[tanmayb123]

This is what I've got so far (the code is pretty messy right now): https://gist.github.com/tanmayb123/c3106c64ee8b9e07b729465b0744477b

(sidenote: I had to use #tfop("Concat" ...) because Tensor.concatenated(...) isn't differentiable)

But I get this error:

Stack dump:
0.	Program arguments: /root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift -frontend -interpret DeepUNet.swift -disable-objc-interop -color-diagnostics -module-name DeepUNet
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44d21cf]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44d0320]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44d24d2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f166318d390]
[0x7f16405d5553]
[0x7f16405cf04d]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0xcb369e]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0xcb7542]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x51b50f]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4efe14]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4eb366]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x49229e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f164edbd830]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4900d9]
Segmentation fault

Did I do something wrong?

[rxwei]

#tfop are not differentiable and will lead to undefined behavior (so don't use them!). You can define your differentiable version of Tensor.concatenated(...) locally in your file using the differentiableFunction(from:) operator or the @differentiable(vjp: ...) attribute. I'll add the derivative for you in stdlib tomorrow.

[tanmayb123]

Ah I see. I'll try again tomorrow. Thanks!

[tanmayb123]

In the meantime, I've created a PR for the UpSampling2D layer to be added to your DeepLearning repo: https://github.com/rxwei/DeepLearning/pull/6

[rxwei]

Excellent!

[tanmayb123]

@rxwei There's one more issue I just came across.

This code: https://gist.github.com/tanmayb123/4c140c011fe1c262d22968172a3475ad crashes with this stack dump:

Stack dump:
0.	Program arguments: /root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift -frontend -interpret abc.swift -disable-objc-interop -color-diagnostics -module-name abc
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44cf04f]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44cd1a0]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44cf352]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f298b8ab390]
[0x7f2968cf9483]
[0x7f2968cf504b]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0xcb3bbe]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0xcb7a62]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x51b50f]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4efe14]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4eb366]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x49229e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f29774db830]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4900d9]
Segmentation fault

When I run with the -O flag:

Stack dump:
0.	Program arguments: /root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift -frontend -interpret abc.swift -disable-objc-interop -O -color-diagnostics -module-name abc
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44cf04f]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44cd1a0]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x44cf352]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fe3aab45390]
[0x7fe3aada113b]
[0x7fe3aad9878a]
[0x7fe3aad98537]
[0x7fe3aad98001]
[0x7fe3aad9847a]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/lib/swift/linux/x86_64/libswiftTensorFlow.so(__tf_hoistable_Float+0x1e)[0x7fe3847ed28e]
[0x7fe3aad88682]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0xcb3bbe]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0xcb7a62]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x51b50f]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4efe14]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4eb366]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x49229e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fe396775830]
/root/ss/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/swift[0x4900d9]
Segmentation fault

Notice that in the code I linked, in the applied(to:) function, I don't use the up1_c2 layer. However, if I comment lines 63 and 54 (where I declare and initialize the up1_c2 layer), the code works just fine.

This stumped me for some time yesterday, any idea where this may be coming from?

[rxwei]

Could you help file a bug on https://bugs.swift.org with the reproducer?

[saeta]

We'd love for someone to open a PR with a working UNet example! :-)

[vballoli]

Can I take this PR up ?

[vballoli]

@saeta @dan-zheng I've finished the UNet model, but I'm unable to find an appropriate dataset for the implementation, for example the dataset used in the paper i.e website has images in tiff format and I'm having some issues loading it, which I'll be putting up on the Google groups, but I'd love your suggestions for the dataset.

[brettkoonce]

oxford pets dataset has segmentation masks: https://www.robots.ox.ac.uk/~vgg/data/pets/

[s1ddok]

I've got a U-Net based generator for my pix2pix implementation. I would be happy to contribute it to this project in case you think that implementation is good

https://github.com/s1ddok/pix2pix-s4tf

[vballoli]

I'm almost done with the UNet example code with the EM Segmentation dataset as implemented in the paper. Sorry about the delay, I was in the midst of my exams and couldn't devote much time. Should be sending a PR in day.

[vballoli]

I'm sorry again about the delay again. Will pe posting the link to my code soon for comments before I officially open a PR.