diffusers-ocaml

diffusers-ocaml copied to clipboard

• Diffusers-OCaml: A Diffusers API in OCaml

This work is a port of the [[https://github.com/LaurentMazare/diffusers-rs/][diffusers-rs]] library written in Rust.

** Stable-Diffusion *** How to run? - Download weights and save it data directory. For instructions on how to do this, refer [[https://github.com/LaurentMazare/diffusers-rs#converting-the-original-weight-files][diffusers-rs-docs]] - Once you have ".ot" files, you are ready to run stable diffusion - I have found that placing unet and clip on GPU, and placing vae on cpu works the best for my GPU. Here is the command that does that. #+begin_src bash dune exec stable_diffusion -- generate "lighthouse at dark" "vae" "data/pytorch_model.ot" "data/vae.ot" "data/unet.ot" #+end_src - To run all models in CPU #+begin_src bash dune exec stable_diffusion -- generate "lighthouse at dark" "all" "data/pytorch_model.ot" "data/vae.ot" "data/unet.ot" #+end_src - To generate more than 1 sample, use the num_samples parameter #+begin_src bash dune exec stable_diffusion -- generate "lighthouse at dark" "all" "data/pytorch_model.ot" "data/vae.ot" "data/unet.ot" --num_samples=2 #+end_src

*** Sample generated images #+CAPTION: lighthouse at dark #+NAME: fig:lighthouse.png [[./media/lighthouse.png]] #+CAPTION: rusty robot holding a torch #+NAME: fig:rusty_robot.png [[./media/sd_final.2.png]] *** Performance on 8GB Nvidia GeForce GTX 1070 Mobile GPU It takes about 27 seconds to generate an image. Measurements were done on a 12 CPU Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz. Running all the steps in CPU takes little more than three minutes. I place vae on CPU; unet and clip on GPU #+begin_src bash real 0m25.110s user 0m39.271s sys 0m11.518s #+end_src ** Image to Image generation *** How to run? #+begin_src bash dune exec img2img -- img2img media/in_img2img.jpg #+end_src *** Sample generated image

• Input image #+CAPTION: img2img #+NAME: fig:in_img2img.png [[./media/in_img2img.jpg]]
• Output image #+CAPTION: img2img #+NAME: fig:out_img2img.png [[./media/out_img2img.png]] *** Performance on 8GB Nvidia GeForce GTX 1070 Mobile GPU I placed vae on CPU; unet and clip on GPU #+begin_src bash real 0m15.628s user 0m34.571s sys 0m5.833s #+end_src ** Inpaint *** How to run? #+begin_src bash dune exec inpaint -- generate media/sd_input.png media/sd_mask.png --cpu="vae" --prompt="Face of a panda, high resolution, sitting on a park bench" #+end_src *** Sample generated image
• Prompt: Face of a panda, high resolution, sitting on a park bench #+CAPTION: inpaint-input #+NAME: fig:sd_input.png [[./media/sd_input.png]] #+CAPTION: inpaint-mask #+NAME: fig:sd_mask.png [[./media/sd_mask.png]] #+CAPTION: inpaint-output #+NAME: fig:panda.png [[./media/panda.png]] *** Performance I placed vae on CPU; unet and clip on GPU #+begin_src bash real 0m28.055s user 0m51.600s sys 0m13.851s #+end_src ** Note Only FP32 weights are supported.