(quick demo; enable sound)

Audio Visual Playground

How it works

• Animated stable diffusion - Hosted on Replicate
• WebGL, three.js, glsl for rendering
• Meyda for audio feature extraction
• MediaRecorder for recording video in-browser
  • Great so I don't need to host any heavyweight servers
• ffmpeg for converting webm ⇒ mp4
  • ffmpeg -i test.webm -ss 0.05 -vf scale="iw/2:ih/2" -c:v libx264 -crf 16 -movflags faststart -pix_fmt yuv420p -r 40 -profile:v main -preset medium test.mp4

TODO

• [x] add stroke styles in addition to fill
• [x] add circle style
• [x] add mirror option
• [x] fix output pixel density
• [x] start/pause/stop should be async
• [ ] separate download or get blob methods
• [ ] mp4 output support
• [ ] render offscreen sped-up
• [ ] add demo to readme
• [ ] add basic docs
• [x] hosted demo
• [ ] explore backgrounds, color palettes, and avatars
• [ ] explore different post-processing effects
• [ ] add descript-style animated captions
• [ ] add UX for generating custom backgrounds using replicate API

Inspiration

• https://www.youtube.com/watch?v=QykkWNOtap4
• https://www.youtube.com/watch?v=Q1bxyKOZ5RI

License

MIT © Travis Fischer

If you found this project interesting, please consider sponsoring me or following me on twitter