emojicam

An emoji filter for your webcam using shumai + bun! (writeup)

For fidelity testing I used some colorful album art. Recognize any of them?

What is this?

A logical extension to real-time ASCII webcams (e.g. p2pvc) trained and served entirely in TypeScript.

Since there are ~2k emoji used for this project, the structure and color of each one would be prohibitively annoying to encode in a lookup table. Instead, it uses a tiny neural network to infer which emoji best represents any given 36x36 patch of pixels. Training involves mutating reference images of emoji (with libvips) and having the neural network learn to predict the original emoji index. It's kind of a hack, but it seems to work fairly well.

At serving time, the neural network is called with a batch of 1,728 patches (36x48) around 5-10 times per second.

Run

(Be sure to install bun)

$ git clone https://github.com/bwasti/emojicam.git
$ cd emojicam
$ bun add @shumai/shumai
$ bun serve.ts

Then open localhost:3000

Train from scratch

$ bun add @shumai/image
$ bun train.ts

Some stills