lucid
lucid copied to clipboard
tensorflow
Research: Scale Activation Factorization
🔬 This is an experiment in doing radically open research. I plan to post all my work on this openly as I do it, tracking it in this issue. I'd love for people to comment, or better yet collaborate! See more.
Please be respectful of the fact that this is unpublished research and that people involved in this are putting themselves in an unusually vulnerable position. Please treat it as you would unpublished work described in a seminar or by a colleague.
Description
In Building Blocks, we create neuron groups by applying Non-Negative Matrix Factorization to the activations of a single example. This can be seen as creating a local basis for explaining that single example.
A natural next step would be to factor activations from many examples. The blocker on this has been figuring out how to scale NMF factorization. I think it's likely that reading documentation for a few hours (especially about mini batch factorization approaches) would give us enough to solve this.
An intermediate approach might be to factor the activation for examples of a single class or set of similar classes.
This sounds incredibly interesting! Had a quick question: by doing an NMF on the activations (which I'm assuming would be performed per layer), it seems like we would extract some representation of the activations on that layer, where these representations ideally have some semantic meaning. These representations, however, likely wouldn't correspond to any neuron in the layer.
My question is then, is the goal of this to uncover the semantic representations to see how the network responds to different inputs? Or would this be used to in some way retrain a new network whose weights are initialized to these extracted representations?
My apologies if this is not the right forum to ask.
Is the plan for this work to also handle activation values from networks other than InceptionV1?
I am interested in visualizing how other networks like MobileNet, SqueezeNet (and beyond) see a given example. But I run into a difficulty with the NMF; specifically the activation values must be non-negative which is not always true with other model architectures. A work-around is groups=0 (in the Activation Grids example), but this looses the "decoherence" feature.
@mn02: Happy to talk about this, but I think it would fit better if you made a separate issue or a slack conversation on the Distill slack. This issue is for a specific research question. Thanks!
A first step would be to utilize acceleration hardware instead of numpy (which is used in sklearn's implementation of NMF), see e.g. here for a PyTorch implementation https://github.com/yoyololicon/pytorch-NMF. Their implementation seems to get at least factor 5 speed-up for me on colab (for a random NN 1000x1000 matrix).
A numpy documentation of minibatch NMF (with code in numpy) can be found here https://rserizel.github.io/minibatchNMF/intro.html, if anyone wants to grab it.
