Questions about netrep

[aaprasad]

I’m trying to use your netrep repo for my own analyses but I’m getting stuck on some errors and I just wanted to make sure I’m implementing it correctly. Essentially, I’m trying to compare the network activations at certain layers between a trained resnet152 model and an untrained resnet152 model in keras. My current workflow is as follows:

Get the activation model at a stopped at a certain layer for both the untrained and trained models Show the activation models an image from ImageNet and store the activations in (n x w x h x d) tensors (U, T) Call conv_layers.convolve_metric(LinearMetric(),T,U) and get the resulting (w x h) matrix

My questions are:

1. Is the score output by Metric.score based on how close they are to each other or how far apart they are. IE if the score is higher does that mean the representations are close together or far apart?

2. Is what I described above, the correct workflow, or is there a better one you would recommend?

3. As a sanity check I called conv_layers.convolve_metric(LinearMetric(), T, T) assuming this should return a matrix of 0s right? but the resulting matrix has an mean value of 0.878 and that doesn't seem right.

4. What exactly do the barycenter.py, conv_layers.py, and rbf_sampler.py scripts do(or where can I find a description for them in your paper/elsewhere) and when should I use them?

5. Is the cka.py metric script fully implemented? I noticed that in the fit() function you just call pass, is that because you don’t need to align the tensors in the shape space? Also, the CKA you calculate seems to be just the angular distance between the covariance matrices which seems to be different from the CKA calculation described here no?

6. A similar question for the kernel.py script: The whole script is commented out, do I just need to uncomment it to use it or is there a reason its left uncommented

7. What is permutation.py calculating? Actually in general, could you point me to where the metrics calculated in each of these scripts is described so I can get a better understanding? In the README, you wrote that

   A forthcoming preprint will describe the various metrics in more detail

   do you have a copy of the preprint you might be able to share with me, or another resource I can use?

Thank you for taking the time to answer my questions, I know you must be busy.