Feature suggestion: Look into using Jacobian representations for neural networks

[JulianFerry]

Hi, thanks for the repo!

I had a go at using the GradCam for a VGG16 network I am building using transfer learning, and I was not able to get any results. This led me to search for other techniques to analyse neural networks and I stumbled upon this paper: Full Jacobian Representation of Neural Networks.

I noticed you don't have any issues corresponding to this so I thought I might leave it here in case it is of interest to you. I only skimmed over it briefly but I saw the technique was referenced in a talk by Alex Graves. It seems to be similar to LIME, in the sense that it uses local approximations.

Here is a figure from the start of the paper:

Figure 1: Illustration of the full-Jacobian representation for a pre-trained VGG-16 network. The bias-Jacobian maps are summed across the channel dimension to produce single-channel maps. According to equation 1, aggregating these spatially gives us the neural network output f(x).

Feel free to close this issue if it's not something you would consider implementing.

[teabolt]

Hi @JulianFerry ,

Thanks for the link. Certainly seems like a great technique for visualizing decisions with more precision.

May I ask why did GradCam not work?

[JulianFerry]

The issue was that there was no highlighted area for about 95% of my pictures. The few that were highlighted did seem make sense however. I'm not quite sure about the reason... At this stage I suspect my model design as much as GradCam.

Things that I thought might affect it (I know nothing about GradCam):

• It's a binary classifier. My sigmoid output is often 1.0 or 0 which is surprisingly confident...
• I assumed that perhaps something was getting lost due to the depth of the network.
• I also created my dataset artificially using a texture pack and image augmentation techniques. My train and validation accuracies are 90% but maybe the design of my dataset is causing something strange - maybe the whole picture is being used in equal proportions ?.
• I had to convert from a tf.keras (TF2.0) model to keras using save_weights and recreating the model architecture with keras

The latter means I got a bit fed up since I also had to alternate between eager and graph mode execution to load my tfrecord datasets as PNGs (I know you are planning tensorflow integration eventually), so I didn't test too much. If I can get round to it I could get a reproducible example but I'm not sure how to design that... I could share the model generation code with my weights and some example images?

[JulianFerry]

Do you expect GradCam to display convincing results in the majority of cases?

[teabolt]

Interesting problem. Feel free to open a separate issue with the details if you wish to.

Unfortunately as far as I know this GradCam implementation hasn't been tested on many real world examples yet. The results may not make sense if the model is not accurate in itself. I can also suggest experimenting with the layer parameter.