A weird thing about the gradient for each channel.

[hxngu]

I have printed out the gradient of feature map of one layer. However, I find a weird thing that the elements in each channel hold the same in all locations. I have written some codes to print out the gradient matrix of the layer4 of ResNet50. Basically speaking, the gradients of one channel in different locations should be different.

The result is shown below. I have tried some different inputs but it still doesn't work.

Looking forward to your reply.

[caihuaiguang]

I think it's because the use of GAP (gloabal average pooling layer) in resnet. The GAP will average each channel's feature map to a single value, here is an illustration (from https://arxiv.org/pdf/1512.04150) :

Then if we use the layer preceding the GAP layer as the target layer, the gradients of each element in the same channel will be same. How about using layer 3 or using other network without GAP (for example VGG-16)?

[hxngu]

Yes, you are right. The weird phenomenon results from the GAP layer of ResNet50. Thanks a lot.

[hxngu]

@caihuaiguang Hi, sry to disturb you again. There is another question. I find that shapleycam compute the hvp for activation map and it's efficient. I just wonder are there any efficient methods to compute the hessian(ignore channel interaction) explicitly? I have tried to slice the activation in one channel but I found I get a None value for output. The torch.autograd.grad only output the valid output when the inputs are the whole activations. I'm so confused about this phenomenon. Here are the codes I try to use. Is it because I break the computing graph when using slice method(I have tried to simply view the activation to other dimensions, it also output None.)?

[caihuaiguang]

@hxngu Hi, Sorry, I can't be of much help. Back when I implemented the HVP operation (even though it was just a few lines of code), it took me nearly two weeks. Any small modification could cause the HVP output to be None, and I was also quite confused. Even now, I haven't fully figured out the mechanism behind HVP.

I don’t recall "any efficient methods to compute the Hessian while ignoring channel interaction explicitly", and I’m not sure if it’s even possible. My only suggestion is to simplify the scenario as much as possible and keep debugging. If it keeps returning None, it might be best to just let it go...

[hxngu]

@caihuaiguang Thanks for your response. I find that if and only if the inputs are the original actiavtions, the torch.autograd.grad function will output valid value. It's so weird...