choasma
fixes to hsic.py
Ok, this is the correct branch to create pull request -- same coment as the one i closed. this one has only one file.
Hi Kurt, I used 3 functions from hsic and noticed a few things
you can allocate much less memory by using in place functions, one shouldnt do that in the model but in the loss function that is ok, because loss is the source of of the gradient it simply copies the value to the grad.
for some reason your kernelmat you cast a new tensor in the -e^(d/v), i removed that - so now if X device is cuda, the device doesnt die. Also, recasting a Float Tensor introduces floating point errors, not significant in this context but i dont know how they would propagate.
I added a linting mute so my vscode doesnt flag torch. or as wrong
If you want look at the reasoning, tests and timing, I have a jupyter in my dev branch where I tested the changes.
https://github.com/xvdp/HSIC-bottleneck/blob/xvdp_dev/jupyter/HSIC_Kernel.ipynb
I only changed the 3 functions that i was using
distmat()
kernelmat()
hsic_normalized_cca()
oh T-H-A-N-K-S! I'll spend days after work to test your code. I'll confirm with you shortly.
no worry. I only touched 3 functions - that I needed. if you really want to speed it up you could write it in cuda and save yourself allocation of all those eye matrices, and since much of the code results in symmetric matrices, probably need half or even less computation. alternatively, I just saw a sparse cuda matrix library - https://github.com/huggingface/pytorch_block_sparse last but not least as a general speed optimization with pytorch, therse a nice tutorial from nvidia, https://www.youtube.com/watch?v=9mS1fIYj1So but thats a general reference. Im more interested in the idea that hsic() or another type of information measure could replace end to end backprop. - OR be used in combination with... thanks for the work.
On Tue, Sep 8, 2020 at 7:14 PM Leopad [email protected] wrote:
oh T-H-A-N-K-S! I'll spend days after work to test your code. I'll confirm with you shortly.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/choasma/HSIC-bottleneck/pull/6#issuecomment-689255259, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJYHPOWML7CG5XYD5UKSILLSE3QJXANCNFSM4Q42PU2A .