yaolinli
R_GAE execution
Which part of the code do you need most urgently: DeCo training, inference, ckpts or the interpretability tool R_GAE?
Originally posted by @yaolinli in #7
I need the code about the interpretability tool R_GAE. How could it be executed between any two layers?
I plan to clarify the R_GAE demo code around February 25. The R-GAE code initializes the matrix as an identity matrix, based on the intuition that each input token's relevance score is initially identical, and will be propagated across layers. To compute relevance between two layers, we can set the start layer (S) as the initial matrix and propagate it through to the end layer (E), executing between any two layers (S and E). For more details, you can refer to Appendix A.1 and A.2 in the paper. The R_GAE code is specific to the MLLM architecture, and we will release the version compatible with the LLaVA-v1.5 model.
