how to do doc cache if there are multiple documents in rag?

[SimonYx42]

trafficstars

generally, we will have several docs for reference while doing rag, wondering if you guys have tested this setting with doc cache?

[SimonYx42]

and also, is it possible to share the rag code and setting? really appreciate it!

[Muennighoff]

Hmm we havn't tested it, but I think it could work; Similar to Query-Doc / Doc-Query, you can just concatenate the key-value states; The problem will be the attention mismatch, but still worth trying I think!

Sure all the RAG code is in https://github.com/ContextualAI/gritlm/tree/main/rag