Question about K2R and Seekr

[ArEnSc]

A friend and I read your paper on K2R and the read the seeker paper, we are still looking into Parlai platform we are a novices to all the machinery,making it difficult to determine the answer to this question. 1.We were wondering ... does the knowledge token get generated by the model upon asking a question? in the context? or 2.We append it after we detect a question or when we want a K2R response along side the knowledge context. We are thinking it is 2.

Thanks! @jaseweston

[klshuster]

Hi there, actually in the actual SeeKeR model, we didn't use a __knowledge__ token for generating the knowledge response (just the vanilla dialogue history). But, if we did, it would have been in the context. So, in that case, it would be 2. The same goes for generating the search query.

BlenderBot 3 is an extension of the SeeKeR and K2R models, and in that one we did use the __knowledge__ token to indicate that we required a knowledge generation; this, too, was appended after the context.

[ArEnSc]

@klshuster do you think that a pretrained model trained on dialog would be able to learn the K2R ability? with fine tuning? using these commands from Parlai? I want to give that a shot wondering if you guys tried the same. I am basically looking for the quickest way to fine tune, so I can build prompt tuning modules for these abilities https://github.com/facebookresearch/ParlAI/issues/4740#event-7452750456

[klshuster]

Yes, definitely. We also offer non-dialogue pre-trained models (e.g., the R2C2 model, which is essentially a 3B parameter BART-style pre-trained model).

[github-actions[bot]]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.