arakoodev
Augmented Large Language Models with Parametric Knowledge Guiding
high costs for most researchers and companies seeking to fine-tune these models for their specific use cases or domains. Moreover, users who can afford to fine-tune must provide their private data to the LLMs’ owner, thereby exposing it to potential risks such as misuse, breaches, or other security threats [BBC, 2023]. These limitations hinder the adaptability of LLMs to diverse scenarios and domains. A common approach to enhance LLMs is to leverage retrieval-based methods that access domainspecific knowledge from external sources [Liu, 2022; Shi et al., 2023; Peng et al., 2023a]. While these methods have shown promise, they face several challenges. First, they heavily rely on modern dual-stream dense retrieval models [Karpukhin et al., 2020] which suffer from shallow interaction between the query and candidate documents. Second, most dense retrieval models are based on smallscale pre-trained models such as BERT [Devlin et al., 2019] and therefore cannot take advantage of the world knowledge of large-scale pre-trained models. Third, retrieval models may struggle with complex knowledge that requires the integration of information from multiple sources or modalities. In this work, we propose the Parametric Knowledge Guiding (PKG) framework, which enables LLMs to access relevant information without modifying their parameters, by incorporating a trainable background knowledge generation module, as illustrated in Figure 1. Unlike retrieval-based methods, our PKG module utilizes open-source and free-to-use "white-box" language models, LLaMa-7B [Touvron et al., 2023], which encode implicit world knowledge from large-scale pre-training. The framework consists of two steps. First, we align the PKG module with the specific task or domain knowledge via instruction fine-tuning [Ouyang et al., 2022] to capture the necessary expertise. Second, for a given input, the PKG module generates the related knowledge, fed as extra context to the background-augmented prompting for LLMs. By supplying the nece
https://arxiv.org/pdf/2305.04757.pdf
