Add a tag for marking the source of factual answers

[finitearth]

In case the user asks a textual question (I. E. "When was the ISS launched?") it would be valuable if the model would not only respond with the correct answer ("November 1998") but also with the source ("November 1998. I found this information on wikipedia: [LINK]").

If we are planning to give the model the ability to "search" a database, I. E. By providing relevant information into the input sequence, it would be cool if the model would learn to reference it.

As the titling of the source may change depending on implementation and is not known during data collection, it would be nice to keep it variable at first.

Also if people agree this is a good idea, we should put it into the guidelines, that if a factual question is asked, the labeler playing the role of the assistant, should tend to provide sources, if applicable.

[bitplane]

I thought the model doesn't really have that sort of information, as it's generating text based on weights and isn't really aware of where the data comes from.

What might be more possible is have a separate system that extracts claims made in some text using the model, and put them into a format where they can be checked. Look them up and check whether they agree or not. If they do then add inline citations. If they don't, it could mark them as unfounded. When the output came from the model it could flag query and output for human review.

That would be a pretty useful tool for evaluating all kinds of text IMO, not just from the model but anywhere a claim is ever made about anything at all.

[finitearth]

I thought the model doesn't really have that sort of information, as it's generating text based on weights and isn't really aware of where the data comes from.

What might be more possible is have a separate system that extracts claims made in some text using the model, and put them into a format where they can be checked. Look them up and check whether they agree or not. If they do then add inline citations. If they don't, it could mark them as unfounded. When the output came from the model it could flag query and output for human review.

That would be a pretty useful tool for evaluating all kinds of text IMO, not just from the model but anywhere a claim is ever made about anything at all.

In my understanding it was planned in this project to paste relevant information from documents (possibly ranked by an rl agent or cosine similarity in embedding by sbert or mirrorbert) into the input context. Essentially enabling "browsing"

[bitplane]

@finitearth I did not know that. That's pretty cool. Admittedly I'm a bit of a noob when it comes to the ML stuff!

[huu4ontocord]

Yes - we will eventually be able to do search and retreival. Citations will be something we want to do. Are you proposing some sort of tag being output like a [citation] tag? Is this more algorithm code done at inference? If so, this could be considered a UI, right?

[finitearth]

Yes - we will eventually be able to do search and retreival. Citations will be something we want to do. Are you proposing some sort of tag being output like a [citation] tag? Is this more algorithm code done at inference? If so, this could be considered a UI, right?

My first idea was indeed that. However after thinking more about it, I think this could run into an issue: What if there where multiple relevant documents? How would we know which one the model "wants" to cite?

I would suggest the following during Data Collection and Training:

1. If a factual answer is cited, we need a referenced paragraph with a title indicating the source (i.e. "wikipedia.org: International Space Station").
2. Prepend the referenced paragraph with title to the input of the model.
3. Reference that title in the answer of the model (i. e. "I found this information on wikipedia: [LINK]")

--> This way the model should learn to cite the references with title and source by itself. To make it easier for the labelers and inorder to reduce the labor, we could just have something like a "modified" citation tag, that we would just need to replace adequately before feeding to the model.

As an example (paragraphs ofc would be longer in the real world data): [SOURCE="wikipedia.org: International Space Station" PARAGRAPH="The first ISS component was launched in 1998"]

->

wikipedia.org: International Space Station
The first ISS component was launched in 1998

During inference we would just have to prepend the relevant documents with the same way of titling.