instruction fine tuning (for text generation)

[almugabo]

Hi, I am new to happy-transformer and impressed how it makes things much much easier. I have a - perhaps naive - question. How would one go about to train it with "instruction" ? I mean for example with instruction like the ones in FLAN ?

Has anyone a working example she/he can share or can give me some hints ?

[ted537]

Reading through this brief article, it appears that FLAN's main idea is transforming very formal training data (such as entailment/premise pairs) into something more natural (1-2 paragraphs of text), such that the model better responds to prompts written by humans (in a variety of formats). Do you have a database of FLAN-compatible data in mind?

[almugabo]

thanks Ted. actually what I have in mind is, I think. much simpler than what they did in Flan. Basically I would like to train a small generative model (like a small GPT neo ), to perform information extraction tasks.

Here an example of a simple task (its self trivial but I think it shows the idea):

{'instruction': 'find the thirteenth word in this sentence', 'context': "Our friends won't buy this analysis, let alone the next 
one we propose", 'answer': 'propose'}
{'instruction': 'find the second word in this sentence', 'context': "One more pseudo generalization and I'm giving up", 
'answer': 'more'}
{'instruction': 'find the fifth word in this sentence', 'context': "One more pseudo generalization or I'm giving up", 'answer': 
'or'}
{'instruction': 'find the second word in this sentence', 'context': 'The more we study verbs, the crazier they get', 'answer': 
'more'}
{'instruction': 'find the seventh word in this sentence', 'context': 'Day by day the facts are getting murkier', 'answer': 
'getting'}

Not sure the best way to provide such examples . I can think of providing them one sentence at a time (line by line) like this :

    find the seventh word in this sentence <|start context|>Day by day the facts are getting murkier<|end 
    context|>getting<|end generation|>

or each element line by line

  find the seventh word in this sentence
  Day by day the facts are getting murkier
  getting<|end generation|>