How to use the past, future and self targets for text generation?

[AhtiAhde]

What is your question?

I noticed that new FairSeq models support past, future and self targets for text generation tasks. However, I have been trying to figure out a way of implementing a generator that would allow me to do it, but it seems that it is really hard to get that to work. The issue is mostly in the fact that it is not entirely clear, which models actually support it. The default model handler sets model.supported_targets as ['future'], which mean we only have access to the old default mode, but also many models do not seem to support the feature (I have spent few days now trying to discover right combination of things to do in order to enable this).

The reason I am enthusiastic about this feature is that I have developed a writing theory based prompting system for AI generated fiction writing, where I am able to generate past-(target)-theme, future-(target)-themes for scenes in a story and character-action prompts that I could use with self-target. I would love to build a model based on this as my entry for the NaNoGenMo AI writing competition by November of this year; so I would be willing to do some contributions if I get some guidance.

I have studied this problem (of dramatic arc generation with AI) since 2016 and I strongly believe that this kind of semi-guided method where author / automation (based on theoretical framework used by real authors of fiction) provides dramatic structure and machine just bridges the gap, would be most interesting as it might actually work. Unfortunately traditional language models have lacked bidirectional causality and self-reflection due to unidirectional prompting, while this method would depend on exactly those features. I have already built an early prototype for the prompt generator.

So my questions would be:

1. Which pretrained models support past, future and self targets for language-modeling tasks?
2. Would it be sufficient if I just change the default handlers (of supported models) so that their model.supported_targets would be ['past', 'self', 'future']? If not, where could I get proper guidance for doing rest of the coding (I am professional data engineer / scientist with lot of experience on NLP; I would not need that much help)?
3. If I would like to contribute a generator pipeline for this "more causal" generation task, how would we conduct such work?

Regarding the pretrained models, I would prefer some simple smallish thingie that I can run with single Quadro T1000 (4GB), though I do have occasional free access to bigger machines too.

[gmryu]

Sorry I have no knowledge in this field. All I find out is In short: using any lm model, change its supported_target method(i.e.) to return {"future", "past", "self"}, a LanguageModelingTask, add --past-target --future-target --self-target to command line may do your job.

More specifically:

1. models: FairseqLanguageModel, BARTModel, RobertaModel have the supported_targets. You can add "past", "self" and "future" as you wished.
2. tasks: LanguageModelingTask , MultilingualLanguageModelingTask utilize those.
   • if you search targets in these two .py, you will know self.targets are determined by --past-target, --self-target, --future-target. See setup_task
   • also, you will find dataset is built with self.targets
3. dataset: MonolingualDataset's _getitem_,_make_source_target has a comment talking about future,self,past targets. Check it out.

Those are all I can find now.

So I do not know about how past-(target)-theme or futrue-(target)-theme should be handled. For MonolingualDataset's comment do you find it makes any sense? If it does, then everything should be ready to go.

fairseq-generate does task.load_dataset. However, fairseq-interactive uses task.build_dataset_for_inference, which does not envoke MonolingualDataset so interactive may not work.

Hope your succeess.

--

For contributing, I guess you have to @ somebody?

[AhtiAhde]

Cool, thanks for the swift reply!

I will give it a try and see what comes out of it. I think I would also need to make some modifications to the beam decoder and make it bi-directional. I think this paper has the same idea: https://arxiv.org/pdf/1906.09601.pdf

Do you know if there is some existing decoder that already delivers the same idea (but instead for translation does that for language modeling task)?

[AhtiAhde]

Here to share some first impressions.

I decided to try just remove the check that throws the "Unsupported language modeling target" exception, mostly because models do not have setattr implemented and thus supported_targets is read-only. As a model I used the Fariseq Pre-trained Adaptive LM Wiki 103 v2 model (found it from the documentation, from the pre-trained models part); this model only had future target by default.

With this setup setting the past, self or future targets had not impact to the end result, it was all deterministic.

How could I check if there exists any models, which have more than one target? Somewhere it said that all bidirectional models ought to have both and self targets. By default causal LM's are left-to-right future target models and typically bidirectional models are not used with prompting, since their prompting capabilities was just discovered in 2021. I believe those white papers have not lead to any practical implementations in the frameworks we use these days?

I might try if BART and RoBerta are different than the fairseq models, but I might just give up if those are dead ends and instead try to solve my problem through some other route.

Oh wait, seems like the bidirectional prompting paper is from April 2022 (the authors names have been wiped due to known bug in Latex, which replaces the author names with "Anonymous ACL"; that will probably be fixed soon; Google's T5 has also been implemented with prompting): https://openreview.net/forum?id=oH-ePEJ57Nz

Maybe I need to wait one more year :)