Hydra pickle issue in generate_multi.py

[vienneraphael]

Figure out the pickle issue mentioned here: https://github.com/facebookresearch/fairseq/issues/5315

Conf file

conf.zip

Current workaround:

@hydra.main(config_path="conf", config_name="generate_multi_full")
def main(config: DictConfig) -> None:
    launcher = hydra.utils.instantiate(config.launcher)
    module = GenerateMultiModule(config)
    asyncio.run(module.run())
    # asyncio.run(tabulate(config))

and modify the run method of GenerateMultiModule class by prepending it with the following:

jobs = self.array()
        for iteration_value in jobs: