Were examples in each batch sampled from the same dataset during M3's training?

[umarbutler]

Hi @staoxiao @hanhainebula, Your paper doesn't seem to mention whether you sampled examples in each batch from the dataset or whether you allowed each batch to contain samples from different datasets. Would you mind sharing what you went with? Also, would you advise going one way or the other in the future?