Guidance Needed on Continuing Training with a New Dataset via LoRA

[TanTaiLi]

I have recently completed training a model using LoRA (referred to as LoRA-1) with Dataset A. I am now considering how best to proceed with training on a new Dataset B.

My question is: Should I continue training directly on LoRA-1 with Dataset B, or would it be more effective to merge LoRA-1 back into the original model, create a new LoRA layer (LoRA-2), and then proceed with training on Dataset B using LoRA-2?

An additional consideration is the difference in data distribution between Dataset A and Dataset B. If the distributions are significantly different, how might this influence the decision on the best approach to take?