A question about training. Only use one of four branches to update the network?

[zhangjunwang]

If we have a batch of 20 images, 10 of high-level-commands are Right and 10 of them are Straight, does this mean the Left and Follow branch heads are not updated at all for this batch? I mean we only use one of four branches to update every time, and do not care about other three that high-level-commands do not choose.