SoftTeacher
SoftTeacher copied to clipboard
microsoft
End of training Sample Ratio decay
The paper describes a gradual decay of the sample ratio at the end of training.
Partially labeled data: sample_ratio=[1, 4]
the data sampling ratio sr is set to 0.2 and gradually decreases to 0 over the last 10k iterations.
Fully labeled data: sample_ratio=[1, 1]
the data sampling ratio sr is set to 0.5 and gradually decreases to 0 in the last 20k iterations.
There are some commented out lines in semi_sampler.py which look similar. They suggest updates to the ratio are accomplished by weighting. Does this mean changing the weighting of the losses?
How are the 10k and 20k iteration decay points set?
Does the sample ratio decreasing to zero mean ending up with only unsupervised samples at the end?
The weight_adjust.py hook looks it could be used to adjust the unsup_weight, at iterations, but this could only be used to reduce the loss weight of unsupervised loss to zero as there don't appear to be provisions in soft_teacher.py to change the supervised loss weight.
A hook like this would step down the unsupervised loss:
dict(type="Weighter", steps=[170000, 172500, 175000, 177500], vals=[4, 3, 2, 1, 0], name="unsup_weight")
Thanks!
Tested the Weighter hook.
dict(type="Weighter", steps=[170000, 172500, 175000, 177500], vals=[4, 3, 2, 1, 0], name="unsup_weight")
It worked as expected.
A potential issue is the learning rate step down doesn't account for less contributions from the unsupervised images in each batch effectively increasing the learning rate on the remaining loss level.
E.g. at the end there is no loss retained from the unsupervised images so the entire learning rate is applied to the single supervised image.
Another issue is the number supervised images at the end of training is effectively decreased relative to what is described in the paper unless the number of iterations is increased. (when using loss weighting rather than adjusting the sample ratio)