Apply annotation vector to both A and B timeseries if we're not using a self-join

[aouyang1]

Currently the annotation vector is only applied to the B timeseries and has assumed that we're doing a self-join. We should extend this to apply the same annotation vector to the A series as well.