Not learning different modalities?

[jtrammell-dla]

My experiments using this code so far (using the settings suggested in the readme) seem to be learning, and producing good classification and localization results, but the separate output layers are not learning different modalities as suggested by the paper. All 8 layers appear to be more or less identical to each other. Am I missing something?