demo 4 trained on DTD not working

[kcyu1993]

Hi @aimerykong

Thank you for sharing you code. I tried your demo 4 locally, switching the CUB dataset to DTD to see the performance. However, the objective is not decreasing after many epochs.

What I did is simply add these lines to your main100 and main101 code.

% main100_demo_PCA_conv53_forDimRed.m
%% prepare data
% dataset: 'CUB', 'MIT', 'DTD', 'aircrafts', 'cars'
if strcmp(dataset, 'CUB')
    num_classes = 200;
    imdbFile = fullfile('imdbFolder', dataset, [lower(dataset) '-seed-01'], 'imdb-seed-1.mat');
elseif strcmp(dataset, 'DTD')
    num_classes = 47;
    imdbFile = fullfile('imdbFolder', dataset, [lower(dataset) '-seed-01'], 'imdb-seed-1.mat');
end
% ... omitting lines between the dataset and the last save line.
save([dataset,'_bisvmEpoch102'], 'Gram', 'U', 's', 'S');

% main101_...
if strcmp(dataset, 'CUB')
    % omit since it is the same as your code.
elseif strcmp(dataset, 'DTD')
    num_classes = 47;
    dataDir = './data/dtd';
    imdbFile = fullfile('imdbFolder', dataset, [lower(dataset) '-seed-01'], 'imdb-seed-1.mat');
%     imdbFile = fullfile('imdbFolder', dataset, [lower(dataset) '-seed-01'], 'imdb-seed-2_flipAug.mat');
    if ~exist(imdbFile, 'file')        
        imdb = dtd_get_database(dataDir);
        
        imdbDir=fileparts(imdbFile);
        if ~isdir(imdbDir)
            mkdir(imdbDir);
        end
        save(imdbFile, '-struct', 'imdb') ;
    end

% omit lines 
% make sure I was loading the init-weights saved by main100.
Pparam = load([dataset,'_bisvmEpoch102']);

Everything else is not touched. And this is the learning curve returned by Matlab.

Would you mind to share how to correctly use your code to train on other datasets? Thank you in advance!

[aimerykong]

Hi, Thanks for trying this out. This happens at my end also. The reason I guess is that updating the big kernel from PCA is non-trivial. I still don't have a reasonable way to handle this and believe this indicates more things like learning rate for different layers. But a workaround is to stack the low-rank bilinear layer on a well-train backend on that dataset, using either cross-entropy loss over the CNN feature maps or bilinear features.

[kcyu1993]

Hi,

Thanks for replying me! You suggest that I should firstly fine-tune the network with conventional FC layer with k-way softmax and categorical cross entropy loss and then use the fine-tuned model to train LRBP.

I am a bit curious, since I checked your paper before finding this repo, and we are currently comparing models on texture recognition datasets. You have reported the accuracy on DTD dataset, so I assume there must be a way to train it. Is this due to you are using another training pipeline to train the bilinear SVM, if possible could you also provide such code?

Many thanks!

[aimerykong]

Hi, For DTD dataset, I trained the models progressively -- training the straightforward baseline with cross-entropy loss, then fine-tuning for bilinear CNN, and finally fine-tuning for LRBP. As you are running matlab, the first step is to stack the cross-entropy loss (softmaxlog) on top fc7 (with relu), then remove fc layers and only keep backend till relu5_3, add the bilinear layer. This seems to be a little overhead, but I could not get the neat scripts at my end now:-(

[kcyu1993]

sure no problem, I found the obstacle hurts the training, there are few lines that are hard-coded for CUB dataset, fixing it resolves the problem.