Potential of the classifier

[tom99763]

Hi, your work is very interesiting

I do a simple test on mnist

I think this loss has potential to extend to hard recognition task, do you plain to do that?

[mingukkang]

It is very intesresting:)

When I developed D2D-CE loss, I tested D2D-CE loss for a fine-grained image retrival task using CUB200 dataset.

I remenber that D2D-CE outperformed one of the cutting-edge metric learning losses: Proxy Anchor loss in some cases.

So, I think D2D-CE is very promissing to solve hard recognition task.

[tom99763]

I use your optimal value formulation in the paper I found the distribution of optimal loss shwon in below

I think the statistic method is an potential tool to fast tune the hyperparameters , distribution of optimal loss is very close to normal distribution, some good properties can be applied to search the parameters

Code :

`def optimal_sampling_distribution(nclass=10,batch_size=64,nsamples=10000):

samples=[]

for i in range(nsamples):

    #Note : maybe sample using class distribuion from training data will be better 
    x=tf.experimental.numpy.random.randint(0,nclass,size=batch_size)

    x=tf.cast(x,'float32')

    optimal_loss=tf.reduce_mean(tf.math.log(1+x))

    samples.append(optimal_loss)

return tf.concat(samples,axis=0)`

[tom99763]

Here is my class distribuion of training dataset :

And I sample batches from multinomial distribuion where the parameters is based on the class distribuion and use the optimal loss formulation, I get the distribuion of optimal loss below

I found during training, loss fast coverge to this distribution under my parameter settings, but it keep making its impact on the optimization

[mingukkang]

Oh nice work!

Can you explain the sentence that "I use your optimal value formulation in the paper I found the distribution of optimal loss shwon in below" more specifically ?

I don't know what that sentence means.

Thank you.

[tom99763]

Oh nice work!

Can you explain the sentence that "I use your optimal value formulation in the paper I found the distribution of optimal loss shwon in below" more specifically ?

I don't know what that sentence means.

Thank you.

The equation A6 in your paper, when model is fitted perfectly on the training set, the D2Dloss is only related to number of flase negative samples, that's the optimal loss

[mingukkang]

I've been a little busy these days, so I'm sorry for the late reply T.T

In my paper, the optimal loss of D2D-CE is as follows:

where, N(i) is the set of indices that point locations of the negative samples whose labels are different from the reference label vyi in the mini-batch. So, |N(i)| is the number of instances which have the same class label with the reference sample i.

However, when I interpret your optimal loss implementation optimal_loss=tf.reduce_mean(tf.math.log(1+x)), I am very confused because you input label itself instead of |N(i)|.

Could you explain the reason for this?

Thank you.

[tom99763]

Here is my updated notebook https://www.kaggle.com/code/tom99763/sampling-distribuion-of-optimal-loss

The way I compute this optimal loss is:

Compute prior class probrability p1,...pC based on the training data Iteration 50000 times: 1. sample a batch of label y1,....yN~Multinomial(n=1, p1,...pC) //1 trail and C classes 2. compute optimal loss based on y1,....yN

By continous sampling statistics, we get the sampling distribuion of the optimal loss

I've been a little busy these days, so I'm sorry for the late reply T.T

In my paper, the optimal loss of D2D-CE is as follows:

where, N(i) is the set of indices that point locations of the negative samples whose labels are different from the reference label vyi in the mini-batch. So, |N(i)| is the number of instances which have the same class label with the reference sample i.

However, when I interpret your optimal loss implementation optimal_loss=tf.reduce_mean(tf.math.log(1+x)), I am very confused because you input label itself instead of |N(i)|.

Could you explain the reason for this?

Thank you.

[tom99763]

I'm going to publish my paper, and I use your idea in my paper but I do image-to-image translation task

The amazing thing is when my model converges the loss distribution is like the above