Different test results of EDR and LCSS

[C-Harlin]

Hi, thanks for your help last time, and now I am getting to the point of testing baseline methods. My test data was generated accroding to your paper (and I also compared my code with yours, which is basicly same). I also use the code of authors of EDwP to implement baseline methods.

The mean rank of EDwP is similar to yours, which convince me the test data is fine. However, when it comes to EDR and LCSS, the results differ a lot, eg. when db size is 60k, the mean rank of LCSS is 2.109, EDR is 0.013, which is quite different from yours.

I guess the problem might due to the threshold setting. I set the threshold of EDR and LCSS to a quarter of the maximum standard deviation between the two trajectories that were examined, which is the suggestion from the authors of EDR. (To make it clear, lets assume the two trajectories are named traj_a and traj_b, then calculate the std on the two dimensions of both traj_a and traj_b, which yield four candidates. And choose a quarter of the maximum std as the threshold.)

I noticed that in the paper you said you adopted the the strategies described in LCSS and EDR. So I am wondering the detail of your implements of EDR and LCSS.

I'll be appreciated for your help! Thanks!

[C-Harlin]

As bellow shows, the results of EDR and LCSS are quite different:

distort rate(dbsize:100k)	0.2	0.4	0.6
lcss	5.434	7.14	10.032
edr	0.039	64.696	85.578
edwp	24.503	21.008	19.551

downsample rate (dbsize:100k)	0.2	0.4	0.6
lcss	21.529	49.509	194.857
edr	28.507	109.29	660.236
edwp	28.038	40.208	131.759

	60k	100k
lcss	2.109	3.518
edr	0.013	0.02
edwp	15.323	25.69

And my pseudo code of calculating EDR is:

for(int i=0;i<1000;i++){
    get traj_a;
    max_std_query = Math.max(calculateSD(traj_a[0]),calculateSD(traj_a[1]));
    for(int j=0;j<dbsize;j++){
        get traj_b;
        max_std_db = Math.max(calculateSD(traj_b[0]),calculateSD(traj_b[1]));
        max_std = Math.max(max_std_query, max_std_db);
        TrajectoryDistance d=new EDR(max_std/4);
        dist[j] = d.getDistance(traj_a,traj_b)[0];
    }

Is this the way of your evaluation? Do you have any idea about it? Thx

[boathit]

I am a little busy and will take time looking into it once available.

[C-Harlin]

Hi again, I'm still confused about this problem for months. Your work is great but that is the only thing I can not figure out. I'm wondering if you can provide your code for testing baseline methods? That would be a great help.

[boathit]

Hi @C-Harlin, my baseline methods were originally held in the school's sever and are not accessible since I have graduated. But I remember that the threshold of LCSS and EDR is a fixed value in my experiments. So I think you are actually doing in a correct way.

[C-Harlin]

That is the point where both EDR and LCSS didn't explain clearly. I think choosing a threshold for the two trajectories under measurement is much more reasonable than setting a fixed value for all trajectories to be measured. Thanks for your help. Best regard!

[LIWEIDENG0830]

Hi, @C-Harlin, do you leverage the released model by authors to test the results? I follow the t2vec.ipynb and load the weights of the released model. However, I cannot reproduce the results reported in the paper by a large margin.

[C-Harlin]

Hi, @C-Harlin, do you leverage the released model by authors to test the results? I follow the t2vec.ipynb and load the weights of the released model. However, I cannot reproduce the results reported in the paper by a large margin.

Hi @LIWEIDENG0830 , I can hardly recall the detail since it has been such long time. But I remember I had trained my own weights, maybe you can have a try.

[LIWEIDENG0830]

Hi, @C-Harlin, do you leverage the released model by authors to test the results? I follow the t2vec.ipynb and load the weights of the released model. However, I cannot reproduce the results reported in the paper by a large margin.

Hi @LIWEIDENG0830 , I can hardly recall the detail since it has been such long time. But I remember I had trained my own weights, maybe you can have a try.

Thanks, @C-Harlin. I solve this problem by training my own weights. Maybe the weights released by the authors is wrong.