Question about ground truth boxes which are used in training and PrototypicalCalibrationBlock

[leesb7426]

I found some errors in your code. In meta_voc.py, when we got more objects of a class than a shot(K), the K samples of the class are randomly selected among objects. However, the seed for sampling is not fixed. As a result, the sampled objects for training and the sampled objects for PCB may not be the same.

When I set a seed to extract the same samples, the performances were dramatically degraded in 2-shot and 3-shot. Avg performance: Performances of VOC(split-1) with original code. | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | --> 1-shot | 30.14 | 51.97 | 31.07 | 30.14 | 51.97 | 31.07 | --> 2-shot | 31.80 | 53.00 | 34.33 | 31.80 | 53.00 | 34.33 | --> 3-shot | 36.79 | 59.16 | 39.64 | 36.79 | 59.16 | 39.64 | --> 5-shot | 36.85 | 61.53 | 40.12 | 36.85 | 61.53 | 40.12 | --> 10-shot | 37.77 | 59.47 | 41.40 | 37.77 | 59.47 | 41.40 |

Performances of VOC(split-1) with fixing seed for sampling. | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | --> 1-shot | 30.13 | 53.46 | 29.86 | 30.13 | 53.46 | 29.86 | --> 2-shot | 29.09 | 49.41 | 31.12 | 29.09 | 49.41 | 31.12 | --> 3-shot | 30.76 | 50.77 | 33.00 | 30.76 | 50.77 | 33.00 | --> 5-shot | 35.87 | 60.16 | 38.98 | 35.87 | 60.16 | 38.98 | --> 10-shot | 38.04 | 60.19 | 41.74 | 38.04 | 60.19 | 41.74 |

All performances: Performances of VOC(split-1) with original code. --> 1-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 30.21 | 53.89 | 30.09 | 30.21 | 53.89 | 30.09 | \n| 1 | 29.97 | 50.93 | 31.75 | 29.97 | 50.93 | 31.75 | \n| 2 | 29.98 | 52.97 | 30.34 | 29.98 | 52.97 | 30.34 | \n| 3 | 29.69 | 50.40 | 30.14 | 29.69 | 50.40 | 30.14 | \n| 4 | 30.84 | 53.84 | 31.65 | 30.84 | 53.84 | 31.65 | \n| 5 | 30.22 | 51.04 | 32.58 | 30.22 | 51.04 | 32.58 | \n| 6 | 28.80 | 49.20 | 29.87 | 28.80 | 49.20 | 29.87 | \n| 7 | 30.62 | 52.85 | 31.61 | 30.62 | 52.85 | 31.61 | \n| 8 | 31.46 | 54.46 | 31.72 | 31.46 | 54.46 | 31.72 | \n| 9 | 29.58 | 50.07 | 30.96 | 29.58 | 50.07 | 30.96 | \n| \xce\xbc | 30.14 | 51.97 | 31.07 | 30.14 | 51.97 | 31.07 | \n| c | 0.43 | 1.09 | 0.54 | 0.43 | 1.09 | 0.54 |'

--> 2-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 31.00 | 50.91 | 34.27 | 31.00 | 50.91 | 34.27 | \n| 1 | 33.64 | 54.77 | 36.80 | 33.64 | 54.77 | 36.80 | \n| 2 | 29.67 | 51.01 | 30.59 | 29.67 | 51.01 | 30.59 | \n| 3 | 33.10 | 55.59 | 34.85 | 33.10 | 55.59 | 34.85 | \n| 4 | 29.30 | 50.05 | 31.24 | 29.30 | 50.05 | 31.24 | \n| 5 | 33.53 | 54.35 | 37.70 | 33.53 | 54.35 | 37.70 | \n| 6 | 33.85 | 56.59 | 36.54 | 33.85 | 56.59 | 36.54 | \n| 7 | 31.73 | 52.02 | 34.55 | 31.73 | 52.02 | 34.55 | \n| 8 | 30.56 | 51.27 | 32.35 | 30.56 | 51.27 | 32.35 | \n| 9 | 31.61 | 53.47 | 34.46 | 31.61 | 53.47 | 34.46 | \n| \xce\xbc | 31.80 | 53.00 | 34.33 | 31.80 | 53.00 | 34.33 | \n| c | 0.99 | 1.33 | 1.39 | 0.99 | 1.33 | 1.39 |'

--> 3-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 36.01 | 57.80 | 39.41 | 36.01 | 57.80 | 39.41 | \n| 1 | 36.86 | 59.94 | 39.32 | 36.86 | 59.94 | 39.32 | \n| 2 | 37.61 | 60.41 | 40.12 | 37.61 | 60.41 | 40.12 | \n| 3 | 36.32 | 58.89 | 38.80 | 36.32 | 58.89 | 38.80 | \n| 4 | 35.89 | 57.38 | 39.11 | 35.89 | 57.38 | 39.11 | \n| 5 | 37.37 | 59.26 | 40.41 | 37.37 | 59.26 | 40.41 | \n| 6 | 36.53 | 58.73 | 38.78 | 36.53 | 58.73 | 38.78 | \n| 7 | 35.94 | 58.29 | 39.30 | 35.94 | 58.29 | 39.30 | \n| 8 | 37.95 | 60.48 | 40.80 | 37.95 | 60.48 | 40.80 | \n| 9 | 37.39 | 60.38 | 40.37 | 37.39 | 60.38 | 40.37 | \n| \xce\xbc | 36.79 | 59.16 | 39.64 | 36.79 | 59.16 | 39.64 | \n| c | 0.44 | 0.66 | 0.43 | 0.44 | 0.66 | 0.43 |'

--> 5-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 36.25 | 60.70 | 39.63 | 36.25 | 60.70 | 39.63 | \n| 1 | 37.34 | 61.82 | 40.60 | 37.34 | 61.82 | 40.60 | \n| 2 | 37.53 | 62.76 | 40.59 | 37.53 | 62.76 | 40.59 | \n| 3 | 36.82 | 61.46 | 39.94 | 36.82 | 61.46 | 39.94 | \n| 4 | 36.15 | 60.92 | 39.71 | 36.15 | 60.92 | 39.71 | \n| 5 | 35.80 | 59.40 | 39.25 | 35.80 | 59.40 | 39.25 | \n| 6 | 36.34 | 60.42 | 39.54 | 36.34 | 60.42 | 39.54 | \n| 7 | 37.33 | 62.64 | 40.73 | 37.33 | 62.64 | 40.73 | \n| 8 | 37.63 | 62.97 | 41.25 | 37.63 | 62.97 | 41.25 | \n| 9 | 37.33 | 62.18 | 39.96 | 37.33 | 62.18 | 39.96 | \n| \xce\xbc | 36.85 | 61.53 | 40.12 | 36.85 | 61.53 | 40.12 | \n| c | 0.39 | 0.68 | 0.37 | 0.39 | 0.68 | 0.37 |'

--> 10-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 38.03 | 59.83 | 42.59 | 38.03 | 59.83 | 42.59 | \n| 1 | 35.99 | 55.93 | 40.42 | 35.99 | 55.93 | 40.42 | \n| 2 | 37.99 | 59.70 | 41.05 | 37.99 | 59.70 | 41.05 | \n| 3 | 37.93 | 59.53 | 41.94 | 37.93 | 59.53 | 41.94 | \n| 4 | 37.91 | 60.19 | 41.63 | 37.91 | 60.19 | 41.63 | \n| 5 | 37.68 | 58.66 | 41.22 | 37.68 | 58.66 | 41.22 | \n| 6 | 37.28 | 58.61 | 40.34 | 37.28 | 58.61 | 40.34 | \n| 7 | 38.28 | 60.35 | 40.89 | 38.28 | 60.35 | 40.89 | \n| 8 | 38.25 | 61.55 | 41.90 | 38.25 | 61.55 | 41.90 | \n| 9 | 38.34 | 60.39 | 41.99 | 38.34 | 60.39 | 41.99 | \n| \xce\xbc | 37.77 | 59.47 | 41.40 | 37.77 | 59.47 | 41.40 | \n| c | 0.41 | 0.89 | 0.43 | 0.41 | 0.89 | 0.43 |'

Performances of VOC(split-1) with fixing seed for sampling. --> 1-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 30.15 | 53.19 | 30.08 | 30.15 | 53.19 | 30.08 | \n| 1 | 29.85 | 52.90 | 29.37 | 29.85 | 52.90 | 29.37 | \n| 2 | 30.52 | 54.41 | 30.51 | 30.52 | 54.41 | 30.51 | \n| 3 | 30.09 | 52.72 | 29.73 | 30.09 | 52.72 | 29.73 | \n| 4 | 30.55 | 53.91 | 30.96 | 30.55 | 53.91 | 30.96 | \n| 5 | 30.16 | 53.38 | 29.83 | 30.16 | 53.38 | 29.83 | \n| 6 | 30.21 | 53.44 | 29.77 | 30.21 | 53.44 | 29.77 | \n| 7 | 29.89 | 53.39 | 29.30 | 29.89 | 53.39 | 29.30 | \n| 8 | 29.80 | 52.90 | 29.53 | 29.80 | 52.90 | 29.53 | \n| 9 | 30.11 | 54.39 | 29.55 | 30.11 | 54.39 | 29.55 | \n| \xce\xbc | 30.13 | 53.46 | 29.86 | 30.13 | 53.46 | 29.86 | \n| c | 0.15 | 0.35 | 0.31 | 0.15 | 0.35 | 0.31 |'

--> 2-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 28.62 | 49.23 | 31.43 | 28.62 | 49.23 | 31.43 | \n| 1 | 28.67 | 48.65 | 30.34 | 28.67 | 48.65 | 30.34 | \n| 2 | 29.21 | 49.89 | 30.79 | 29.21 | 49.89 | 30.79 | \n| 3 | 28.80 | 48.55 | 31.55 | 28.80 | 48.55 | 31.55 | \n| 4 | 29.02 | 49.07 | 31.11 | 29.02 | 49.07 | 31.11 | \n| 5 | 29.01 | 50.02 | 30.51 | 29.01 | 50.02 | 30.51 | \n| 6 | 29.05 | 48.80 | 30.78 | 29.05 | 48.80 | 30.78 | \n| 7 | 29.50 | 50.18 | 31.46 | 29.50 | 50.18 | 31.46 | \n| 8 | 29.23 | 49.88 | 31.28 | 29.23 | 49.88 | 31.28 | \n| 9 | 29.79 | 49.81 | 31.95 | 29.79 | 49.81 | 31.95 | \n| \xce\xbc | 29.09 | 49.41 | 31.12 | 29.09 | 49.41 | 31.12 | \n| c | 0.21 | 0.36 | 0.30 | 0.21 | 0.36 | 0.30 |'

--> 3-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 30.79 | 50.88 | 33.44 | 30.79 | 50.88 | 33.44 | \n| 1 | 31.04 | 50.74 | 33.55 | 31.04 | 50.74 | 33.55 | \n| 2 | 31.12 | 51.27 | 33.69 | 31.12 | 51.27 | 33.69 | \n| 3 | 30.59 | 51.02 | 33.14 | 30.59 | 51.02 | 33.14 | \n| 4 | 30.30 | 50.43 | 32.69 | 30.30 | 50.43 | 32.69 | \n| 5 | 30.71 | 50.90 | 32.63 | 30.71 | 50.90 | 32.63 | \n| 6 | 31.22 | 50.67 | 33.35 | 31.22 | 50.67 | 33.35 | \n| 7 | 30.88 | 50.76 | 32.11 | 30.88 | 50.76 | 32.11 | \n| 8 | 30.80 | 50.80 | 32.76 | 30.80 | 50.80 | 32.76 | \n| 9 | 30.16 | 50.28 | 32.61 | 30.16 | 50.28 | 32.61 | \n| \xce\xbc | 30.76 | 50.77 | 33.00 | 30.76 | 50.77 | 33.00 | \n| c | 0.20 | 0.16 | 0.30 | 0.20 | 0.16 | 0.30 |'

--> 5-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 35.84 | 60.28 | 39.79 | 35.84 | 60.28 | 39.79 | \n| 1 | 35.89 | 59.92 | 38.88 | 35.89 | 59.92 | 38.88 | \n| 2 | 35.78 | 60.51 | 39.01 | 35.78 | 60.51 | 39.01 | \n| 3 | 35.90 | 60.07 | 39.02 | 35.90 | 60.07 | 39.02 | \n| 4 | 35.88 | 59.74 | 37.86 | 35.88 | 59.74 | 37.86 | \n| 5 | 35.92 | 60.21 | 39.00 | 35.92 | 60.21 | 39.00 | \n| 6 | 35.86 | 60.27 | 39.38 | 35.86 | 60.27 | 39.38 | \n| 7 | 35.89 | 60.11 | 38.39 | 35.89 | 60.11 | 38.39 | \n| 8 | 35.97 | 60.27 | 39.47 | 35.97 | 60.27 | 39.47 | \n| 9 | 35.75 | 60.21 | 39.02 | 35.75 | 60.21 | 39.02 | \n| \xce\xbc | 35.87 | 60.16 | 38.98 | 35.87 | 60.16 | 38.98 | \n| c | 0.04 | 0.13 | 0.32 | 0.04 | 0.13 | 0.32 |'

--> 10-shot b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 | \n|:---|:------|:-------|:-------|:------|:--------|:--------| \n| 0 | 38.03 | 60.52 | 41.19 | 38.03 | 60.52 | 41.19 | \n| 1 | 38.24 | 59.83 | 42.42 | 38.24 | 59.83 | 42.42 | \n| 2 | 38.10 | 59.79 | 41.78 | 38.10 | 59.79 | 41.78 | \n| 3 | 38.29 | 60.53 | 42.64 | 38.29 | 60.53 | 42.64 | \n| 4 | 39.06 | 60.68 | 42.44 | 39.06 | 60.68 | 42.44 | \n| 5 | 37.65 | 60.41 | 40.93 | 37.65 | 60.41 | 40.93 | \n| 6 | 37.30 | 59.76 | 40.94 | 37.30 | 59.76 | 40.94 | \n| 7 | 37.56 | 59.60 | 41.43 | 37.56 | 59.60 | 41.43 | \n| 8 | 37.50 | 60.42 | 41.32 | 37.50 | 60.42 | 41.32 | \n| 9 | 38.70 | 60.39 | 42.31 | 38.70 | 60.39 | 42.31 | \n| \xce\xbc | 38.04 | 60.19 | 41.74 | 38.04 | 60.19 | 41.74 | \n| c | 0.33 | 0.23 | 0.39 | 0.33 | 0.23 | 0.39 |'