pointMLP-pytorch
pointMLP-pytorch copied to clipboard
ma-xu
About results
Hello ! Thank you for really nice work. Now, I have trained the pointMLP, but I only obtain the best accuracy is 93.2. So, I want to know whether is the special parameters.
@Hello-hubKing Thanks for your interest.
Results on ModelNet40 are very unstable, as we discussed here: https://github.com/ma-xu/pointMLP-pytorch/issues/1#issuecomment-1032642340. In our paper, we report the best result of several runs for ModelNet40 and use voting method.
We would like to suggest run experiments on another dataset, scanobjectNN, and report the mean/std values as shown in Table 3. We believe this would be a better evaluation protocol for point cloud classification task.
Please let me know if you have any further questions.
Hi, First of all, thanks for your excellent work! All of the previous issues only talk about the performance on ModelNet40 dataset, but none of them speak about the ScanObjectNN dataset. Is there such a performance gap on the ScanObjectNN dataset as well? I trained your work on my workstation (RTX2080Ti) but only got overall accuracy of 78.8% and mean class accuracy of 76.5%
@ja604041062 Thanks for your interest.
First, about the results on ModelNet40, we have clearly explained the gap (unstable training).
Second, there would be a little variance on ScanObjectNN, as shown in Tab. 3 and Fig. 3.
Third, we reported several runs results on ScanObjectNN and the checkpoints can be found in the ReadME file. In case of any mistakes, we also retrain the model based on the release repo today. Here is a simple screenshot.
Clearly, only 38 epochs, PointMLP already achieves 81.575% overall accuracy (much higher than your results at 200 epochs), which is roughly align with our previous release checkpoints/logs at same epochs. We didn't finish the training considering the time, and the screenshot can prove the effectiveness of PointMLP already.
I assume there may be some issues with your training. You can upload your logs/training scripts and I can help debugging. Feel free to let me know if you have further questions or concerns.
Hi, the argument of my training is:
checkpoint: checkpoints/pointMLP-20220823181946 msg: None batch_size: 16 model: pointMLP num_classes: 15 epoch: 200 num_points: 1024 learning_rate: 0.01 weight_decay: 0.0001 smoothing: False seed: None workers: 4
The difference is the batch size. Due to hardware limitations (VRAM: 11G), I set batch size of 16. I don't know if it's causing the performance drop, but I don't think batch size affects that much.
and here is the training log: Epoch-Num Learning-Rate Train-Loss Train-acc-B Train-acc Valid-Loss Valid-acc-B Valid-acc 0.000000 0.009999 2.770000 8.452000 12.631000 2.568000 9.543000 16.759000 1.000000 0.009998 2.521000 14.481000 19.828000 2.433000 19.117000 25.330000 2.000000 0.009995 2.346000 21.191000 27.761000 2.221000 27.001000 35.357000 3.000000 0.009990 2.232000 27.053000 35.423000 2.105000 36.536000 45.524000 4.000000 0.009985 2.159000 31.827000 40.524000 2.047000 39.960000 48.126000 5.000000 0.009978 2.066000 36.960000 46.099000 1.939000 44.624000 54.511000 6.000000 0.009970 1.996000 41.298000 50.859000 1.892000 49.293000 57.703000 7.000000 0.009961 1.944000 44.847000 53.892000 1.876000 51.325000 57.078000 8.000000 0.009951 1.891000 48.614000 57.205000 1.819000 52.670000 60.444000 9.000000 0.009939 1.830000 52.428000 60.221000 1.815000 53.696000 59.924000 10.000000 0.009926 1.789000 54.785000 62.553000 1.788000 55.547000 62.353000 11.000000 0.009912 1.749000 58.349000 65.138000 1.784000 54.406000 60.826000 12.000000 0.009897 1.709000 60.419000 66.997000 1.720000 60.002000 64.747000 13.000000 0.009881 1.687000 62.589000 68.259000 1.716000 60.645000 64.920000 14.000000 0.009863 1.662000 64.370000 69.705000 1.728000 58.985000 64.539000 15.000000 0.009844 1.635000 65.519000 70.915000 1.699000 62.419000 67.176000 16.000000 0.009825 1.619000 67.284000 71.818000 1.697000 60.206000 66.031000 17.000000 0.009803 1.590000 69.455000 73.659000 1.672000 63.077000 66.586000 18.000000 0.009781 1.588000 69.495000 74.053000 1.667000 62.480000 66.482000 19.000000 0.009758 1.562000 71.319000 75.430000 1.633000 65.760000 69.431000 20.000000 0.009733 1.550000 72.183000 76.017000 1.629000 66.158000 70.125000 21.000000 0.009707 1.524000 73.380000 77.156000 1.598000 68.855000 71.860000 22.000000 0.009680 1.512000 75.134000 78.322000 1.644000 65.350000 69.917000 23.000000 0.009652 1.501000 75.475000 78.734000 1.612000 67.851000 71.201000 24.000000 0.009623 1.481000 76.897000 79.883000 1.693000 65.218000 67.349000 25.000000 0.009593 1.468000 77.086000 80.119000 1.645000 67.303000 69.882000 26.000000 0.009561 1.460000 77.568000 80.531000 1.587000 69.650000 72.241000 27.000000 0.009529 1.448000 78.904000 81.408000 1.566000 71.893000 74.289000 28.000000 0.009495 1.429000 79.793000 82.425000 1.606000 68.598000 70.958000 29.000000 0.009460 1.423000 80.400000 82.819000 1.598000 70.156000 72.866000 30.000000 0.009425 1.405000 81.914000 83.880000 1.592000 69.899000 72.831000 31.000000 0.009388 1.402000 81.598000 83.853000 1.616000 68.758000 70.888000 32.000000 0.009350 1.384000 82.832000 84.756000 1.578000 70.364000 73.005000 33.000000 0.009311 1.382000 83.162000 85.124000 1.550000 71.362000 74.358000 34.000000 0.009271 1.374000 83.105000 85.221000 1.585000 71.795000 72.970000 35.000000 0.009229 1.362000 84.451000 86.343000 1.543000 72.257000 75.295000 36.000000 0.009187 1.349000 85.663000 87.123000 1.574000 71.715000 73.560000 37.000000 0.009144 1.338000 85.360000 87.211000 1.576000 72.016000 73.976000 38.000000 0.009100 1.328000 86.596000 87.894000 1.606000 68.438000 71.548000 39.000000 0.009055 1.320000 87.009000 88.368000 1.541000 74.049000 75.607000 40.000000 0.009008 1.309000 87.704000 89.087000 1.558000 72.649000 74.809000 41.000000 0.008961 1.311000 87.137000 88.745000 1.543000 73.772000 76.544000 42.000000 0.008913 1.305000 87.667000 89.043000 1.554000 72.421000 75.087000 43.000000 0.008864 1.295000 88.888000 89.832000 1.537000 74.100000 75.330000 44.000000 0.008814 1.292000 88.872000 89.884000 1.543000 72.378000 74.601000 45.000000 0.008763 1.291000 88.679000 90.007000 1.531000 74.031000 75.746000 46.000000 0.008711 1.275000 89.943000 90.989000 1.529000 74.160000 76.718000 47.000000 0.008658 1.270000 90.413000 91.339000 1.582000 71.469000 73.838000 48.000000 0.008605 1.260000 90.842000 91.760000 1.553000 72.894000 75.330000 49.000000 0.008550 1.249000 91.841000 92.637000 1.535000 72.674000 75.642000 50.000000 0.008495 1.244000 91.573000 92.435000 1.519000 74.950000 76.544000 51.000000 0.008439 1.241000 92.063000 92.777000 1.556000 73.029000 76.197000 52.000000 0.008381 1.238000 92.220000 92.979000 1.532000 74.137000 75.954000 53.000000 0.008323 1.229000 92.700000 93.426000 1.562000 72.898000 75.434000 54.000000 0.008265 1.226000 93.387000 93.925000 1.555000 70.652000 74.254000 55.000000 0.008205 1.220000 93.420000 94.066000 1.538000 70.986000 74.532000 56.000000 0.008145 1.220000 93.041000 93.706000 1.521000 74.643000 76.475000 57.000000 0.008084 1.220000 93.411000 93.899000 1.580000 71.651000 74.323000 58.000000 0.008022 1.212000 93.960000 94.407000 1.525000 74.978000 76.544000 59.000000 0.007960 1.206000 94.432000 94.846000 1.555000 71.321000 74.636000 60.000000 0.007896 1.200000 94.399000 94.995000 1.552000 73.798000 75.503000 61.000000 0.007832 1.193000 94.969000 95.415000 1.556000 71.148000 74.705000 62.000000 0.007768 1.187000 95.159000 95.678000 1.566000 73.338000 74.115000 63.000000 0.007702 1.184000 95.525000 95.889000 1.539000 74.119000 75.850000 64.000000 0.007636 1.184000 95.420000 95.766000 1.533000 73.160000 75.746000 65.000000 0.007570 1.177000 95.861000 96.117000 1.548000 72.382000 75.156000 66.000000 0.007503 1.172000 96.143000 96.520000 1.518000 74.619000 76.648000 67.000000 0.007435 1.174000 96.002000 96.310000 1.546000 73.519000 76.371000 68.000000 0.007366 1.172000 95.956000 96.292000 1.532000 73.600000 76.960000 69.000000 0.007297 1.166000 96.480000 96.800000 1.567000 73.510000 75.399000 70.000000 0.007228 1.163000 96.750000 96.985000 1.534000 74.070000 76.683000 71.000000 0.007158 1.153000 97.228000 97.423000 1.503000 75.557000 78.036000 72.000000 0.007087 1.160000 96.742000 96.967000 1.573000 72.266000 74.427000 73.000000 0.007016 1.156000 97.053000 97.134000 1.534000 72.011000 75.677000 74.000000 0.006944 1.153000 97.106000 97.326000 1.515000 73.724000 76.787000 75.000000 0.006872 1.148000 97.205000 97.511000 1.527000 74.291000 77.273000 76.000000 0.006800 1.146000 97.497000 97.730000 1.546000 73.116000 75.885000 77.000000 0.006727 1.144000 97.522000 97.589000 1.533000 73.233000 75.815000 78.000000 0.006653 1.141000 97.703000 97.809000 1.540000 74.589000 76.336000 79.000000 0.006580 1.138000 97.852000 97.958000 1.544000 73.658000 76.266000 80.000000 0.006505 1.135000 97.939000 98.159000 1.502000 75.462000 77.654000 81.000000 0.006431 1.140000 97.675000 97.861000 1.538000 74.080000 76.613000 82.000000 0.006356 1.136000 97.941000 98.080000 1.518000 74.596000 76.995000 83.000000 0.006281 1.132000 97.825000 98.133000 1.523000 74.576000 76.787000 84.000000 0.006206 1.128000 98.257000 98.405000 1.517000 74.767000 76.995000 85.000000 0.006130 1.125000 98.326000 98.431000 1.525000 74.817000 76.960000 86.000000 0.006054 1.128000 98.274000 98.413000 1.530000 72.895000 75.434000 87.000000 0.005978 1.126000 98.377000 98.396000 1.521000 73.757000 76.648000 88.000000 0.005901 1.118000 98.560000 98.676000 1.510000 74.901000 77.203000 89.000000 0.005824 1.123000 98.452000 98.519000 1.483000 76.744000 78.695000 90.000000 0.005747 1.120000 98.487000 98.633000 1.521000 75.027000 77.412000 91.000000 0.005670 1.118000 98.677000 98.825000 1.489000 76.064000 78.105000 92.000000 0.005593 1.115000 98.888000 98.896000 1.543000 73.699000 75.885000 93.000000 0.005516 1.119000 98.414000 98.615000 1.515000 74.455000 76.683000 94.000000 0.005438 1.113000 98.973000 99.018000 1.553000 73.630000 75.885000 95.000000 0.005361 1.114000 98.873000 98.983000 1.500000 76.706000 78.661000 96.000000 0.005283 1.115000 98.794000 98.896000 1.543000 73.757000 76.093000 97.000000 0.005205 1.110000 98.992000 99.123000 1.500000 75.272000 77.273000 98.000000 0.005128 1.110000 98.873000 99.045000 1.496000 75.038000 77.759000 99.000000 0.005050 1.107000 99.021000 99.097000 1.527000 74.788000 77.377000 100.000000 0.004972 1.107000 99.028000 99.132000 1.472000 76.833000 79.285000 101.000000 0.004895 1.107000 98.965000 98.992000 1.470000 76.287000 79.042000 102.000000 0.004817 1.107000 99.161000 99.132000 1.495000 75.709000 78.244000 103.000000 0.004739 1.105000 99.130000 99.194000 1.506000 74.970000 77.446000 104.000000 0.004662 1.106000 99.059000 99.097000 1.510000 75.711000 77.897000 105.000000 0.004584 1.103000 99.180000 99.272000 1.488000 76.698000 78.522000 106.000000 0.004507 1.101000 99.233000 99.334000 1.506000 74.709000 77.516000 107.000000 0.004430 1.096000 99.526000 99.509000 1.487000 75.842000 78.626000 108.000000 0.004353 1.097000 99.450000 99.518000 1.489000 75.253000 77.654000 109.000000 0.004276 1.098000 99.291000 99.325000 1.512000 74.381000 77.238000 110.000000 0.004199 1.098000 99.463000 99.483000 1.529000 74.158000 76.371000 111.000000 0.004122 1.095000 99.429000 99.500000 1.507000 74.673000 77.238000 112.000000 0.004046 1.096000 99.566000 99.562000 1.509000 74.667000 77.550000 113.000000 0.003970 1.096000 99.568000 99.553000 1.493000 75.004000 77.377000 114.000000 0.003894 1.093000 99.643000 99.649000 1.473000 76.855000 79.598000 115.000000 0.003819 1.094000 99.505000 99.535000 1.493000 75.508000 77.932000 116.000000 0.003744 1.092000 99.649000 99.649000 1.509000 74.660000 76.891000 117.000000 0.003669 1.089000 99.730000 99.719000 1.531000 75.449000 77.273000 118.000000 0.003595 1.091000 99.607000 99.588000 1.517000 75.665000 77.863000 119.000000 0.003520 1.089000 99.759000 99.711000 1.508000 75.258000 77.689000 120.000000 0.003447 1.091000 99.571000 99.579000 1.514000 74.747000 77.169000 121.000000 0.003373 1.091000 99.528000 99.544000 1.489000 77.106000 78.695000 122.000000 0.003300 1.089000 99.624000 99.658000 1.498000 75.563000 77.897000 123.000000 0.003228 1.086000 99.743000 99.737000 1.481000 76.869000 79.112000 124.000000 0.003156 1.086000 99.723000 99.719000 1.498000 76.318000 78.522000 125.000000 0.003084 1.087000 99.720000 99.693000 1.492000 76.403000 78.314000 126.000000 0.003013 1.086000 99.810000 99.798000 1.515000 75.763000 77.481000 127.000000 0.002942 1.084000 99.824000 99.842000 1.503000 76.112000 78.071000 128.000000 0.002872 1.084000 99.791000 99.833000 1.485000 76.439000 78.210000 129.000000 0.002803 1.086000 99.687000 99.719000 1.509000 75.387000 77.585000 130.000000 0.002734 1.083000 99.784000 99.816000 1.512000 75.990000 78.418000 131.000000 0.002665 1.085000 99.747000 99.772000 1.501000 75.363000 77.654000 132.000000 0.002597 1.083000 99.787000 99.798000 1.497000 75.771000 78.383000 133.000000 0.002530 1.085000 99.671000 99.684000 1.506000 75.462000 77.793000 134.000000 0.002464 1.082000 99.799000 99.816000 1.498000 75.976000 77.967000 135.000000 0.002398 1.081000 99.881000 99.886000 1.488000 75.876000 78.140000 136.000000 0.002332 1.083000 99.822000 99.816000 1.494000 76.605000 78.695000 137.000000 0.002268 1.082000 99.822000 99.816000 1.487000 76.564000 78.383000 138.000000 0.002204 1.082000 99.757000 99.816000 1.499000 75.346000 77.932000 139.000000 0.002140 1.081000 99.840000 99.842000 1.493000 76.684000 78.175000 140.000000 0.002078 1.081000 99.783000 99.825000 1.469000 76.715000 79.146000 141.000000 0.002016 1.080000 99.766000 99.807000 1.503000 75.689000 77.863000 142.000000 0.001955 1.080000 99.827000 99.825000 1.502000 74.935000 77.203000 143.000000 0.001895 1.080000 99.849000 99.860000 1.503000 74.575000 77.620000 144.000000 0.001835 1.079000 99.788000 99.790000 1.482000 77.105000 79.251000 145.000000 0.001777 1.079000 99.829000 99.825000 1.478000 76.389000 78.522000 146.000000 0.001719 1.078000 99.945000 99.930000 1.490000 75.526000 77.793000 147.000000 0.001661 1.078000 99.887000 99.904000 1.489000 76.745000 78.765000 148.000000 0.001605 1.081000 99.832000 99.860000 1.486000 76.690000 78.348000 149.000000 0.001550 1.078000 99.877000 99.860000 1.484000 76.878000 78.661000 150.000000 0.001495 1.078000 99.856000 99.895000 1.473000 77.582000 79.285000 151.000000 0.001442 1.078000 99.910000 99.904000 1.487000 75.778000 78.105000 152.000000 0.001389 1.077000 99.911000 99.921000 1.502000 75.622000 77.897000 153.000000 0.001337 1.078000 99.837000 99.860000 1.500000 75.508000 77.724000 154.000000 0.001286 1.078000 99.864000 99.869000 1.502000 75.423000 77.654000 155.000000 0.001236 1.076000 99.913000 99.912000 1.483000 76.901000 78.869000 156.000000 0.001187 1.077000 99.892000 99.860000 1.482000 76.302000 78.695000 157.000000 0.001139 1.077000 99.868000 99.886000 1.476000 77.172000 79.042000 158.000000 0.001092 1.076000 99.946000 99.947000 1.491000 75.826000 78.591000 159.000000 0.001045 1.077000 99.905000 99.912000 1.501000 75.957000 78.036000 160.000000 0.001000 1.077000 99.893000 99.912000 1.491000 76.007000 78.348000 161.000000 0.000956 1.075000 99.942000 99.930000 1.481000 76.566000 78.661000 162.000000 0.000913 1.077000 99.866000 99.886000 1.490000 76.297000 78.452000 163.000000 0.000871 1.075000 99.884000 99.886000 1.493000 76.239000 78.487000 164.000000 0.000829 1.075000 99.923000 99.904000 1.482000 75.875000 78.244000 165.000000 0.000789 1.076000 99.901000 99.904000 1.483000 76.560000 78.591000 166.000000 0.000750 1.076000 99.847000 99.869000 1.490000 75.955000 78.210000 167.000000 0.000712 1.075000 99.901000 99.912000 1.492000 75.930000 78.279000 168.000000 0.000675 1.075000 99.945000 99.947000 1.491000 76.281000 78.348000 169.000000 0.000640 1.076000 99.838000 99.904000 1.493000 75.966000 78.314000 170.000000 0.000605 1.075000 99.895000 99.912000 1.495000 76.501000 78.418000 171.000000 0.000571 1.075000 99.932000 99.912000 1.486000 77.250000 79.042000 172.000000 0.000539 1.074000 99.912000 99.921000 1.502000 75.722000 77.897000 173.000000 0.000507 1.074000 99.954000 99.939000 1.494000 75.573000 78.140000 174.000000 0.000477 1.076000 99.897000 99.904000 1.490000 76.545000 78.591000 175.000000 0.000448 1.075000 99.914000 99.904000 1.478000 76.988000 78.938000 176.000000 0.000420 1.073000 99.956000 99.947000 1.484000 76.267000 78.695000 177.000000 0.000393 1.074000 99.903000 99.930000 1.482000 77.061000 78.765000 178.000000 0.000367 1.075000 99.934000 99.939000 1.483000 76.449000 78.834000 179.000000 0.000342 1.076000 99.889000 99.895000 1.484000 76.635000 78.834000 180.000000 0.000319 1.076000 99.876000 99.869000 1.480000 77.047000 79.146000 181.000000 0.000297 1.074000 99.902000 99.921000 1.480000 77.385000 79.389000 182.000000 0.000275 1.074000 99.915000 99.939000 1.486000 77.313000 79.251000 183.000000 0.000256 1.075000 99.921000 99.930000 1.477000 76.899000 78.765000 184.000000 0.000237 1.074000 99.937000 99.956000 1.503000 76.059000 78.140000 185.000000 0.000219 1.076000 99.885000 99.877000 1.481000 76.512000 78.557000 186.000000 0.000203 1.075000 99.905000 99.904000 1.497000 75.674000 77.759000 187.000000 0.000188 1.074000 99.921000 99.930000 1.481000 75.918000 78.383000 188.000000 0.000174 1.075000 99.909000 99.904000 1.479000 76.130000 78.452000 189.000000 0.000161 1.073000 99.942000 99.956000 1.480000 76.821000 79.008000 190.000000 0.000149 1.074000 99.920000 99.930000 1.494000 75.823000 77.863000 191.000000 0.000139 1.073000 99.937000 99.912000 1.487000 76.407000 78.557000 192.000000 0.000130 1.073000 99.950000 99.947000 1.485000 76.677000 78.591000 193.000000 0.000122 1.073000 99.934000 99.947000 1.480000 76.471000 78.869000 194.000000 0.000115 1.074000 99.951000 99.956000 1.481000 76.876000 79.112000 195.000000 0.000110 1.073000 99.931000 99.939000 1.476000 76.694000 78.973000 196.000000 0.000105 1.074000 99.858000 99.895000 1.488000 75.980000 78.487000 197.000000 0.000102 1.072000 99.942000 99.947000 1.481000 76.897000 78.765000 198.000000 0.000101 1.074000 99.927000 99.947000 1.496000 75.780000 78.210000 199.000000 0.000100 1.073000 99.945000 99.939000 1.501000 76.206000 77.967000
@ja604041062 Thanks for the information. The only difference is bs. Let me submit a job see what the result would be. I will get back to you by Sunday.
Hi @ja604041062 I just tested batch size 16, there is no problem. This link provides the logs of pointMLP/ pointMLP-elite trained with batchsize16 and batchsize32 (all these models are trained today, see the timestamp). Each experiment is repeated twice. google drive logs.
Here is a summary:
| Model | BatchSize | mAcc. | mAcc. in paper | OA | OA in paper |
|---|---|---|---|---|---|
| PointMLP | 32 | [83.514,83.301] | 83.9±0.5 | [85.253,84.663] | 85.4±0.3 |
| PointMLP | 16 | [83.699,84.457] | - | [85.149,85.947] | - |
| PointMLP-elite | 32 | [82.05,82.019] | 81.8±0.8 | [83.969,84.004] | 83.8±0.6 |
| PointMLP-elite | 16 | [81.045,81.885] | - | [82.929,83.657] | - |
Clearly, there is almost no performance gap between bs16 and bs32.
For your convenience, the sbatch scripts are also included in the google drive link.
Could you please double check the environment or try one more time (maybe on other machines)? The results are wired.
Let me know if you have any further questions.
@ja604041062 @Hello-hubKing Since there is no further discussion, I will close this issue. Feel free to reopen it if necessary.
Sorry for late reply, I trained on other machine with bs 32 and got the similar accuracy (84.3%). I will check my environment in the near future, thank you for your assistance!
Xu Ma @.***> 於 2022年8月30日 週二 晚上10:25寫道：
Closed #52 https://github.com/ma-xu/pointMLP-pytorch/issues/52 as completed.
— Reply to this email directly, view it on GitHub https://github.com/ma-xu/pointMLP-pytorch/issues/52#event-7288129206, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIZE3ABOEOKVE5QBS6UNAQDV3YKUPANCNFSM56HJURKQ . You are receiving this because you were mentioned.Message ID: @.***>