patRoon
patRoon copied to clipboard
rickhelmus
Grouping optimization issue (sub-optimal formulation for "score")
Hi,
first off, thanks for your nice code package, I switched over after having some issues with IPO. The comparatively verbose output is particularly useful!
This made me notice an issue with the grouping optimization algorithm, which attempts to optimize norm(GS)+norm(RCS). Unfortunately that can lead to some issues in edge cases as in the results below. The optimization nominally succeeded, but clearly is not a good choice overall based on the abysmal GS.
In many scenarios, it would seem to help to carry forward GS and RCS between iterations, and calculating the overall norms and scores on the total set, rather than just within the iteration (would also be more consistent with the behavior of feature finding optimization). But overall, some non-linearity in weighing GS and RCS may be required to really avoid falling into parameter space minima where one drastically dominates the other,
Regards, Thomas
Response: (iteration 2)
| exp_index | good_groups | bad_groups | GS | RCS | retcor_done | experiment | score |
|---|---|---|---|---|---|---|---|
| 1 | 22 | 53271 | 9.085619e-03 | 7711.6919 | 1 | 1 | 0.9353964 |
| 2 | 3132 | 20278 | 4.837471e+02 | 611.6350 | 1 | 2 | 1.0039817 |
| 3 | 21 | 51671 | 8.534768e-03 | 7452.1002 | 1 | 3 | 0.9013410 |
| 4 | 2948 | 19128 | 4.543446e+02 | 581.2830 | 1 | 4 | 0.9392182 |
| 5 | 49 | 52094 | 4.608976e-02 | 8204.1659 | 1 | 5 | 1.0000776 |
| 6 | 3018 | 20606 | 4.420229e+02 | 630.4261 | 1 | 6 | 0.9201931 |
| 7 | 43 | 50487 | 3.662329e-02 | 7940.6152 | 1 | 7 | 0.9654844 |
| 8 | 2868 | 19425 | 4.234453e+02 | 601.1880 | 1 | 8 | 0.8779533 |
| 9 | 2176 | 26091 | 1.814793e+02 | 1009.5788 | 1 | 9 | 0.4313277 |
| 10 | 37 | 52515 | 2.606874e-02 | 7839.8853 | 1 | 10 | 0.9522485 |
| 11 | 2907 | 20047 | 4.215418e+02 | 604.1467 | 1 | 11 | 0.8744066 |
| 12 | 2215 | 26942 | 1.821032e+02 | 1040.7310 | 1 | 12 | 0.4367042 |
| 13 | 2086 | 25707 | 1.692689e+02 | 1000.2034 | 1 | 13 | 0.4048561 |
| 14 | 2164 | 26064 | 1.796691e+02 | 1009.0638 | 1 | 14 | 0.4275181 |
| 15 | 2494 | 25309 | 2.457638e+02 | 1006.8286 | 1 | 15 | 0.5638580 |
| 16 | 2176 | 26091 | 1.814793e+02 | 1009.5788 | 1 | 16 | 0.4313277 |
Best params: groupArgs: list(bw = 0.25, mzwid = 0.00912, method = "density"); retcorArgs: list(distFunc = "cor_opt", gapInit = 0.3, gapExtend = 2.4, profStep = 1, method = "obiwarp"); Best results: exp_index: 1; good_groups: 45; bad_groups: 51404; GS: 0.0393938214924909; RCS: 8105.90554502488; retcor_done: 1
