Machine Learning that Matters

[howardyclo]

Metadata

• Author: Kiri L. Wagstaff
• Organization: Jet Propulsion Laboratory, California Institute of Technology
• Conference: ICML 2012
• Paper: https://arxiv.org/pdf/1206.4656.pdf

[howardyclo]

TL;DR

This papers identifies several key obstacles in machine learning research and gives some inspirations for machine learning research that matters, aiming to address the gap between research and real-world problems.

Key Obstacles

• Machine learning for machine learning sake.
• Hyper-focus on benchmark datasets.
  • Reproducibility for evaluation results.
  • Meaningful interpretation of evaluation results (e.g., error analysis, why the particular datasets were chosen, etc.)
  • De-emphasized the need to learn how to formulate problems and define features, leaving young researchers unprepared to tackle new problems.
• Hyper-focus on abstract metrics
  • Abstract metrics like accuracy, RMSE, F-measure ignores problem-specific details.
  • Performance obtained by training a model M on dataset X may not reflect M’s performance on other datasets drawn from the same problem.

Necessary Components of Any Research with a Real Impact

• Determine what data should be collected.
• Select or extract relevant features.
• Choose an appropriate learning method.
• Select an meaningful evaluation method.
• Interpret the results.
• Involve domain experts.
• Publicize the results to the relevant scientific community.
• Persuade users to adopt the technique.

Making Machine Learning Matter

• In addition to traditional measures of performance, we can measure dollars saved, lives preserved, time conserved, effort reduced, quality of living increased, and so on. Focusing our metrics on impact will help motivate upstream restructuring of research efforts. They will guide how we select data sets, structure experiments, and define objective functions. At a minimum, publications can report how a given improvement in accuracy translates to impact for the originating problem domain.
• Involve domain experts: They could provide an independent assessment of the performance, utility, and impact of the work in relevant domain.
• Consider potential impact when selecting which research problems to tackle, not merely how interesting or challenging they are from the ML perspective.

Examples of Impact Challenges of Machine Learning that Matters:

1. A law passed or legal decision made that relies on the result of an ML analysis.
2. $100M saved through improved decision making provided by an ML system.
3. A conflict between nations averted through highquality translation provided by an ML system.
4. A 50% reduction in cybersecurity break-ins through ML defenses.
5. A human life saved through a diagnosis or intervention recommended by an ML system.
6. Improvement of 10% in one country’s Human Development Index (HDI) (Anand & Sen, 1994) attributable to an ML system.

Obstacles to ML Impact

• Jargon: Communication problem between peoples in and out of ML domains.
• Risk of deploying ML system to real world applications.
• Complexity: The field has not yet matured to a point where researchers from other areas can simply apply ML to the problem of their choice.

Read More

• Deep Reinforcement Learning That Matters by Peter Henderson et al. AAAI 2018
• Machine Learning Research that Matters for Music Creation: A Case Study by Bob L. Sturm et al. Journal of New Music Research 2018.