fastai parity

[lorenzoh]

This issue tracks the progress on fastai parity.

Last updated 2021/08/11

Datasets

• [x] easily download datasets from fastai dataset collection (datasetpath)
• [x] flexible APIs for loading datasets in varied formats (Data container tutorial)

Data pipelines

• [x] create data pipelines from data block information (BlockMethod)
  • [x] visualizations based on blocks
• [x] fast, paralllelized data loading (DataLoaders.jl)
• [x] fast, composable affine augmentations for images, masks and keypoints (DataAugmentation.jl)
  • [ ] on GPU: GPU support is still WIP, see https://github.com/lorenzoh/DataAugmentation.jl/issues/48
• advanced data augmentation
  • [ ] MixUp
  • [ ] CutMix

Models

• architectures
  • [x] XResNet (FastAI.Models.xresnet*)
  • [x] UNet (FastAI.Models.UNetDynamic)
• [x] pretrained weights (Metalhead.jl#70)

Training

• training schedules

  • [x] hyperparameter scheduling (Hyperparameter scheduling in FluxTraining.jl)
  • [x] one-cycle schedule One-cycle in FastAI.jl)
  • [x] finetuning schedule Finetuning in FastAI.jl)
  • [ ] flat cosine schedule

• [ ] mixed precision training

  AFAIK this is currently in the works by the Flux.jl team

• [ ] distributed data parallel training

  Lots of progress at DaggerFlux.jl. Needs to be integrated with FastAI.jl

• callbacks

  • [x] early stopping (EarlyStopping)

  • [x] checkpointing (Checkpointer)

    Works but could use some improvements such as conditional checkpointing

  • [x] stopping on NaN loss (StopOnNaNLoss)

  • [x] metrics and history tracking (Metrics, Recorder)

  • logging modalities

    • [x] hyperparameters
    • [x] metrics

  • logging backends

    • [x] TensorBoard (TensorBoardBackend)
    • [x] WandB (WandB.jl)
    • [ ] Neptune.ai

  • [ ] gradient accumulation

• [ ] metrics and loss functions

  Many metrics and loss functions are still missing, see discussion below

Applications

• computer vision
  • image classification
    • [x] single-label
    • [x] multi-label
  • [x] image segmentation
  • [x] image keypoint regression
• tabular
  • [x] classification
  • [x] regression
• [ ] recommender systems
• [ ] natural language processing

[lorenzoh]

Updated the list with pointers to existing implementations

[ToucheSir]

I vote for reviving MLMetrics instead of rolling them into FluxTraining. I'm sure there would be many base Flux, Knet and other library users who could make use of them. Currently everybody has a incomplete subset of metrics with incompatible APIs (e.g. Avalon.jl). I would hope that we could get rid of those for a "NNlib of metrics" instead.

[DoktorMike]

I vote for reviving MLMetrics instead of rolling them into FluxTraining. I'm sure there would be many base Flux, Knet and other library users who could make use of them. Currently everybody has a incomplete subset of metrics with incompatible APIs (e.g. Avalon.jl). I would hope that we could get rid of those for a "NNlib of metrics" instead.

Since most of these metrics are really distances in one form or another wouldn't it be natural to use the existing implementation in Distances.jl ? But maybe I'm missing something.

[darsnack]

Yeah I think implementing the distance part of it in Distances.jl makes a lot of sense.

[ToucheSir]

I think there's still a need for a metrics package because most of the classification metrics don't make sense in Distances.jl. There's also the case of domain-specific metrics like ~~Dice score~~(edit: potentially general enough to go in something like Distances) and BLEU.

[CarloLucibello]

there should be some distinction between losses and metrics?

[ToucheSir]

At the very least, the metrics package/namespace could reexport losses that are also metrics.

[darsnack]

Yeah I think the hierarchy would be Distances -> Metrics -> Losses. There will be losses that are not metrics (i.e. defined completely in a loss package), and losses that just reexport a metric. Similarly there will be metrics that are completely defined the metrics package, but many will reexport or build upon standard distances.

To that end, I agree that we should make use of Distances.jl as much as possible. And if there is a metric that generalizes to a distance, then we can submit a PR to Distances.jl.

[lorenzoh]

I agree with @darsnack! Every loss can be a metric (not in a strict mathematical sense but as a measure of model performance), but not the reverse.

[lorenzoh]

Also, if MLMetrics.jl is revived, FluxTraining.jl should depend on it.

[lorenzoh]

Updated the list

[darsnack]

@lorenzoh do you want to transfer stuff from here to FastAI.jl issues?

[lorenzoh]

I went ahead and transferred the issue to the FastAI.jl repo which should make it easier to track 👍