Documentation of Monte-Varlo DropOut at inference time

[Kensuke-Mitsuzawa]

trafficstars

📚 Documentation

Issue Description

Hi, I am providing an example of how to enable Monte Carlo Dropout during inference to improve accessibility and usability. FairSeq includes an implementation of dropout, but the option to enable it during inference time is not documented.

Suggestion

Add documentation or a built-in method to enable dropout during inference.

Goal

I aim to implement "Monte Carlo Dropout" (Yarin Gal, Zoubin Ghahramani, 2015) within the FairSeq interface.

Example Code

from fairseq.hub_utils import GeneratorHubInterface
from fairseq.modules.fairseq_dropout import FairseqDropout

def enable_mc_dropout(fairseq_interface: GeneratorHubInterface) -> GeneratorHubInterface:
    """
    Enables Monte Carlo Dropout during inference by setting apply_during_inference to True
    for all FairseqDropout layers in the model's decoder.
    """
    model_decoder = fairseq_interface.models[0].decoder
    is_dropout_enabled = False
    
    for name, module in model_decoder.named_modules():
        if isinstance(module, FairseqDropout):
            module.apply_during_inference = True
            assert module.apply_during_inference, f"Failed to enable dropout for {name}"
            is_dropout_enabled = True
    
    assert is_dropout_enabled, "No FairseqDropout layers found to enable MC Dropout"
    return fairseq_interface

Explanation

A decoder model includes a dropout class called FairseqDropout. This class has an attribute apply_during_inference, which, when set to True, enables dropout during inference.

This example code demonstrates how to modify an existing FairSeq model to use Monte Carlo Dropout by setting apply_during_inference = True for all FairseqDropout layers.

Tested with: fairseq 0.12.2.