If I want to debug fairseq-train how do I do it?

[htdepot]

I want to explore the details of the code, but I don't know how to debug fairseq-train

[htdepot]

Resolved

[mavisguan]

You can take a look at https://github.com/microsoft/Graphormer/issues/133

[whu-dft]

1. Install the Graphormer

git clone [email protected]:microsoft/Graphormer.git
cd Graphormer
bash install.sh

2. Find the way to start the training of Graphormer

cd example/oc20
cat oc20.sh
# try to start the training
bash oc20.sh

3. Assuming you can training the graphormer correctly. You can find that the training of graphormer is lauched by fairseq-train. Then you find the location of fairseq-train, which is still a python scripts:

which  fairseq-train

4. Open the fairseq-train with your IDE (for example, vscode), and use it as the start scripts. Setting your breakpoints. Don't forget to config the launch.json (don't forget to set justMyCode to False:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false,
            // graphormer 3D
            "args": [
                "--user-dir=/home/fdd/projects/Graphormer/graphormer", "/raid/open_catalyst_data/is2res_train_val_test_lmdbs/data/is2re/all",
                "--valid-subset=val_id,val_ood_ads,val_ood_cat,val_ood_both", "--best-checkpoint-metric=loss", 
                "--num-workers=0", "--ddp-backend=c10d", "--task=is2re", "--criterion=mae_deltapos", "--arch=graphormer3d_base", 
                "--optimizer=adam", "--adam-betas=(0.9, 0.98)", "--adam-eps=1e-6", "--clip-norm=5", "--lr-scheduler=polynomial_decay", 
                "--lr=3e-4", "--warmup-updates=10000", "--total-num-update=1000000", "--batch-size=2", "--dropout=0.0", "--attention-dropout=0.0","--activation-dropout=0.0",  
                // "--weight-decay=0.001", "--update-freq=1", "--seed=1", "--fp16", "--fp16-init-scale=4", "--fp16-scale-window=256", 
                "--weight-decay=0.001", "--update-freq=1", "--seed=1", "--fp16-init-scale=4", "--fp16-scale-window=256", 
                "--tensorboard-logdir=./tsbs", "--embed-dim=768", "--ffn-embed-dim=768", "--attention-heads=48", "--max-update=1000000", 
                "--log-interval=1",  "--log-format=simple", "--save-interval-updates=5000",  "--validate-interval-updates=2500",  "--keep-interval-updates=30",
                "--no-epoch-checkpoints", "--save-dir=./ckpts", "--layers=12", "--blocks=4", "--required-batch-size-multiple=1", "--node-loss-weight=15"
            ],
            "env": {
                "CUDA_VISIBLE_DEVICES": "1"
            }
        }
    ]
}

Good luck!