cosmoflow: yaml file for the small example

[laszewsk]

Steve, please can you send me or check in a yaml file for the small example. For some reason when I try to modify parameters to use the small dataset its not properly working

Thanks

[sparticlesteve]

Hi Gregor,

The easiest thing if you just want to test is to just use the regular cosmo.yaml config and override the settings for number of samples, e.g. command line args --n-train 64 --n-valid 64 or in the yaml:

data:
    ...
    n_train: 64
    n_valid: 64

I don't have a "mini" dataset that's compatible with MLPerf HPC v1.0 and v2.0. There is only the single big tarball: https://github.com/mlcommons/hpc/tree/main/cosmoflow#mlperf-hpc-v10-preliminary-dataset

Alternatively, for a small test, you could try the dummy dataset config: https://github.com/mlcommons/hpc/blob/main/cosmoflow/configs/cosmo_dummy.yaml Unfortunately I haven't tested it in a while, but if you have problems I could try to fix them.

Steve