nowcasting_dataset icon indicating copy to clipboard operation
nowcasting_dataset copied to clipboard

enable nowcasting dataloader to append to the CSV files to create more batches

Open JackKelly opened this issue 2 years ago • 0 comments

At the moment, the workflow is:

  1. Specify the number of batches for each split in the config YAML
  2. Run prepare_ml_data.py. Which first creates CSV files specifying the locations of each example.
  3. Once the CSV files exist, it'd then possible to re-create batches for each modality. But it's not possible to increase the number of batches requested in the config YAML.

There should be a command line-option like --append_more_examples_to_csv_files which, as the name suggests, increases the number of examples specified in the CSV files, and then creates those new batches for each modality.

JackKelly avatar Nov 22 '21 11:11 JackKelly