nowcasting_dataset

nowcasting_dataset copied to clipboard

Speed up `Manager.sample_spatial_and_temporal_locations_for_examples()` by splitting `shuffled_t0_datetimes` across multliple processes

2

but wait until we've implemented #305, as #305 is likely to be sufficient!

JackKelly

Split_data config

1

## Detailed Description values where hard coded as part of https://github.com/openclimatefix/nowcasting_dataset/issues/390 and #424 Would be good to move - train_test_validation_split - train_validation_test_datetime_split to the config model file ## Context Useful...

peterdudfield

Not currently relevant because we're not using `local_temp_path` (because we're creating batches locally)

JackKelly

## Detailed Description For example, if split method produces no validation batches. ## Context See PR #445 for more context. And see the rules defined in the `description` strings in...

JackKelly

## Detailed Description Mostly to make debugging a bit easier :slightly_smiling_face: ## Context Related issues: - #439 - #323 Although maybe this isn't strictly necessary because the "Example is the...

JackKelly

In `prepare_ml_data.py`, maybe don't allow `create_files_specifying_spatial_and_temporal_locations_of_each_example` to be run if a subset of `data_sources` is passed in at the command line

1

The issue is that, if the files specifying the spatial and temporal locations of each example are computed with _less_ DataSources than the number of DataSources used to create batches,...

JackKelly

**Describe the bug** If only train csv file has been created then it assumes the validation and test are also there [code](https://github.com/openclimatefix/nowcasting_dataset/blob/main/nowcasting_dataset/manager.py#L209) **To Reproduce** Steps to reproduce the behavior: 1....

peterdudfield

Enable config option for the MW threshold at which to drop GSP PV power

6

TBH, I'm increasingly thinking that we shouldn't drop any GSPs. More GSPs means more training examples :slightly_smiling_face: But it would be nice to be able to configure the threshold_mw in...

JackKelly

## Detailed Description Remove SAT_FILENAME from consts SAT_FILENAME however seems to be used in two places: 1. [here](https://github.com/openclimatefix/nowcasting_dataset/blob/8e85735dbd26dbd517b4bf2b3495de1d3e99b8a5/conftest.py#L34) 2. [here](https://github.com/openclimatefix/nowcasting_dataset/blob/8e85735dbd26dbd517b4bf2b3495de1d3e99b8a5/scripts/generate_data_for_tests/generate_satellite_test_data.py#L22) ## Context Good to get rid of stuff not used...

peterdudfield

## Detailed Description At present, `Manager` first gets a list of all available t0 datetimes (independent of location), and then samples available locations from GSPDataSource for a random sample of...

JackKelly