raster-vision
raster-vision copied to clipboard
Edge case: `RandomWindowGeoDataset` can sometimes cause the training to end prematurely
🐛 Bug
See the following discussion on Gitter: https://gitter.im/azavea/raster-vision?at=6298bff6ef00bd1dc60fc7ee
Whenever a RandomWindowGeoDataset
for a scene fails to find an eligible chip in max_sample_attempts
attempts, it throws a StopIteration
error and causes the training to end, ignoring any other datasets that might still not be exhausted. This causes the training epoch to end prematurely.
To Reproduce
Steps to reproduce the behavior:
- Use a training set with at least one scene where there are no eligible chips (e.g. due to AOI being too small) and at least one normal scene with multiple eligible chips.
- Use a
GeoDataConfig(methods=GeoDataWindowMethod.random)
for both. - Attempt to train by running the
train
command.
The training should end immediately even though there are several unused chips in the second scene.
Expected behavior
The training should not end until all eligible chips have been used.
Environment
- How you installed and are running Raster Vision (pip install on local vs. inside Docker image): N/A
- Raster Vision version or commit: 0.13 and later
- OS (e.g., Linux): N/A
- Python version: N/A
- CUDA/cuDNN version if running on GPU: N/A
- Any other relevant information: N/A
Additional context
It is possible that using an IterableDataset
might be more appropriate for RandomWindowGeoDataset
.