series.apply big tiff / bad block

[mmann1123]

I am running into an issue with large raster stacks, when I am applying my interpolator. Looks like maybe we just need to add a kw args parameter for series.apply. I will take a stab at it.

       with gw.series(
            a_grid, transfer_lib="numpy"
        ) as src:  # , window_size=[128, 128],
            src.apply(
                func=interpolate_nan(
                    interp_type="linear",
                    missing_value=missing_data,
                    count=len(src.filenames),
                ),
                outfile=f"./{band_name}/interpolated/S2_SR_interpolated_{grid}_{dates[0].strftime('%Y_%m')}_{dates[-1].strftime('%Y_%m')}.tif",
                num_workers=6,  # src.nchunks,
                bands=1,
            )

The directory already exists. Skipping.
working on grid 0000000000-0000046592
['./B12/S2_SR_B12_M_2023_01-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_02-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_03-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_04-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_05-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_06-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_07-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_08-0000000000-0000046592.tif']
[datetime.datetime(2023, 1, 1, 0, 0), datetime.datetime(2023, 2, 1, 0, 0), datetime.datetime(2023, 3, 1, 0, 0), datetime.datetime(2023, 4, 1, 0, 0), datetime.datetime(2023, 5, 1, 0, 0), datetime.datetime(2023, 6, 1, 0, 0), datetime.datetime(2023, 7, 1, 0, 0), datetime.datetime(2023, 8, 1, 0, 0)]
working on B12 0000000000-0000046592
NOTE: Dates are unknown, assuming regular time interval
 86%|████████▌ | 4338/5032 [59:27<10:02,  1.15it/s]  ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO
 86%|████████▌ | 4339/5032 [59:28<09:29,  1.22it/s]
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: [./B12/interpolated/S2_SR_interpolated_0000000000-0000046592_2023_01_2023_08.tif](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/B12/interpolated/S2_SR_interpolated_0000000000-0000046592_2023_01_2023_08.tif): WriteEncodedTile/Strip() failed.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
---------------------------------------------------------------------------
CPLE_AppDefinedError                      Traceback (most recent call last)
File rasterio/_io.pyx:1741, in rasterio._io.DatasetWriterBase.write()

File rasterio/_io.pyx:174, in rasterio._io.io_multi_band()

File rasterio/_err.pyx:195, in rasterio._err.exc_wrap_int()

CPLE_AppDefinedError: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
[/home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py) in line 50
     [126](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=125) print(f"working on {band_name} {grid}")
     [127](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=126) with gw.series(
     [128](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=127)     a_grid, transfer_lib="numpy"
     [129](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=128) ) as src:  # , window_size=[128, 128],
---> [130](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=129)     src.apply(
     [131](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=130)         func=interpolate_nan(
     [132](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=131)             interp_type="linear",
     [133](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=132)             missing_value=missing_data,
     [134](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=133)             count=len(src.filenames),
     [135](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=134)         ),
     [136](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=135)         outfile=f"[./](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/){band_name}[/interpolated/S2_SR_interpolated_](https://file+.vscode-resource.vscode-cdn.net/interpolated/S2_SR_interpolated_){grid}_{dates[0].strftime('%Y_%m')}_{dates[-1].strftime('%Y_%m')}.tif",
...
   [1293](file:///home/mmann1123/miniconda3/envs/xr_fresh/lib/python3.9/site-packages/geowombat/core/api.py?line=1292) )

File rasterio/_io.pyx:1743, in rasterio._io.DatasetWriterBase.write()

RasterioIOError: Read or write failed. An error occurred while writing a dirty block from GDALRasterBand::IRasterIO