fit_generator not working for yolo3_generator

[alefjonathan29]

When trying to run the BCCD example, with the data set and the code provided, the following error is displayed:

blood_yolo %>% fit_generator( generator = train_blood_yolo_generator, epochs = 1, steps_per_epoch = 19, validation_data = valid_blood_yolo_generator, validation_steps = 5, callbacks = list(callback_model_checkpoint("BCCD/blood_w.hdf5", save_best_only = TRUE, save_weights_only = TRUE) ) )

Error in py_call_impl(callable, dots$args, dots$keywords) : StopIteration:

Detailed traceback: File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func return func(*args, **kwargs) File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1479, in fit_generator initial_epoch=initial_epoch) File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper return method(self, *args, **kwargs) File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 815, in fit model=self) File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1112, in init model=model)

[maju116]

@alefjonathan29 can you tell me what version of tensorflow (python module) and OS do you have ? Also please copy output form sessionInfo()

[alefjonathan29]

Hi, @maju116. I'm including the requested information here.

tensorflow::tf_version()

[1] ‘2.2’

sessionInfo()

R version 4.0.2 (2020-06-22) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Catalina 10.15.6

Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale: [1] pt_BR.UTF-8/pt_BR.UTF-8/pt_BR.UTF-8/C/pt_BR.UTF-8/pt_BR.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods
[7] base

other attached packages: [1] here_0.1 abind_1.4-5 platypus_0.1.1
[4] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.2
[7] purrr_0.3.4 readr_1.3.1 tidyr_1.1.2
[10] tibble_3.0.3 ggplot2_3.3.2 tidyverse_1.3.0 [13] keras_2.3.0.0 tensorflow_2.2.0

loaded via a namespace (and not attached): [1] Rcpp_1.0.5 lubridate_1.7.9 lattice_0.20-41
[4] prettyunits_1.1.1 assertthat_0.2.1 zeallot_0.1.0
[7] rprojroot_1.3-2 digest_0.6.25 R6_2.4.1
[10] cellranger_1.1.0 backports_1.1.10 reprex_0.3.0
[13] httr_1.4.2 pillar_1.4.6 tfruns_1.4
[16] rlang_0.4.7 progress_1.2.2 readxl_1.3.1
[19] rstudioapi_0.11 whisker_0.4 blob_1.2.1
[22] Matrix_1.2-18 reticulate_1.16-9001 labeling_0.3
[25] munsell_0.5.0 broom_0.7.0 compiler_4.0.2
[28] modelr_0.1.8 pkgconfig_2.0.3 base64enc_0.1-3
[31] tidyselect_1.1.0 gridExtra_2.3 XML_3.99-0.5
[34] fansi_0.4.1 crayon_1.3.4 dbplyr_1.4.4
[37] withr_2.3.0 grid_4.0.2 jsonlite_1.7.1
[40] gtable_0.3.0 lifecycle_0.2.0 DBI_1.1.0
[43] magrittr_1.5 scales_1.1.1 cli_2.0.2
[46] stringi_1.4.6 farver_2.0.3 fs_1.5.0
[49] remotes_2.2.0 xml2_1.3.2 ellipsis_0.3.1
[52] generics_0.0.2 vctrs_0.3.4 RColorBrewer_1.1-2
[55] tools_4.0.2 glue_1.4.2 hms_0.5.3
[58] colorspace_1.4-1 rvest_0.3.6 haven_2.3.1
Epoch 1/1000

[maju116]

@alefjonathan29 Issue with custom R generators is unfortunately coming back to me all the time. In current version of R keras package generator functions are not working correctly (please refer to https://github.com/rstudio/keras/issues/1073 and https://github.com/rstudio/keras/issues/1090). I thought that I solved this problem but it seems like this solution works only for some versions of tensorflow.

I will investigate this but it could take some time. For now you can do one of two things:

1. Use fit function instead of fit_generator. To load the data you can use:

train_blood_yolo_generator <- yolo3_generator(
  annot_path = file.path(BCCD_path, "train", "Annotations/"),
  images_path = file.path(BCCD_path, "train", "JPEGImages/"),
  net_h = net_h,
  net_w = net_w,
  batch_size = 364, # ALL IMAGES
  shuffle = FALSE,
  labels = blood_labels
)

blood_data <- train_blood_yolo_generator %>% generator_next() # Read data into memory

history <- blood_yolo %>% fit(blood_data[[1]], blood_data[[2]], epochs = 5)

But this solution will work only for small datasets.

2. for larger datasets you can write your own loop using train_on_batch function. In platypus package you have access to custom_fit_generator and custom_predict_generator functions. Using them you can create methods for YOLOv3:

yolo3_fit_generator <- function(model, generator, epochs, steps_per_epoch,
                                validation_generator = NULL,
                                validation_steps_per_epoch = NULL,
                                model_filepath = NULL) {
  metric_names <- c("loss", "grid1_loss", "grid2_loss", "grid3_loss",
                    "grid1_avg_IoU", "grid2_avg_IoU", "grid3_avg_IoU")
  custom_fit_generator(metric_names, model, generator, epochs, steps_per_epoch,
                       validation_generator,
                       validation_steps_per_epoch,
                       model_filepath)
}

yolo3_predict_generator <- function(model, generator, steps) {
  predictions <- custom_predict_generator(model, generator, steps)
  grid1 <- predictions %>% map(~ .[[1]]) %>%
    abind(along = 1)
  grid2 <- predictions %>% map(~ .[[2]]) %>%
    abind(along = 1)
  grid3 <- predictions %>% map(~ .[[3]]) %>%
    abind(along = 1)
  list(grid1, grid2, grid3)
}

But here you won't have full functionality like callbacks.

Hope this helps for now.

[alefjonathan29]

@maju116, thank you very much for your help, I'm still having some errors. I will be waiting for the new versions of the platypus package. Congratulations for all your effort and work, I know it is not easy.

[maju116]

@alefjonathan29 please install platypus package from yolo3_fix branch: remotes::install_github("maju116/platypus@yolo3_fix") and instead of fit_generator function use yolo3_fit_generator function like in the example below: https://github.com/maju116/platypus/blob/yolo3_fix/examples/Blood%20Cell%20Detection/Blood-Cell-Detection.md

It's still work in progress, but I hope it will be enough for now. Let me know if it's working for you.

[alefjonathan29]

@maju116, I haven't found success yet. When I put it to running:

history <- blood_yolo %>%
  yolo3_fit_generator(
    generator = train_blood_yolo_generator,
    epochs = 5,
    steps_per_epoch = 37,
    validation_generator = valid_blood_yolo_generator,
    validation_steps_per_epoch = 9,
    model_filepath = "development/BCCD/blood_w.hdf5",
    save_best_only = TRUE,
    monitor = "val_loss"
  )

I get the error:

Error in py_call_impl(callable, dots$args, dots$keywords) : InvalidArgumentError: Length for attr 'output_shapes' of 0 must be at least minimum 1 ; NodeDef: {{node TensorDataset}}; Op<name=TensorDataset; signature=components: -> handle:variant; attr=Toutput_types:list(type),min=1; attr=output_shapes:list(shape),min=1; is_stateful=true> [Op:TensorDataset]

Detailed traceback: File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1346, in train_on_batch class_weight) File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1383, in single_batch_iterator dataset = dataset_ops.DatasetV2.from_tensors(data) File "/Users/alefjonathan/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 562, in from_tensors return TensorDataset(tensors) File "/Users/alefjonathan/Library/r-miniconda/envs/

[maju116]

@alefjonathan29 This solution works for #82 (at least partially) . I didn't test it for older python versions yet, so if it's possible could you upgrade python version to at least 3.7 and check if it's working for you ? I will try to add more CI tests soon.