Parallelization in Seurat with future is not working

[nsabath]

Hi

I followed your Parallelization vignette (https://satijalab.org/seurat/articles/future_vignette.html) and got very different results. Only in FindMarkers the multiprocess is actually faster: fxn time strategy 1 ScaleData 1.857668 sequential 2 FindMarkers 34.242740 sequential 3 NormalizeData 2.215912 sequential 4 FindClusters 3.373150 sequential 5 ScaleData 2.320315 multiprocess 6 FindMarkers 14.840654 multiprocess 7 NormalizeData 4.397746 multiprocess 8 FindClusters 3.616188 multiprocess

any idea why?

Thanks,

Niv

Code:

library(future)

timing.comparisons <- data.frame(fxn = character(), time = numeric(), strategy = character())

plan("sequential") plan()

start <- Sys.time() Mono.Mac <- ScaleData(Mono.Mac, verbose = FALSE) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "ScaleData", time = as.numeric(end -start, units = "secs"), strategy = "sequential")) Idents(Mono.Mac) = Mono.Mac$Cell.type

start <- Sys.time() markers <- FindMarkers(Mono.Mac, ident.1 = "Mono.Mac.Mcm", verbose = FALSE) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "FindMarkers", time = as.numeric(end-start, units = "secs"), strategy = "sequential"))

start <- Sys.time() Mono.Mac <- NormalizeData(Mono.Mac, verbose = FALSE) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "NormalizeData", time = as.numeric(end-start, units = "secs"), strategy = "sequential"))

start <- Sys.time() Mono.Mac <- FindClusters(Mono.Mac, resolution = 0.5) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "FindClusters", time = as.numeric(end-start, units = "secs"), strategy = "sequential"))

plan("multiprocess", workers = 4) plan()

start <- Sys.time() Mono.Mac <- ScaleData(Mono.Mac, verbose = FALSE) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "ScaleData", time = as.numeric(end -start, units = "secs"), strategy = "multiprocess")) Idents(Mono.Mac) = Mono.Mac$Cell.type

start <- Sys.time() markers <- FindMarkers(Mono.Mac, ident.1 = "Mono.Mac.Mcm", verbose = FALSE) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "FindMarkers", time = as.numeric(end - start, units = "secs"), strategy = "multiprocess"))

start <- Sys.time() Mono.Mac <- NormalizeData(Mono.Mac, verbose = FALSE) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "NormalizeData", time = as.numeric(end-start, units = "secs"), strategy = "multiprocess"))

start <- Sys.time() Mono.Mac <- FindClusters(Mono.Mac, resolution = 0.5) end <- Sys.time() timing.comparisons <- rbind(timing.comparisons, data.frame(fxn = "FindClusters", time = as.numeric(end-start, units = "secs"), strategy = "multiprocess"))

timing.comparisons

sessionInfo()

R version 4.0.5 (2021-03-31) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.5 LTS

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale: [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8 [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] grid stats graphics grDevices utils datasets methods [8] base

other attached packages: [1] cowplot_1.1.1 future_1.21.0 reshape2_1.4.4 [4] docstring_1.0.0 dplyr_1.0.9 gplots_3.1.1 [7] ggplot2_3.3.6 ComplexHeatmap_2.6.2 SeuratDisk_0.0.0.9019 [10] SeuratObject_4.0.2 Seurat_4.0.3

loaded via a namespace (and not attached): [1] Rtsne_0.15 colorspace_2.0-3 rjson_0.2.20 [4] deldir_0.2-10 ellipsis_0.3.2 ggridges_0.5.3 [7] circlize_0.4.13 GlobalOptions_0.1.2 clue_0.3-59 [10] spatstat.data_2.1-0 farver_2.1.0 roxygen2_7.1.1 [13] leiden_0.3.8 listenv_0.8.0 ggrepel_0.9.1 [16] bit64_4.0.5 fansi_1.0.3 xml2_1.3.2 [19] codetools_0.2-18 splines_4.0.5 knitr_1.39 [22] polyclip_1.10-0 jsonlite_1.8.0 Cairo_1.5-12.2 [25] ica_1.0-2 cluster_2.1.2 png_0.1-7 [28] uwot_0.1.10 shiny_1.6.0 sctransform_0.3.2 [31] spatstat.sparse_2.0-0 compiler_4.0.5 httr_1.4.2 [34] assertthat_0.2.1 Matrix_1.3-4 fastmap_1.1.0 [37] lazyeval_0.2.2 limma_3.46.0 cli_3.3.0 [40] later_1.2.0 htmltools_0.5.1.1 tools_4.0.5 [43] igraph_1.2.6 gtable_0.3.0 glue_1.6.2 [46] RANN_2.6.1 Rcpp_1.0.8.3 scattermore_0.7 [49] vctrs_0.4.1 nlme_3.1-152 lmtest_0.9-38 [52] xfun_0.31 stringr_1.4.0 globals_0.14.0 [55] mime_0.10 miniUI_0.1.1.1 lifecycle_1.0.1 [58] irlba_2.3.3 gtools_3.9.2 goftest_1.2-2 [61] MASS_7.3-54 zoo_1.8-9 scales_1.2.0 [64] spatstat.core_2.2-0 promises_1.2.0.1 spatstat.utils_2.2-0 [67] parallel_4.0.5 RColorBrewer_1.1-3 reticulate_1.22 [70] pbapply_1.4-3 gridExtra_2.3 rpart_4.1-15 [73] stringi_1.7.6 S4Vectors_0.28.1 caTools_1.18.2 [76] BiocGenerics_0.36.1 shape_1.4.6 bitops_1.0-7 [79] rlang_1.0.3 pkgconfig_2.0.3 matrixStats_0.59.0 [82] lattice_0.20-44 ROCR_1.0-11 purrr_0.3.4 [85] tensor_1.5 labeling_0.4.2 patchwork_1.1.1 [88] htmlwidgets_1.5.3 bit_4.0.4 tidyselect_1.1.2 [91] parallelly_1.26.0 RcppAnnoy_0.0.18 plyr_1.8.6 [94] magrittr_2.0.3 R6_2.5.1 IRanges_2.24.1 [97] generics_0.1.2 DBI_1.1.1 pillar_1.7.0 [100] withr_2.5.0 mgcv_1.8-36 fitdistrplus_1.1-5 [103] survival_3.2-11 abind_1.4-5 tibble_3.1.7 [106] future.apply_1.7.0 crayon_1.5.1 hdf5r_1.3.3 [109] KernSmooth_2.23-20 utf8_1.2.2 spatstat.geom_2.2-0 [112] plotly_4.9.4.1 GetoptLong_1.0.5 data.table_1.14.0 [115] digest_0.6.29 xtable_1.8-4 tidyr_1.2.0 [118] httpuv_1.6.1 stats4_4.0.5 munsell_0.5.0 [121] viridisLite_0.4.0