Parallelization of runCorrelation

[BenjaminDEMAILLE]

runCorrelation <- function (exprMat_filtered, scenicOptions) 
{
    registerDoParallel(makeCluster(parallel::detectCores()))
    exprMat_filtered <- t(exprMat_filtered)
    print("Do correlation")
    corrMat <- foreach(i = seq_len(ncol(exprMat_filtered)),
                       .combine = rbind,
                       .multicombine = TRUE,
                       .inorder = FALSE) %dopar% {
                         cor(exprMat_filtered[,i], exprMat_filtered, method = 'spearman')
}

    saveRDS(corrMat, file = getIntName(scenicOptions, "corrMat"))
}

Hello ! is it possible to update runCorrelation to this ? mono core when analysing big dataset is very slow

[Polligator]

I think the idea is good, but I think the posted code is wrong. also, you need to be careful when use all cores to do the calculation, ram will easily overflow when dealing with big data, it's better to use scenicOptions defined ncores to have more control.

[BenjaminDEMAILLE]

what's wrong for you ? is preferable to use mcmapply ?