address 0x2b1b5e393b48, cause 'memory not mapped' error

[basetwin]

adata_combined = ad.concat(adata_list, index_unique=None, join='outer', fill_value = 0) <- I have made h5ad file using this code

data <- open_matrix_anndata_hdf5('10x/adata_before_meta.h5ad') write_matrix_dir(mat = data, dir = 'bpc')

mat <- open_matrix_dir(dir = 'bpc') meta_data <- read_csv('10x/metadata.csv')

obj <- CreateSeuratObject(counts = mat, meta.data = meta_data) saveRDS(obj, 'adata_filtered_seurat.RDS') ~

My data is scRNA-seq data of 2 million cells. As data is so large, I merged representative scRNA-seq sample data in python (scanpy), and then I converted it into R seurat object using BPCells (upside code). After that, I processed my seurat object (Normalize, FindVariableFeatures, ScaleData). These steps (normalize ~scale) do not raise errors but in RunPCA step, the error arises.

Warning: Converting to a dense matrix may use excessive memory This message is displayed once every 8 hours.

*** caught segfault *** address 0x2b780a393b48, cause 'memory not mapped'

Traceback: 1: build_csparse_matrix_double_cpp(iter) 2: asMethod(object) 3: as(from, "dgCMatrix") 4: as.matrix(.) 5: as(from, "dgCMatrix") %>% as.matrix() 6: asMethod(object) 7: as(x, "matrix") 8: as.matrix.IterableMatrix(X) 9: as.matrix(X) 10: apply(X = data.use, MARGIN = 1L, FUN = var) 11: PrepDR5(object = object, features = features, layer = layer, verbose = verbose) 12: RunPCA.StdAssay(object = object[[assay]], assay = assay, features = features, npcs = npcs, rev.pca = rev.pca, weight.by.var = weight.by.var, verbose = verbose, ndims.print = ndims.print, nfeatures.print = nfeatures.print, reduction.key = reduction.key, seed.use = seed.use, ...) 13: RunPCA(object = object[[assay]], assay = assay, features = features, npcs = npcs, rev.pca = rev.pca, weight.by.var = weight.by.var, verbose = verbose, ndims.print = ndims.print, nfeatures.print = nfeatures.print, reduction.key = reduction.key, seed.use = seed.use, ...) 14: RunPCA.Seurat(obj, npcs = 30) 15: RunPCA(obj, npcs = 30)

how can I deal with this error? or there are any other ways to input extracted reduction information into seurat object? I have also converted preprocessed data (from scanpy, from normalization to runumap) using seuratdisk, but this step also raises errors. Loading required package: reticulate Warning: Unknown file type: h5ad Warning: 'assay' not set, setting to 'RNA' Creating h5Seurat file for version 3.1.5.9900 Adding X as data Adding X as counts Adding meta.features from var Validating h5Seurat file Warning: Feature names cannot have underscores ('_'), replacing with dashes ('-') Initializing RNA with data Error in sparseMatrix(i = x[["indices"]][] + 1, p = x[["indptr"]][], x = x[["data"]][], : 'p' must be a nondecreasing vector c(0, ...) Calls: LoadH5Seurat ... as.matrix.H5Group -> as.sparse -> as.sparse.H5Group -> sparseMatrix In addition: Warning message: In sparseMatrix(i = x[["indices"]][] + 1, p = x[["indptr"]][], x = x[["data"]][], : NAs introduced by coercion to integer range Execution halted

please help me..

[juliap2000]

Hi, we have a similar issue. When processing a snRNA-seq dataset of ~380k cells (Normalize, FindVariableFeatures, ScaleData, RunHarmony), we get an error in the RunUMAP step:

 *** caught segfault ***
address (nil), cause 'memory not mapped'

Traceback:
 1: .Primitive(".Call")("eigs_shift_sym", new("dgCMatrix", i = c(0L, 871L, 2623L, 3140L, 4902L, 5600L, 6374L, 7285L, 7559L, 7975L, 9036L, 9094L, 10912L, 55518L, 58114L, 60039L, 60848L, 61549L, 64513L, 66170L, 66423L, 68573L, 71910L, 138930L, 141019L, 162907L, 168442L, 169368L, 172898L, 225968L, 3407
 2: do.call(.Call, args = dot_call_args)
 3: eigs_real_sym(A, nrow(A), k, which, sigma, opts, mattype = "sym_dgCMatrix",     extra_args = list(use_lower = as.logical(lower)))
 4: eigs_sym.dgCMatrix(L, k = k, which = "LM", sigma = 0, opt = opt)
 5: RSpectra::eigs_sym(L, k = k, which = "LM", sigma = 0, opt = opt)
 6: doTryCatch(return(expr), name, parentenv, handler)
 7: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 8: tryCatchList(expr, classes, parentenv, handlers)
 9: tryCatch(RSpectra::eigs_sym(L, k = k, which = "LM", sigma = 0,     opt = opt), error = function(c) {    NULL})
10: withCallingHandlers(expr, warning = function(w) if (inherits(w,     classes)) tryInvokeRestart("muffleWarning"))
11: suppressWarnings(res <- tryCatch(RSpectra::eigs_sym(L, k = k,     which = "LM", sigma = 0, opt = opt), error = function(c) {    NULL}))
12: rspectra_eigs_sym(L, ndim, verbose = verbose)
13: rspectra_normalized_laplacian_init(A, ndim, verbose = FALSE)
14: spectral_init(V, ndim = n_components, verbose = verbose)
15: uwot(X = X, n_neighbors = n_neighbors, n_components = n_components,     metric = metric, n_epochs = n_epochs, alpha = learning_rate,     scale = scale, init = init, init_sdev = init_sdev, spread = spread,     min_dist = min_dist, set_op_mix_ratio = set_op_mix_ratio,     local_connectivity = loca
16: umap(X = object, n_threads = nbrOfWorkers(), n_neighbors = as.integer(x = n.neighbors),     n_components = as.integer(x = n.components), metric = metric,     n_epochs = n.epochs, learning_rate = learning.rate, min_dist = min.dist,     spread = spread, set_op_mix_ratio = set.op.mix.ratio, local_c
17: RunUMAP.default(object = data.use, reduction.model = reduction.model,     return.model = return.model, assay = assay, umap.method = umap.method,     n.neighbors = n.neighbors, n.components = n.components, metric = metric,     n.epochs = n.epochs, learning.rate = learning.rate, min.dist = min.dis
18: RunUMAP(object = data.use, reduction.model = reduction.model,     return.model = return.model, assay = assay, umap.method = umap.method,     n.neighbors = n.neighbors, n.components = n.components, metric = metric,     n.epochs = n.epochs, learning.rate = learning.rate, min.dist = min.dist,     s
19: RunUMAP.Seurat(., reduction = this_reduction, dims = 1:n_dims,     verbose = FALSE)
20: RunUMAP(., reduction = this_reduction, dims = 1:n_dims, verbose = FALSE)
21: seu_obj %>% RunPCA(npcs = n_dims, verbose = FALSE) %>% RunHarmony(c("sample_id"),     theta = theta, verbose = TRUE) %>% RunUMAP(reduction = this_reduction,     dims = 1:n_dims, verbose = FALSE)

An irrecoverable exception occurred. R is aborting now ...

We have processed datasets of similar size (and also much bigger) before and we didn't have problems. What could be the issue here?

Thanks for your help!

[kennethabarr]

I have the same issue when trying to do RunPCA on a relatively small object. 89k cells. I have worked with much larger objects in the past without issue.

`test <- RunPCA(test) Warning: Converting to a dense matrix may use excessive memory This message is displayed once every 8 hours.

*** caught segfault *** address 0x7f1bd4864c94, cause 'memory not mapped'

Traceback: 1: build_csparse_matrix_double_cpp(iter) 2: asMethod(object) 3: as(from, "dgCMatrix") 4: as.matrix(.) 5: as(from, "dgCMatrix") %>% as.matrix() 6: asMethod(object) 7: as(x, "matrix") 8: as.matrix.IterableMatrix(X) 9: as.matrix(X) 10: apply(X = data.use, MARGIN = 1L, FUN = var) 11: PrepDR5(object = object, features = features, layer = layer, verbose = verbose) 12: RunPCA.StdAssay(object = object[[assay]], assay = assay, features = features, npcs = npcs, rev.pca = rev.pca, weight.by.var = weight.by.var, verbose = verbose, ndims.print = ndims.print, nfeatures.print = nfeatures.print, reduction.key = reduction.key, seed.use = seed.use, ...) 13: RunPCA(object = object[[assay]], assay = assay, features = features, npcs = npcs, rev.pca = rev.pca, weight.by.var = weight.by.var, verbose = verbose, ndims.print = ndims.print, nfeatures.print = nfeatures.print, reduction.key = reduction.key, seed.use = seed.use, ...) 14: RunPCA.Seurat(test) 15: RunPCA(test) `

[sarahhp]

I've been getting the same issue on a dataset of 40k cells and 75k features, with counts ect. stored in file using BPCells.

My workaround was to subset the Seurat object to one only containing variable features. [ And then recall FindVariableFeatures to workaround the subset issue #8555 ]. Then RunPCA worked with moderate memory usage and I assigned the pca back into the main object.

filt

An object of class Seurat 75117 features across 40047 samples within 1 assay Active assay: RNA (75117 features, 2000 variable features) 3 layers present: counts, data, scale.data

var.feat = VariableFeatures(filt)
for_pca = subset(filt, features=var.feat)
for_pca = FindVariableFeatures(for_pca)
for_pca 

An object of class Seurat 2000 features across 40047 samples within 1 assay Active assay: RNA (2000 features, 2000 variable features) 3 layers present: counts, data, scale.data

for_pca = RunPCA(for_pca)
filt[["pca"]] = for_pca[["pca"]]

RunPCA is supposed to use variable features by default, so idk maybe there's a clue there :)

[ollieeknight]

I'm getting this error too, when not using BPCells

[hx07-code]

Hi, I belive that this problem is caused by Rcpp. I fixed this problem after removing current Rcpp and RcppAnnoy and re-compiling them in current enviroment. I guess this glitch was caused by the different c++ you using from different enviroment.