seurat-disk
seurat-disk copied to clipboard
Published
20 hours ago •
mojaveazure
mojaveazure
HDF5 errors on `h5$copy_obj_to(...)` calls
Hello @mojaveazure
Thanks for your work in creating this package, I believe it is very valuable to be able to store seurat objects as h5 files, as well as convert between h5seurat and h5ad.
I'm encountering a strange issue which occurs some of the times. I attached my input files, scripts and output as a zip file: seurat_disk_issue_with_hdf5r.zip
When using SeuratDisk to convert input1.h5ad and input2.h5ad (in that order) to h5seurat, everything works fine (see script1.R). When the order is reversed, I get an error because it was not able to copy meta.data/_index to cell.names (see script2.R). The error occurs at Convert.R#L566-L570.
script1.R
library(SeuratDisk)
converted <- Convert("input1.h5ad", dest = "output1.h5seurat", overwrite = TRUE)
# Warning: Unknown file type: h5ad
# Warning: 'assay' not set, setting to 'RNA'
# Creating h5Seurat file for version 3.1.2
# Adding X as data
# Adding raw/X as counts
# Adding meta.features from raw/var
# Adding X_pca as cell embeddings for pca
# Adding X_umap as cell embeddings for umap
# Adding PCs as feature loadings fpr pca
# Adding miscellaneous information for pca
# Adding standard deviations for pca
# Adding miscellaneous information for umap
# Saving nearest-neighbor graph as RNA_umap
# Adding filteringParameters to miscellaneous data
# Adding hvgParameters to miscellaneous data
# Adding nearestNeighbourParameters to miscellaneous data
# Adding normalizationParameters to miscellaneous data
# Adding pcaParameters to miscellaneous data
converted <- Convert("input2.h5ad", dest = "output2.h5seurat", overwrite = TRUE)
# Warning: Unknown file type: h5ad
# Warning: 'assay' not set, setting to 'RNA'
# Creating h5Seurat file for version 3.1.2
# Adding X as data
# Adding raw/X as counts
# Adding meta.features from raw/var
# Adding filteringParameters to miscellaneous data
script2.R
library(SeuratDisk)
converted <- Convert("input2.h5ad", dest = "output2.h5seurat", overwrite = TRUE)
# Warning: Unknown file type: h5ad
# Warning: 'assay' not set, setting to 'RNA'
# Creating h5Seurat file for version 3.1.2
# Adding X as data
# Adding raw/X as counts
# Adding meta.features from raw/var
# Adding filteringParameters to miscellaneous data
# Error in dfile[["misc"]]$obj_copy_from(src_loc = source[["uns"]], src_name = i, :
# HDF5-API Errors:
# error #000: ../../src/H5Ocopy.c in H5Ocopy(): line 233: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #001: ../../src/H5Ocopy.c in H5O__copy(): line 314: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #002: ../../src/H5Ocopy.c in H5O__copy_obj(): line 1218: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #003: ../../src/H5Ocopy.c in H5O__copy_header(): line 1162: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #004: ../../src/H5Ocopy.c in H5O__copy_header_real(): line 869: unable to perform 'post copy' operation on message
# class: HDF5
# major: Object header
# minor: Unable to initialize object
#
# error #005: ../../src/H5Ostab.c in H5O__stab_post_copy_f
# Error in private$closeFun(id) : HDF5-API Errors:
# error #000: ../../src/H5F.c in H5Fclose(): line 675: closing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #001: ../../src/H5Fint.c in H5F__close(): line 1952: decrementing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to decrement reference count
#
# error #002: ../../src/H5I.c in H5I_dec_app_ref(): line 1300: can't decrement ID ref count
# class: HDF5
# major: Object atom
# minor: Unable to decrement reference count
#
# error #003: ../../src/H5Fint.c in H5F__close_cb(): line 2010: can't close file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #004: ../../src/H5Fint.c in H5F_try_close(): line 2181: problems closing file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #005: ../../src/H5Fint.c in H5F__dest(): line 1277: p
converted <- Convert("input1.h5ad", dest = "output1.h5seurat", overwrite = TRUE)
# Warning: Unknown file type: h5ad
# Warning: 'assay' not set, setting to 'RNA'
# Creating h5Seurat file for version 3.1.2
# Adding X as data
# Adding raw/X as counts
# Adding meta.features from raw/var
# Error in dfile$obj_copy_from(src_loc = dfile, src_name = paste0("meta.data/", :
# HDF5-API Errors:
# error #000: ../../src/H5Ocopy.c in H5Ocopy(): line 233: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #001: ../../src/H5Ocopy.c in H5O__copy(): line 314: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
# error #002: ../../src/H5Ocopy.c in H5O__copy_obj(): line 1218: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #003: ../../src/H5Ocopy.c in H5O__copy_header(): line 1162: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #004: ../../src/H5Ocopy.c in H5O__copy_header_real(): line 887: unable to re-tag metadata entries
# class: HDF5
# major: Object cache
# minor: Unable to tag metadata in the cache
#
# error #005: ../../src/H5AC.c in H5AC_retag_copied_metadata(): line
# Error in private$closeFun(id) : HDF5-API Errors:
# error #000: ../../src/H5F.c in H5Fclose(): line 675: closing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #001: ../../src/H5Fint.c in H5F__close(): line 1952: decrementing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to decrement reference count
#
# error #002: ../../src/H5I.c in H5I_dec_app_ref(): line 1300: can't decrement ID ref count
# class: HDF5
# major: Object atom
# minor: Unable to decrement reference count
#
# error #003: ../../src/H5Fint.c in H5F__close_cb(): line 2010: can't close file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #004: ../../src/H5Fint.c in H5F_try_close(): line 2181: problems closing file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #005: ../../src/H5Fint.c in H5F__dest(): line 1277: p
converted <- Convert("input2.h5ad", dest = "output2.h5seurat", overwrite = TRUE)
# Warning: Unknown file type: h5ad
# Warning: 'assay' not set, setting to 'RNA'
# Creating h5Seurat file for version 3.1.2
# Adding X as data
# Adding raw/X as counts
# Adding meta.features from raw/var
# Error in dfile$obj_copy_from(src_loc = dfile, src_name = paste0("meta.data/", :
# HDF5-API Errors:
# error #000: ../../src/H5Ocopy.c in H5Ocopy(): line 233: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #001: ../../src/H5Ocopy.c in H5O__copy(): line 314: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #002: ../../src/H5Ocopy.c in H5O__copy_obj(): line 1218: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #003: ../../src/H5Ocopy.c in H5O__copy_header(): line 1162: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #004: ../../src/H5Ocopy.c in H5O__copy_header_real(): line 887: unable to re-tag metadata entries
# class: HDF5
# major: Object cache
# minor: Unable to tag metadata in the cache
#
# error #005: ../../src/H5AC.c in H5AC_retag_copied_metadata(): line
# Error in private$closeFun(id) : HDF5-API Errors:
# error #000: ../../src/H5F.c in H5Fclose(): line 675: closing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #001: ../../src/H5Fint.c in H5F__close(): line 1952: decrementing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to decrement reference count
#
# error #002: ../../src/H5I.c in H5I_dec_app_ref(): line 1300: can't decrement ID ref count
# class: HDF5
# major: Object atom
# minor: Unable to decrement reference count
#
# error #003: ../../src/H5Fint.c in H5F__close_cb(): line 2010: can't close file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #004: ../../src/H5Fint.c in H5F_try_close(): line 2181: problems closing file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #005: ../../src/H5Fint.c in H5F__dest(): line 1277: p
Execution environment
$ h5cc -showconfig
SUMMARY OF THE HDF5 CONFIGURATION
=================================
General Information:
-------------------
HDF5 Version: 1.10.5
Configured on: Wed Jan 29 04:16:03 UTC 2020
Configured by: mockbuild@
Host system: x86_64-redhat-linux-gnu
Uname information: Linux buildvm-24.phx2.fedoraproject.org 5.4.8-200.fc31.x86_64 #1 SMP Mon Jan 6 16:44:18 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Byte sex: little-endian
Installation point: /usr
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Fedora 32 (Workstation Edition)
Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblas-r0.3.9.so
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8 LC_MONETARY=en_GB.UTF-8
[6] LC_MESSAGES=en_GB.UTF-8 LC_PAPER=en_GB.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] SeuratDisk_0.0.0.9010
loaded via a namespace (and not attached):
[1] httr_1.4.1 tidyr_1.1.0 bit64_0.9-7 hdf5r_1.3.2 jsonlite_1.7.0 viridisLite_0.3.0 splines_4.0.2
[8] leiden_0.3.3 assertthat_0.2.1 ggrepel_0.8.2 globals_0.12.5 pillar_1.4.4 lattice_0.20-41 glue_1.4.1
[15] reticulate_1.16 digest_0.6.25 RColorBrewer_1.1-2 colorspace_1.4-1 cowplot_1.0.0 htmltools_0.5.0 Matrix_1.2-18
[22] plyr_1.8.6 pkgconfig_2.0.3 tsne_0.1-3 listenv_0.8.0 purrr_0.3.4 patchwork_1.0.1 scales_1.1.1
[29] RANN_2.6.1 Rtsne_0.15 tibble_3.0.1 generics_0.0.2 ggplot2_3.3.2 ellipsis_0.3.1 withr_2.2.0
[36] ROCR_1.0-11 pbapply_1.4-2 lazyeval_0.2.2 cli_2.0.2 survival_3.1-12 magrittr_1.5 crayon_1.3.4
[43] fansi_0.4.1 future_1.17.0 nlme_3.1-148 MASS_7.3-51.6 ica_1.0-2 tools_4.0.2 fitdistrplus_1.1-1
[50] data.table_1.12.8 lifecycle_0.2.0 stringr_1.4.0 plotly_4.9.2.1 munsell_0.5.0 cluster_2.1.0 irlba_2.3.3
[57] compiler_4.0.2 rsvd_1.0.3 rlang_0.4.6 grid_4.0.2 ggridges_0.5.2 rstudioapi_0.11 RcppAnnoy_0.0.16
[64] rappdirs_0.3.1 htmlwidgets_1.5.1 igraph_1.2.5 Seurat_3.1.5 gtable_0.3.0 codetools_0.2-16 reshape2_1.4.4
[71] R6_2.4.1 gridExtra_2.3 zoo_1.8-8 dplyr_1.0.0 bit_1.1-15.2 future.apply_1.6.0 uwot_0.1.8
[78] KernSmooth_2.23-17 ape_5.4 stringi_1.4.6 parallel_4.0.2 Rcpp_1.0.4.6 vctrs_0.3.1 sctransform_0.2.1
[85] png_0.1-7 tidyselect_1.1.0 lmtest_0.9-37
The issue isn't related to R 4.0, since I'm able to replicate it in rocker/tidyverse:3.6.3:
Replicate issue in Docker
$ docker pull rocker/tidyverse:3.6.3
$ docker run -it -v `pwd`:/mount --entrypoint bash rocker/tidyverse:3.6.3
$ cd /mount
$ sudo apt-get install libhdf5-dev
$ h5cc -showconfig | grep Version
HDF5 Version: 1.10.4
$ R
> install.packages("hdf5r")
> remotes::install_github("mojaveazure/seurat-disk")
> library(SeuratDisk)
> converted <- Convert("input2.h5ad", dest = "output2.h5seurat", overwrite = TRUE)
# Warning: Unknown file type: h5ad
# Warning: 'assay' not set, setting to 'RNA'
# Creating h5Seurat file for version 3.1.2
# Adding X as data
# Adding raw/X as counts
# Adding meta.features from raw/var
# Error in dfile$obj_copy_from(src_loc = dfile, src_name = paste0("meta.data/", :
# HDF5-API Errors:
# error #000: ../../../src/H5Ocopy.c in H5Ocopy(): line 233: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #001: ../../../src/H5Ocopy.c in H5O__copy(): line 317: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #002: ../../../src/H5Ocopy.c in H5O__copy_obj(): line 1221: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #003: ../../../src/H5Ocopy.c in H5O__copy_header(): line 1165: unable to copy object
# class: HDF5
# major: Object header
# minor: Unable to copy object
#
# error #004: ../../../src/H5Ocopy.c in H5O__copy_header_real(): line 890: unable to re-tag metadata entries
# class: HDF5
# major: Object cache
# minor: Unable to tag metadata in the cache
#
# error #005: ../../../src/H5AC.c in H5AC_retag_copie
# Error in private$closeFun(id) : HDF5-API Errors:
# error #000: ../../../src/H5F.c in H5Fclose(): line 668: closing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #001: ../../../src/H5Fint.c in H5F__close(): line 2047: decrementing file ID failed
# class: HDF5
# major: File accessibilty
# minor: Unable to decrement reference count
#
# error #002: ../../../src/H5I.c in H5I_dec_app_ref(): line 1309: can't decrement ID ref count
# class: HDF5
# major: Object atom
# minor: Unable to decrement reference count
#
# error #003: ../../../src/H5Fint.c in H5F__close_cb(): line 2105: can't close file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #004: ../../../src/H5Fint.c in H5F_try_close(): line 2276: problems closing file
# class: HDF5
# major: File accessibilty
# minor: Unable to close file
#
# error #005: ../../../src/H5Fint.c in H5F__de
> sessionInfo()
# R version 3.6.3 (2020-02-29)
# Platform: x86_64-pc-linux-gnu (64-bit)
# Running under: Debian GNU/Linux 10 (buster)
#
# Matrix products: default
# BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so
#
# locale:
# [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
# [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
# [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=C
# [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
# [9] LC_ADDRESS=C LC_TELEPHONE=C
# [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#
# attached base packages:
# [1] stats graphics grDevices utils datasets methods base
#
# other attached packages:
# [1] SeuratDisk_0.0.0.9010
#
# loaded via a namespace (and not attached):
# [1] httr_1.4.1 tidyr_1.0.2 bit64_0.9-7
# [4] hdf5r_1.3.2 jsonlite_1.6.1 viridisLite_0.3.0
# [7] splines_3.6.3 lsei_1.2-0 leiden_0.3.3
# [10] gtools_3.8.2 assertthat_0.2.1 ggrepel_0.8.2
# [13] globals_0.12.5 pillar_1.4.3 lattice_0.20-38
# [16] glue_1.4.0 reticulate_1.15 digest_0.6.25
# [19] RColorBrewer_1.1-2 colorspace_1.4-1 cowplot_1.0.0
# [22] htmltools_0.4.0 Matrix_1.2-18 plyr_1.8.6
# [25] pkgconfig_2.0.3 tsne_0.1-3 listenv_0.8.0
# [28] purrr_0.3.4 patchwork_1.0.0 scales_1.1.0
# [31] RANN_2.6.1 gdata_2.18.0 Rtsne_0.15
# [34] tibble_3.0.1 ggplot2_3.3.0 ellipsis_0.3.0
# [37] withr_2.2.0 ROCR_1.0-7 pbapply_1.4-2
# [40] lazyeval_0.2.2 cli_2.0.2 survival_3.1-8
# [43] magrittr_1.5 crayon_1.3.4 fansi_0.4.1
# [46] future_1.17.0 nlme_3.1-144 MASS_7.3-51.5
# [49] gplots_3.0.3 ica_1.0-2 tools_3.6.3
# [52] fitdistrplus_1.0-14 data.table_1.12.8 lifecycle_0.2.0
# [55] stringr_1.4.0 plotly_4.9.2.1 munsell_0.5.0
# [58] cluster_2.1.0 irlba_2.3.3 compiler_3.6.3
# [61] rsvd_1.0.3 caTools_1.18.0 rlang_0.4.5
# [64] grid_3.6.3 ggridges_0.5.2 RcppAnnoy_0.0.16
# [67] rappdirs_0.3.1 htmlwidgets_1.5.1 igraph_1.2.5
# [70] bitops_1.0-6 Seurat_3.1.5 npsurv_0.4-0
# [73] gtable_0.3.0 codetools_0.2-16 reshape2_1.4.4
# [76] R6_2.4.1 gridExtra_2.3 zoo_1.8-7
# [79] dplyr_0.8.5 bit_1.1-15.2 uwot_0.1.8
# [82] future.apply_1.5.0 KernSmooth_2.23-16 ape_5.3
# [85] stringi_1.4.6 parallel_3.6.3 Rcpp_1.0.4.6
# [88] vctrs_0.2.4 sctransform_0.2.1 png_0.1-7
# [91] tidyselect_1.0.0 lmtest_0.9-37
I did manage to find a workaround by closing and re-opening the output file right before the error occurs, though this is nothing more than a temporary hack, and a real solution should be found.
Do you have any ideas that could solve this problem?
Kind regards, Robrecht
Hello again!
I had the same issue in some other function as well. After some more digging, it seems that the error occurs specifically when executing h5$copy_obj_to(...) calls. For now, I simply commented out all calls to this function and provided a small workaround (which will be slower to run but does seem to work on all platforms). For example:
# temporary workaround (see mojaveazure/seurat-disk#10)
# source[['graphs']]$obj_copy_to(
# dst_loc = dgraph,
# dst_name = 'distances',
# src_name = graph
# )
dgraph[["distances"]] <- source[[paste0("graphs/", graph)]][]
# end temporary workaround
I'm still trying to determine when the problem occurs and when it doesn't. I've heard reports that this bug does not occur with HDF5 1.10.2 and have found that it does occur for versions 1.10.4 and 1.10.5, but I have not managed to replicate an environment containing HDF5 1.10.2 to see if it indeed does or does not occur there.
@rcannood have you found a solution for this, I face the same problem when I am trying to convert anndata object to seurat with convert() ? thanks in advance
