Segfault when copying dataset with attributes

[takluyver]

Describe the bug

We've come across a case where H5Ocopy causes a segfault when copying an object with attributes. We originally came across this via h5py, but I can replicate it with the h5copy command:

h5copy -i repacked2.h5 -o test2.h5 -s value -d value

Here is the input file to reproduce this (zipped to please Github): repacked2.zip

I suspect that this is related to dense attribute storage - when deleting attributes, I could reproduce it with 6 attributes, but not with 4 (6 is the default min_dense, so below this it will switch back to compound storage). Passing -f noattr to h5copy avoids the error.

I haven't been yet been able to reproduce it with a newly created file - the example file above is a heavily cut down version of a file from the European XFEL data acquisition system. I believe it's a valid HDF5 file - I can read all of the attributes with no errors. And ideally even if the file is corrupt, we'd like HDF5 to catch that without segfaulting.

We saw a segfault here on various HDF5 versions (1.10.6, 1.12.1, 1.12.2, 1.14.0), but I've focused on 1.14.0 for investigation.

GDB Backtrace from h5copy 1.14.0

#0  0x00002aaaab271ec4 in H5T__conv_vlen (src_id=216172782113784125, dst_id=216172782113784126, cdata=0x699580, 
    nelmts=1, buf_stride=0, bkg_stride=0, buf=0x69ae78, bkg=0x0) at H5Tconv.c:3296
#1  0x00002aaaab256178 in H5T_convert (tpath=0x699510, src_id=216172782113784125, dst_id=216172782113784126, 
    nelmts=1, buf_stride=0, bkg_stride=0, buf=0x69ae78, bkg=0x0) at H5T.c:5449
#2  0x00002aaaaad5cc27 in H5A__attr_copy_file (attr_src=0x697830, file_dst=0x682710, 
    recompute_size=0x7fffffffd2ff, cpy_info=0x7fffffffd5a0) at H5Aint.c:2367
#3  0x00002aaaaad5dbbe in H5A__dense_post_copy_file_cb (attr_src=0x697830, _udata=0x7fffffffd310) at H5Aint.c:2556
#4  0x00002aaaaad4c930 in H5A__dense_iterate_bt2_cb (_record=0x696ac8, _bt2_udata=0x7fffffffd210)
    at H5Adense.c:1098
#5  0x00002aaaaad8e498 in H5B2__iterate_node (hdr=0x695ef0, depth=0, curr_node=0x696000, parent=0x695ef0, 
    op=0x2aaaaad4c51c <H5A__dense_iterate_bt2_cb>, op_data=0x7fffffffd210) at H5B2int.c:1702
#6  0x00002aaaaad75cbe in H5B2_iterate (bt2=0x695640, op=0x2aaaaad4c51c <H5A__dense_iterate_bt2_cb>, 
    op_data=0x7fffffffd210) at H5B2.c:423
#7  0x00002aaaaad4d0ba in H5A__dense_iterate (f=0x67be60, loc_id=0, ainfo=0x687d80, idx_type=H5_INDEX_NAME, 
    order=H5_ITER_NATIVE, skip=0, last_attr=0x0, attr_op=0x7fffffffd300, op_data=0x7fffffffd310)
    at H5Adense.c:1225
#8  0x00002aaaaad5e150 in H5A__dense_post_copy_file_all (src_oloc=0x7fffffffd720, ainfo_src=0x687d80, 
    dst_oloc=0x7fffffffd670, ainfo_dst=0x694880, cpy_info=0x7fffffffd5a0) at H5Aint.c:2621
#9  0x00002aaaab092095 in H5O__ainfo_post_copy_file (src_oloc=0x7fffffffd720, mesg_src=0x687d80, 
    dst_oloc=0x7fffffffd670, mesg_dst=0x694880, mesg_flags=0x7fffffffd424, cpy_info=0x7fffffffd5a0)
    at H5Oainfo.c:468
#10 0x00002aaaab0bbe3f in H5O__copy_header_real (oloc_src=0x7fffffffd720, oloc_dst=0x7fffffffd670, 
    cpy_info=0x7fffffffd5a0, obj_type=0x0, udata=0x0) at H5Ocopy.c:719
#11 0x00002aaaab0bcfff in H5O__copy_header (oloc_src=0x7fffffffd720, oloc_dst=0x7fffffffd670, 
    ocpypl_id=792633534417207348, lcpl_id=792633534417207349) at H5Ocopy.c:1008
#12 0x00002aaaab0bd3cd in H5O__copy_obj (src_loc=0x7fffffffd760, dst_loc=0x7fffffffd7d0, 
    dst_name=0x67a750 "value", ocpypl_id=792633534417207348, lcpl_id=792633534417207349) at H5Ocopy.c:1062
#13 0x00002aaaab0b9585 in H5O__copy (loc=0x7fffffffd7e0, src_name=0x67a730 "value", dst_loc=0x7fffffffd7d0, 
    dst_name=0x67a750 "value", ocpypl_id=792633534417207348, lcpl_id=792633534417207349) at H5Ocopy.c:167
#14 0x00002aaaab3b981c in H5VL__native_object_copy (src_obj=0x67be60, loc_params1=0x7fffffffd950, 
    src_name=0x67a730 "value", dst_obj=0x682710, loc_params2=0x7fffffffd980, dst_name=0x67a750 "value", 
    ocpypl_id=792633534417207348, lcpl_id=792633534417207349, dxpl_id=792633534417207304, req=0x0)
    at H5VLnative_object.c:155
#15 0x00002aaaab38f617 in H5VL__object_copy (src_obj=0x67be60, src_loc_params=0x7fffffffd950, 
    src_name=0x67a730 "value", dst_obj=0x682710, dst_loc_params=0x7fffffffd980, dst_name=0x67a750 "value", 
    cls=0x642150, ocpypl_id=792633534417207348, lcpl_id=792633534417207349, dxpl_id=792633534417207304, req=0x0)
    at H5VLcallback.c:5848
---Type <return> to continue, or q <return> to quit---
#16 0x00002aaaab38f8b9 in H5VL_object_copy (src_obj=0x681e60, src_loc_params=0x7fffffffd950, 
    src_name=0x67a730 "value", dst_obj=0x684d90, dst_loc_params=0x7fffffffd980, dst_name=0x67a750 "value", 
    ocpypl_id=792633534417207348, lcpl_id=792633534417207349, dxpl_id=792633534417207304, req=0x0)
    at H5VLcallback.c:5887
#17 0x00002aaaab07f7e6 in H5O__copy_api_common (src_loc_id=72057594037927936, src_name=0x67a730 "value", 
    dst_loc_id=72057594037927937, dst_name=0x67a750 "value", ocpypl_id=792633534417207348, 
    lcpl_id=792633534417207349, token_ptr=0x0, _vol_obj_ptr=0x0) at H5O.c:461
#18 0x00002aaaab07fb22 in H5Ocopy (src_loc_id=72057594037927936, src_name=0x67a730 "value", 
    dst_loc_id=72057594037927937, dst_name=0x67a750 "value", ocpypl_id=792633534417207348, 
    lcpl_id=792633534417207349) at H5O.c:553
#19 0x000000000040690e in main (argc=9, argv=0x7fffffffdc68) at h5copy.c:458

Expected behavior

Not segfaulting. :wink:

Platform (please complete the following information)

• HDF5 version (if building from a maintenance branch, please include the commit hash): 1.14.0
• OS and version: CentOS 7
• Compiler and version: gcc 4.8.5
• Build system (e.g. CMake, Autotools) and version: Autotools
• Any configure options you specified: --enable-build-mode=debug

[takluyver]

HDF group made a JIRA issue for this: https://jira.hdfgroup.org/browse/HDFFV-11360

[derobins]

We're going to be moving our product-specific project planning to GitHub, so I've added this to the 1.14.1 release project.

[kyang2014]

Maybe this is known. I just tried 1.14.2 with the same file. It is still seg fault.

 "./h5copy -i repacked2.h5 -o test2.h5 -s value -d value
Segmentation fault (core dumped)"

valgrind doesn't give me what I want. However, it outputs the following message:

"h5copy error: Could not open output file <test2.h5>...Exiting"

[ajelenak]

h5copy -i repacked2.h5 -o test2.h5 -s value -d value

Works for me with 1.14.4.

[takluyver]

Thanks!