aldanor
Confusing error when creating a dataset on top of existing
Hi there,
First, thanks for all of your work on this library. There's obviously been a lot of thought and time put into it.
Secondly, I am trying to store some Monte Carlo data in hdf5 using your work: I'm trying to serialize 1-dimension arrays of f64 data. I'm running on OpenSuSE Tumbleweed, hdf5-devel version 1.10.7 and blosc-devel version 1.21.1. When I go about actually serializing the data either an attribute or a dataset, the unwrap fails with nothing more than unknown library error.
Even with RUST_BACKTRACE=full and running a debug build, I get no pointers as to where in the hdf5 library the error happens (full backtrace at the bottom of this issue). The exact line of the error happens either on the attribute or dataset writing, whichever happens first, e.g.
let _ds = builder
.with_data(&arr1(&data))
// finalize and write the dataset
.create(&*format!("Dispersion {}", param))
.unwrap();
Since this is my first time using this library, it's quite likely that I'm using it incorrectly. At the top of my serialization function, I call blosc_set_nthreads(4); and then, every builder initialization is like so: let builder = run_group.new_dataset_builder().blosc_zstd(9, true);
Solution
I've found that by removing all references to blosc, the serialization is successful, and all of the data is present.
Backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: unknown library error', src/serialize.rs:140:14
stack backtrace:
0: 0x55e123f3780c - std::backtrace_rs::backtrace::libunwind::trace::h09f7e4e089375279
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x55e123f3780c - std::backtrace_rs::backtrace::trace_unsynchronized::h1ec96f1c7087094e
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x55e123f3780c - std::sys_common::backtrace::_print_fmt::h317b71fc9a5cf964
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:67:5
3: 0x55e123f3780c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he3555b48e7dfe7f0
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:46:22
4: 0x55e123f5a93c - core::fmt::write::h513b07ca38f4fb1b
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/fmt/mod.rs:1149:17
5: 0x55e123f33ca5 - std::io::Write::write_fmt::haf8c932b52111354
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/io/mod.rs:1697:15
6: 0x55e123f393f0 - std::sys_common::backtrace::_print::h195c38364780a303
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:49:5
7: 0x55e123f393f0 - std::sys_common::backtrace::print::hc09dfdea923b6730
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:36:9
8: 0x55e123f393f0 - std::panicking::default_hook::{{closure}}::hb2e38ec0d91046a3
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:211:50
9: 0x55e123f38fa5 - std::panicking::default_hook::h60284635b0ad54a8
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:228:9
10: 0x55e123f39aa4 - std::panicking::rust_panic_with_hook::ha677a669fb275654
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:606:17
11: 0x55e123f39580 - std::panicking::begin_panic_handler::{{closure}}::h976246fb95d93c31
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:502:13
12: 0x55e123f37cb4 - std::sys_common::backtrace::__rust_end_short_backtrace::h38077ee5b7b9f99a
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:139:18
13: 0x55e123f394e9 - rust_begin_unwind
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
14: 0x55e1235a26f1 - core::panicking::panic_fmt::h35f3a62252ba0fd2
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
15: 0x55e1235a27e3 - core::result::unwrap_failed::hb53671404b9e33c2
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/result.rs:1613:5
16: 0x55e1235f3019 - core::result::Result<T,E>::unwrap::h872e53f299f901b9
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/result.rs:1295:23
17: 0x55e1235e9218 - nyxpeg::serialize::serialize::h88b1d40028d049f1
at /home/chris/redacted/src/serialize.rs:136:19
18: 0x55e1235cd5b2 - montecarlo::main::hfe86ac0737a16e8e
at /home/chris/redacted/src/bin/montecarlo/main.rs:132:5
19: 0x55e1235d0612 - core::ops::function::FnOnce::call_once::h3f8cde54c75e2f42
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:227:5
20: 0x55e1235ba04a - std::sys_common::backtrace::__rust_begin_short_backtrace::h2bd796ed255e3d99
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:123:18
21: 0x55e1235e119c - std::rt::lang_start::{{closure}}::hc8d32e34b8ecbcff
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:145:18
22: 0x55e123f3749b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h7e688d7cdfeb7e00
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:259:13
23: 0x55e123f3749b - std::panicking::try::do_call::h4be824d2350b44c9
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
24: 0x55e123f3749b - std::panicking::try::h0a6fc7affbe5088d
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
25: 0x55e123f3749b - std::panic::catch_unwind::h22c320f732ec805e
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
26: 0x55e123f3749b - std::rt::lang_start_internal::{{closure}}::hd38309c108fe679d
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:128:48
27: 0x55e123f3749b - std::panicking::try::do_call::h8fcaf501f097a28e
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
28: 0x55e123f3749b - std::panicking::try::h20e906825f98acc1
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
29: 0x55e123f3749b - std::panic::catch_unwind::h8c5234dc632124ef
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
30: 0x55e123f3749b - std::rt::lang_start_internal::hc4dd8cd3ec4518c2
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:128:20
31: 0x55e1235e1160 - std::rt::lang_start::h118c7256a7dd8f86
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:144:17
32: 0x55e1235cd6fc - main
33: 0x7f8178f295c0 - __libc_start_call_main
34: 0x7f8178f2967c - __libc_start_main_alias_2
35: 0x55e1235a2eb5 - _start
at /home/abuild/rpmbuild/BUILD/glibc-2.34/csu/../sysdeps/x86_64/start.S:116
36: 0x0 - <unknown>
Does the simple example of this crate compile and run for you? I.e. cargo r --example simple --features blosc
Yes, it does.
So after returning to the code about thirty minutes ago, it's become clear that the issue was when I was trying to add a dataset of name X in group Y when a dataset of the same name and in the same group already existed. I called an unwrap() on that call and that triggered the error above. By checking the existence, that issue was resolved.
So I guess that my only recommendation, if at all possible, would be to specify what function call caused the error and maybe add a tip to why the call failed.
Specifically, the following call failed:
let builder = dispersions_group.new_dataset_builder();
let _ds = builder
.with_data(&arr1(&data))
// finalize and write the dataset
.create(&*format!("Dispersion {}", param))
.unwrap();
It would have been helpful if that runtime failure would report `error on creation of dataset "Disperion blah" (tip: does this dataset already exist). Now, I have no idea whether that is possible to given that you're plugging into the C bindings.
with_data is a wrapper around many calls, it would indeed be nice if error at the specific layer returned such a hint. We have discussed introducing some helper methods for dataset creating, e.g. #151, which could mimimise the confusion a bit.