noria-mysql
noria-mysql copied to clipboard
Published
20 hours ago •
mit-pdos
mit-pdos
Problems starting basic example
Hi, I'm trying to get Noria working with JDBC. I'm aware of JDBC issues, I thought I'd take a stab at seeing where it fails, so I looked at issue #15 but I fail to even get the basics working. Hopefully I'm doing something wrong.
I'm starting a zookeeper instance:
docker run --rm -it --name some-zookeeper -p 2181:2181 zookeeper
Seems alright. In a new shell, clean noria checkout:
cargo r --release --bin noria-server -- --deployment myapp --no-reuse --shards 0
Seems ok, in another shell, in a clean noria-mysql checkout:
cargo run --release -- --deployment myapp -z 127.0.0.1:2181
Starts up fine. Then in a final terminal:
mysql -h 127.0.0.1
Seems to work:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.1.10-alpha-msql-proxy
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
Create a table (like in #15):
CREATE TABLE Cats (id int, name VARCHAR(255), PRIMARY KEY(id));
Query OK, 0 rows affected (0.01 sec)
But the next statement fails:
mysql> INSERT INTO Cats (id, name) VALUES (2, "Burt");
ERROR 1105 (HY000): TransportError(ClientDropped
0: backtrace::backtrace::trace
1: backtrace::capture::Backtrace::new_unresolved
2: failure::backtrace::internal::InternalBacktrace::new
3: failure::error::Error::from_boxed_compat
4: <noria::table::TableError as core::convert::From<alloc::boxed::Box<dyn std::error::Error+core::marker::Send+core::marker::Sync>>>::from
5: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
6: <core::future::from_generator::GenFuture<T> as core::
mysql>
noria-mysql trace:
thread 'tokio-runtime-worker' panicked at 'BrokenTransportRecv', /Users/frank/.cargo/git/checkouts/noria-8612be89bc6721f9/e427387/noria/src/table.rs:237:62
stack backtrace:
151.517275884s ERROR noria_mysql::backend: failed error=Client was dropped
0: 0x101dcc8b4 - std::backtrace_rs::backtrace::libunwind::trace::he9497f4525a08d2b
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
1: 0x101dcc8b4 - std::backtrace_rs::backtrace::trace_unsynchronized::h10a0a59d96f4a16b
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/../../backtrace/src/backtrace/mod.rs:66
2: 0x101dcc8b4 - std::sys_common::backtrace::_print_fmt::h6fd8fbf82b7783c5
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:79
3: 0x101dcc8b4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hfb2db7374ccd4d34
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:58
4: 0x101debfec - core::fmt::write::h7e381316a10f5e82
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/fmt/mod.rs:1117
5: 0x101dc6629 - std::io::Write::write_fmt::h53862e22ed93af5e
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/io/mod.rs:1510
6: 0x101dce525 - std::sys_common::backtrace::_print::h5642011e0f38c919
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:61
7: 0x101dce525 - std::sys_common::backtrace::print::hbcc5d5a374cdc250
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:48
8: 0x101dce525 - std::panicking::default_hook::{{closure}}::h1a87f1c0565a051b
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:200
9: 0x101dce262 - std::panicking::default_hook::he29ab2038604c86f
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:219
10: 0x101dceac5 - std::panicking::rust_panic_with_hook::h9dab6d214201c5da
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:569
11: 0x101dce65b - std::panicking::begin_panic_handler::{{closure}}::hf4bcec37fb7265f5
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:476
12: 0x101dcccf8 - std::sys_common::backtrace::__rust_end_short_backtrace::h8c49942bd9c628f8
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:153
13: 0x101dce61a - rust_begin_unwind
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:475
14: 0x101e0d84b - std::panicking::begin_panic_fmt::h769a003a4fabac0e
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:429
15: 0x101b9a912 - <noria::table::TableEndpoint as tower_service::Service<()>>::call::{{closure}}::{{closure}}::head2a7e0d2cc3450
at /Users/frank/.cargo/git/checkouts/noria-8612be89bc6721f9/e427387/noria/src/table.rs:237
16: 0x101b9a912 - tokio_tower::multiplex::client::Client<T,E,Request>::with_error_handler::{{closure}}::ha8bb50bc2271906b
at /Users/frank/.cargo/git/checkouts/tokio-tower-98a883543a8cb142/e58bb90/src/multiplex/client.rs:197
17: 0x101b9a912 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he442d7d330fcd79a
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/future/mod.rs:78
18: 0x101b3c1c9 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h3d742ba66b05f6e6
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:163
19: 0x101b3c1c9 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h38277425b57d9489
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/loom/std/unsafe_cell.rs:14
20: 0x101b3c1c9 - tokio::runtime::task::core::Core<T,S>::poll::h9f81880f2d783c9d
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:148
21: 0x101b4d4bd - tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::hc2cb169c4543b548
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:108
22: 0x101b4d4bd - core::ops::function::FnOnce::call_once::ha83341f8a50e8565
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/ops/function.rs:233
23: 0x101b4d4bd - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h5f2483b0736864a1
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:318
24: 0x101b92bb8 - std::panicking::try::do_call::h2863e39e882eb1e0
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
25: 0x101b92bb8 - std::panicking::try::hd3cd23ca9b4aac77
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
26: 0x101b92bb8 - std::panic::catch_unwind::h85aaa291c5135e2c
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
27: 0x101b92bb8 - tokio::runtime::task::harness::Harness<T,S>::poll::h4598909b59fa3963
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:84
28: 0x101d2a15e - tokio::runtime::task::raw::RawTask::poll::h84b006a1e3342a09
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/raw.rs:66
29: 0x101d2a15e - tokio::runtime::task::Notified<S>::run::h46ad02afe7f7542b
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/mod.rs:169
30: 0x101d2a15e - tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}::h4de65ade9c6380b1
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:308
31: 0x101d2a01c - tokio::coop::budget::{{closure}}::h161591f49f5d979a
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/coop.rs:97
32: 0x101d2a01c - std::thread::local::LocalKey<T>::try_with::ha2ee22c28756a0b2
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/local.rs:267
33: 0x101d2a01c - std::thread::local::LocalKey<T>::with::h5ca0065eec38a20c
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/local.rs:243
34: 0x101d2a01c - tokio::coop::budget::h238e8b9c04d253e0
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/coop.rs:79
35: 0x101d2a01c - tokio::runtime::thread_pool::worker::Context::run_task::ha306ca1dcf61ac54
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:307
36: 0x101d297ce - tokio::runtime::thread_pool::worker::Context::run::h2fffcc7ed0a4f790
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:279
37: 0x101d19e63 - tokio::runtime::thread_pool::worker::run::{{closure}}::h90c666296baf9cc6
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:264
38: 0x101d19e63 - tokio::macros::scoped_tls::ScopedKey<T>::set::h54e39d6f67b0a67d
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/macros/scoped_tls.rs:64
39: 0x101d292c4 - tokio::runtime::thread_pool::worker::run::h1bfb873f95891b28
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:261
40: 0x101d0ede3 - tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}::hbbcaf2cabc73169d
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/thread_pool/worker.rs:240
41: 0x101d0ede3 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h88b256f57da30ac6
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/task.rs:38
42: 0x101d0ede3 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h34ee386b95d91512
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:163
43: 0x101d0ede3 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h13a5f633ba7f2696
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/loom/std/unsafe_cell.rs:14
44: 0x101d2e2a7 - tokio::runtime::task::core::Core<T,S>::poll::hc9e3fae42dd9cfb9
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/core.rs:148
45: 0x101d2e2a7 - tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::h84d381f280474d6b
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:108
46: 0x101d2e2a7 - core::ops::function::FnOnce::call_once::h6ac9ab05edebdd81
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/ops/function.rs:233
47: 0x101d2e2a7 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h5286e8a3ca09e6f3
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:318
48: 0x101d0d351 - std::panicking::try::do_call::h1f1664dd3b5e303b
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
49: 0x101d0d351 - std::panicking::try::h929c6a8ee482c1d3
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
50: 0x101d0d351 - std::panic::catch_unwind::h1a2e1c6a3c6621a3
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
51: 0x101d0d351 - tokio::runtime::task::harness::Harness<T,S>::poll::h36246b0d5d13a035
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/harness.rs:84
52: 0x101d0feff - tokio::runtime::task::raw::RawTask::poll::h84b006a1e3342a09
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/raw.rs:66
53: 0x101d0feff - tokio::runtime::task::Notified<S>::run::h7bbcf40f3dff391d
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/task/mod.rs:169
54: 0x101d0feff - tokio::runtime::blocking::pool::Inner::run::hae12476ec7e28d9e
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/pool.rs:230
55: 0x101d1af82 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}}::h98f502aeb1ad0003
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/pool.rs:210
56: 0x101d1af82 - tokio::runtime::context::enter::h3c16f4071e58296d
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/context.rs:72
57: 0x101d194fe - tokio::runtime::handle::Handle::enter::h06b6d376502de6b9
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/handle.rs:39
58: 0x101d194fe - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hb8e3fa9f6c1ffde2
at /Users/frank/.cargo/git/checkouts/tokio-377c595163f99a10/3137c6f/tokio/src/runtime/blocking/pool.rs:209
59: 0x101d194fe - std::sys_common::backtrace::__rust_begin_short_backtrace::hb72a8072285616e3
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys_common/backtrace.rs:137
60: 0x101d1cec1 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::hd81198c5ab0d100d
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/mod.rs:475
61: 0x101d1cec1 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h545f887c76e457c1
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:318
62: 0x101d1cec1 - std::panicking::try::do_call::hfcbf1e36cf2b3658
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:373
63: 0x101d1cec1 - std::panicking::try::hc1941a32d0ce03b0
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panicking.rs:337
64: 0x101d1cec1 - std::panic::catch_unwind::haba6d86a749d0033
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/panic.rs:394
65: 0x101d1cec1 - std::thread::Builder::spawn_unchecked::{{closure}}::h25d389d6eab7bd3a
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/thread/mod.rs:474
66: 0x101d1cec1 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h043a476368fcf197
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/core/src/ops/function.rs:233
67: 0x101dd19cd - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h85dfeaba80b7a665
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/boxed.rs:1025
68: 0x101dd19cd - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h05752f0c5860f139
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/alloc/src/boxed.rs:1025
69: 0x101dd19cd - std::sys::unix::thread::Thread::new::thread_start::h39b071b0d2fb7aea
at /rustc/7e6d6e5f535321c2223f044caba16f97b825009c/library/std/src/sys/unix/thread.rs:87
70: 0x7fff717ba109 - __pthread_start
Noria terminates with a segfault:
Segmentation fault: 11
Ideas?
Huh, that is very weird indeed. What platform are you on?
Can you try adding -v to where you launch noria-server? That should give you some more details about what it did just before it failed.
Also, can you try the griddle branch and see if the same issue occurs there?
The noria output doesn't add much:
Running `target/release/noria-server --deployment myapp --no-reuse --shards 0 -v -z '127.0.0.1:2181'`
Aug 17 20:48:53.223 INFO became leader at epoch 3
Aug 17 20:48:53.232 INFO found initial leader
Aug 17 20:48:53.232 INFO leader listening on external address 127.0.0.1:6033
Aug 17 20:48:53.233 INFO connected to controller, src: 127.0.0.1:52725
Aug 17 20:48:53.233 INFO listening for reads, on: 127.0.0.1:52726
Aug 17 20:48:53.233 WARN Connected to new leader
Aug 17 20:48:53.237 INFO new worker registered from 127.0.0.1:52725, which listens on 127.0.0.1:52723
Aug 17 20:49:28.533 INFO starting migration
Aug 17 20:49:28.533 INFO Schema version advanced from 0 to 1
Aug 17 20:49:28.536 INFO adding new base, node: 1
Aug 17 20:49:28.536 INFO finalizing migration, #nodes: 1
Aug 17 20:49:28.539 INFO sending domain 0.0 to worker Ok(127.0.0.1:52723)
Aug 17 20:49:28.542 INFO booted domain, nodes: 1, shard: 0, domain: 0
Aug 17 20:49:28.543 INFO informed controller that domain 0.0 is at 127.0.0.1:52735
Aug 17 20:49:28.543 INFO bringing up inter-domain connections
Aug 17 20:49:28.543 INFO initializing new materializations
Aug 17 20:49:28.544 INFO adding lookup index to view, columns: [0], node: 1
Aug 17 20:49:28.544 INFO no need to replay empty new base, node: 1
Aug 17 20:49:28.569 WARN migration completed, ms: 35
Segmentation fault: 11
I'll build the griddle branch and get back..
Ah, sorry, yes, you'll also want to swap the # on these two lines to enable trace logging: https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/server/Cargo.toml#L42-L41
Not much more info:
Aug 17 21:29:53.331 TRCE readying empty node, local: 0, shard: 0, domain: 0
Aug 17 21:29:53.332 TRCE node ready, node: 1
Aug 17 21:29:53.332 WARN migration completed, ms: 24
Aug 17 21:29:53.332 INFO starting migration
Aug 17 21:29:53.332 DEBG 1 queries, 0 of which are named, version: 5
Aug 17 21:29:53.332 INFO Schema version advanced from 4 to 5
Aug 17 21:29:53.332 INFO finalizing migration, #nodes: 0
Aug 17 21:29:53.332 DEBG booting new domains
Aug 17 21:29:53.332 DEBG mutating existing domains
Aug 17 21:29:53.332 INFO bringing up inter-domain connections
Aug 17 21:29:53.332 INFO initializing new materializations
Aug 17 21:29:53.332 WARN migration completed, ms: 0
Aug 17 21:29:53.332 INFO starting migration
Aug 17 21:29:53.332 DEBG 1 queries, 0 of which are named, version: 6
Aug 17 21:29:53.332 INFO Schema version advanced from 5 to 6
Aug 17 21:29:53.332 INFO finalizing migration, #nodes: 0
Aug 17 21:29:53.332 DEBG booting new domains
Aug 17 21:29:53.332 DEBG mutating existing domains
Aug 17 21:29:53.332 INFO bringing up inter-domain connections
Aug 17 21:29:53.332 INFO initializing new materializations
Aug 17 21:29:53.332 WARN migration completed, ms: 0
Aug 17 21:30:06.161 TRCE creating table, for: Cats
Aug 17 21:30:18.252 DEBG accepted new connection, from: 127.0.0.1:53190, id: 0.0
Aug 17 21:30:18.252 DEBG established new connection, base: true, id: 0.0
Segmentation fault: 11
It's on a macbook, rust version:
frank@MacBook:~/git/noria (griddle)$ rustc --version
rustc 1.47.0-nightly (7e6d6e5f5 2020-08-16)
That's really strange indeed. I haven't tested Noria much on macOS, as all the benchmarks and dev work I've done has been on Linux. If it works on macOS, it's by accident :sweat_smile: It certainly should not be segfaulting though. If you could figure out where it segfaults by running through gdb, that would help a lot in trying to nail this down!
Ok, first tried (without much success) to debug on Mac. Then, just to check, dusted off my linux box, and I have the exact same problem. I'll attempt debugging on the Linux box later, but for now I can say it doesn't seem to be Mac related.
Ok, I got the same issue now with gdb on my linux box, looks something like this:
Aug 22 16:28:34.597 TRCE creating table, for: Cats2
Aug 22 16:28:34.605 TRCE creating table, for: Cats3
Aug 22 16:28:36.342 DEBG accepted new connection, from: 127.0.0.1:58644, id: 1.0
Aug 22 16:28:36.342 DEBG established new connection, base: true, id: 1.0
--Type <RET> for more, q to quit, c to continue without paging--
Thread 8 "worker" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff43fd700 (LWP 3927862)]
0x00005555564ce92a in noria_dataflow::payload::Packet::dst (self=0x7fffde229500) at server/dataflow/src/payload.rs:246
246 Packet::Input { ref inner, .. } => unsafe { inner.deref() }.dst,
(gdb) where
#0 0x00005555564ce92a in noria_dataflow::payload::Packet::dst (self=0x7fffde229500) at server/dataflow/src/payload.rs:246
#1 0x00005555561a622a in noria_dataflow::group_commit::GroupCommitQueueSet::should_append (self=0x7fffdd653510, p=0x7fffde229500, nodes=0x7fffdd653238) at server/dataflow/src/group_commit.rs:24
#2 0x00005555561a4bfb in noria_dataflow::domain::Domain::on_event (self=0x7fffdd653218, executor=..., event=...) at server/dataflow/src/domain/mod.rs:3008
#3 0x0000555555bfbe71 in <noria_server::worker::replica::Replica as core::future::future::Future>::poll::{{closure}} (p=0x7fffde229500) at server/src/worker/replica.rs:640
#4 0x0000555555e5b5e9 in <noria_server::worker::replica::Replica as core::future::future::Future>::poll (self=..., cx=0x7ffff43f94d0) at server/src/worker/replica.rs:640
#5 0x000055555594ce98 in noria_server::worker::listen_df::{{closure}}::{{closure}}::{{closure}} () at /home/frank/git/noria/server/src/worker/mod.rs:341
#6 0x0000555555aaf1c9 in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll (self=..., cx=0x7ffff43f94d0) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/future/mod.rs:79
#7 0x00005555558a2609 in tokio::runtime::task::core::Core<T,S>::poll::{{closure}} (ptr=0x7fffdd652c38) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
#8 0x00005555559facd7 in tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut (self=0x7fffdd652c38, f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
#9 0x00005555558a1323 in tokio::runtime::task::core::Core<T,S>::poll (self=0x7fffdd652c30, header=0x7fffdd652c00) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
#10 0x0000555555839a3c in tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
#11 0x000055555599c270 in core::ops::function::FnOnce::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/ops/function.rs:233
#12 0x000055555595f65a in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=()) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:318
#13 0x0000555555a35c4d in std::panicking::try::do_call (data=0x7ffff43f9850 "ș?\364\377\177\000") at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:373
#14 0x0000555555a3a75d in __rust_try ()
#15 0x0000555555a32b75 in std::panicking::try (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:337
#16 0x00005555559610aa in std::panic::catch_unwind (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:394
#17 0x0000555555829c89 in tokio::runtime::task::harness::Harness<T,S>::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
#18 0x00005555559de320 in tokio::runtime::task::raw::poll (ptr=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
#19 0x000055555755894f in tokio::runtime::task::raw::RawTask::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
#20 0x0000555557531601 in tokio::runtime::task::Notified<S>::run (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
#21 0x0000555557544042 in tokio::runtime::thread_pool::worker::Context::run_task::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:353
#22 0x0000555557524546 in tokio::coop::with_budget::{{closure}} (cell=0x7ffff43fd2ca) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:127
#23 0x00005555575488b0 in std::thread::local::LocalKey<T>::try_with (self=0x55555809b128, f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/local.rs:267
#24 0x0000555557547b7e in std::thread::local::LocalKey<T>::with (self=0x55555809b128, f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/local.rs:243
#25 0x0000555557543ea4 in tokio::coop::with_budget (budget=..., f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:120
#26 tokio::coop::budget (f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:96
#27 tokio::runtime::thread_pool::worker::Context::run_task (self=0x7ffff43fa138, task=..., core=0x7ffff7617640) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:352
#28 0x0000555557543856 in tokio::runtime::thread_pool::worker::Context::run (self=0x7ffff43fa138, core=0x7ffff7617640) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:324
#29 0x0000555557543643 in tokio::runtime::thread_pool::worker::run::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:309
#30 0x0000555557523426 in tokio::macros::scoped_tls::ScopedKey<T>::set (self=0x55555809b550 <tokio::runtime::thread_pool::worker::CURRENT>, t=0x7ffff43fa138, f=...)
at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/macros/scoped_tls.rs:63
#31 0x000055555754353f in tokio::runtime::thread_pool::worker::run (worker=Arc(strong=8, weak=0) = {...}) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:306
#32 0x0000555555b438db in tokio::runtime::thread_pool::worker::block_in_place::{{closure}}::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:259
#33 0x00005555558d372b in <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll (self=..., _cx=0x7ffff43fa2b0)
at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/task.rs:41
#34 0x00005555558a3e79 in tokio::runtime::task::core::Core<T,S>::poll::{{closure}} (ptr=0x7ffff7636130) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
#35 0x00005555559fdc77 in tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut (self=0x7ffff7636130, f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
#36 0x00005555558a1dcd in tokio::runtime::task::core::Core<T,S>::poll (self=0x7ffff7636130, header=0x7ffff7636100) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
#37 0x000055555583c79c in tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
#38 0x000055555599ca70 in core::ops::function::FnOnce::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/ops/function.rs:233
#39 0x000055555595fb5a in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=()) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:318
#40 0x0000555555a3703d in std::panicking::try::do_call (data=0x7ffff43fa630 "\250\247?\364\377\177\000") at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:373
#41 0x0000555555a3a75d in __rust_try ()
#42 0x0000555555a300b5 in std::panicking::try (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:337
#43 0x0000555555960c2a in std::panic::catch_unwind (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:394
#44 0x0000555555831159 in tokio::runtime::task::harness::Harness<T,S>::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
#45 0x00005555559de5f0 in tokio::runtime::task::raw::poll (ptr=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
#46 0x000055555755894f in tokio::runtime::task::raw::RawTask::poll (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
#47 0x0000555557531671 in tokio::runtime::task::Notified<S>::run (self=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
#48 0x000055555755a0f3 in tokio::runtime::blocking::pool::Inner::run (self=0x7ffff763a250) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:230
#49 0x0000555557559e3e in tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:210
#50 0x000055555750a2e3 in tokio::runtime::context::enter (new=..., f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/context.rs:72
#51 0x00005555574f80bd in tokio::runtime::handle::Handle::enter (self=0x7ffff43fad58, f=...) at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/handle.rs:76
--Type <RET> for more, q to quit, c to continue without paging--
#52 0x0000555557559ecf in tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}} () at /home/frank/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:209
#53 0x0000555557507472 in std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/sys_common/backtrace.rs:137
#54 0x00005555574dda61 in std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/mod.rs:475
#55 0x0000555557511081 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=()) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:318
#56 0x000055555753cad2 in std::panicking::try::do_call (data=0x7ffff43fafb8 "\002\000") at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:373
#57 0x00005555575474bd in __rust_try ()
#58 0x000055555753c9b4 in std::panicking::try (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panicking.rs:337
#59 0x0000555557511143 in std::panic::catch_unwind (f=...) at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/panic.rs:394
#60 0x00005555574dd859 in std::thread::Builder::spawn_unchecked::{{closure}} () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/std/src/thread/mod.rs:474
#61 0x000055555752855f in core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/core/src/ops/function.rs:233
#62 0x00005555577f7f7a in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/alloc/src/boxed.rs:1025
#63 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/792c645ca7d11a8d254df307d019c5bf01445c37/library/alloc/src/boxed.rs:1025
#64 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:87
#65 0x00007ffff7fa6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#66 0x00007ffff7ca2103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
Does this clarify anything?
Oh, that is very weird. That should only segfault if you have called the hidden i_promise_dst_is_same_process method, which I don't think you have. My guess is that there's a version mismatch between the different ways in which the noria crate is compiled here. In particular, noria-mysql is probably linked to an older version of noria, which causes something to go awry in the serialization/deserialization. Perhaps try cargo update in the noria-mysql directory before building it?
I've tried it on another box. Nothing rust related on it. Rustup-ed, checked out all, did the same steps, with the same result. cargo update did not seem to help. If there is a version mismatch, it's on the git master branches as well
So, just to check:
$ git clone https://github.com/mit-pdos/noria.git noria
$ git clone https://github.com/mit-pdos/noria-mysql.git noria-mysql
$ cd noria && cargo update
$ cd noria-mysql && cargo update
And then you run cargo run --release both binaries?
Can you try patching this function: https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/internal/proto.rs#L50
in each match arm so that it prints whether the received thing is "local" or not? You'll probably have to modify the patch.crates-io section of noria-mysql/Cargo.toml to point at the local path you've checked out noria into.
Ok, deleted all the Noria things, re-build & ran noria (same result). Pointed the noria-mysql dependencies to the local:
-noria = { git = "https://github.com/mit-pdos/noria.git" }
-noria-server = { git = "https://github.com/mit-pdos/noria.git" }
-# noria = { path = "../noria/noria/" }
-# noria-server = { path = "../noria/server/" }
+# noria = { git = "https://github.com/mit-pdos/noria.git" }
+# noria-server = { git = "https://github.com/mit-pdos/noria.git" }
+noria = { path = "../noria/noria/" }
+noria-server = { path = "../noria/server/" }
(same result)
Patched the proto.rs:
pub unsafe fn deref(&self) -> &T {
match self {
- LocalOrNotInner::Local(ref l) => l.deref(),
- LocalOrNotInner::Not(ref t) => t,
+ LocalOrNotInner::Local(ref l) => { println!("Local!"); l.deref()},
+ LocalOrNotInner::Not(ref t) => { println!("Not!"); t},
}
}
Results in:
% cargo r --release --bin noria-server -- --deployment myapp --no-reuse --shards 0
Compiling noria v0.7.0 (/Users/f.lyaruu/git/noria/noria)
Compiling noria-common v0.7.0 (/Users/f.lyaruu/git/noria/server/common)
Compiling noria-dataflow v0.7.0 (/Users/f.lyaruu/git/noria/server/dataflow)
Compiling noria-mir v0.7.0 (/Users/f.lyaruu/git/noria/server/mir)
Compiling noria-server v0.7.0 (/Users/f.lyaruu/git/noria/server)
Finished release [optimized + debuginfo] target(s) in 2m 43s
Running `target/release/noria-server --deployment myapp --no-reuse --shards 0`
Local!
zsh: segmentation fault cargo r --release --bin noria-server -- --deployment myapp --no-reuse --shard
%
And we see local!
Thank you for digging! That's bizarre indeed... Local should only be used when for_local_transfer is used:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/internal/proto.rs#L105
for_local_transfer is in turn only used for writes when dst_is_local is true:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L501-L502
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L541-L543
dst_is_local defaults to false
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L383
and is only set to true if i_promise_dst_is_same_process is called:
https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/table.rs#L613-L614
and that is only ever called in the vote and vote-dbtoaster-style binaries, neither of which you are running, and does not appear in noria-mysql. This is what makes me think it must be serialization error somehow --- it should not be possible for a Local to be generated (and therefore it should not be possible for the server to receive one).
Looking at the implementation, this line looks a little questionable, but I also suspect that it probably isn't hit anyway (maybe try adding a print): https://github.com/mit-pdos/noria/blob/f54dfcef4057b490ccb43f4a31c6aaf35c923fb7/noria/src/internal/proto.rs#L34
Just for my own sanity's sake, could you try removing the tokio patch in noria-mysql/Cargo.toml? It shouldn't make a difference, but worth to double-check.
Ok, went back to this one.
I've added some debug around the deserialization code:
println!("Before deserialize!");
let result = usize::deserialize(deserializer).map(|p| LocalBypass(p as *mut T));
println!("After deserialize!");
And it does get hit:
Before deserialize!
After deserialize!
Local!
zsh: segmentation fault
Removing the tokio patch does not lead to discernible effect. It baffles me that I can reproduce the problem so consistently, yet you can not. I've tried on two different MacBooks, a Linux pc, and a Linux VM inside a Windows box, with completely consistent results.
I had a theory that it was somehow related to my mysql client (because that was the same accross all my attempts, an up-to-date mysql 8.0.21 client, but somehow an older client 8.0.15 does not connect at all. I also tried a JS based mysql client but that also does not seem to be able to connect at all.
When cloning the latest versions of noria and noria-mysql (with noria-mysql patched to look locally), I got this same issue. I also saw this warning in the Noria-mysql build:
Patch `noria-server v0.7.0 (/home/alex/noria/server)` was not used in the crate graph
I was able to fix this with the following Cargo.toml patch:
12c12
< noria = "0.4"
---
> noria = "0.7"
30c30
< noria-server = "0.4"
---
> noria-server = "0.7"
along with a patch in noria-mysql to not refer to a function that has been removed:
sed -i 's%noria::trace_my_next_op%// noria::trace_my_next_op%g' src/backend.rs
With this, I'm able to run those example SQL statements without anything crashing
Yes, works for me! So I guess it would make sense then to release Noria 0.7, as the master of Noria MySQL depends on it...? Would it help to set up a ci pipeline for Noria? Issues like the one we just had are really tricky to debug but should be pretty straightforward to detect in a CI pipeline.