crypto-crawler
crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
I am using the following code (collected from your examples):
1 use crypto_crawler::{crawl_trade, MarketType};
2
3 #[tokio::main(flavor = "multi_thread")]
4 async fn main() {
5 let (tx, rx) = std::sync::mpsc::channel();
6
7 tokio::task::spawn(async move {
8 crawl_trade(
9 "kucoin",
10 MarketType::LinearSwap,
11 Some(&["SOLUSDTM".to_string()]),
12 /*
13 MarketType::Spot,
14 Some(&["SOL-USDT".to_string()]),
15 */
16 tx,
17 )
18 .await
19 });
20 for msg in rx {
21 //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
22 let trades =
23 crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
24 let trade = &trades[0];
25 println!("{:#?}", trade);
26 }
27 }
~
Only for MarketType::LinearSwap (but not for MarketType::Spot because https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L198 never gets called for Spot) I get the following panic with the debug build (this never happens with a release build, i think it's something debug-specific? with blocking http_get("https://api-futures.kucoin.com/api/v1/contracts/active") reqwest in crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181 ).
alecm@ark2:~/qkt$ ./target/debug/qk
thread 'main' panicked at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/shutdown.rs:51:21
stack backtrace:
0: std::panicking::begin_panic
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/std/src/panicking.rs:588:12
1: tokio::runtime::blocking::shutdown::Receiver::wait
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/shutdown.rs:51:21
2: tokio::runtime::blocking::pool::BlockingPool::shutdown
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/pool.rs:261:12
3: <tokio::runtime::blocking::pool::BlockingPool as core::ops::drop::Drop>::drop
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/pool.rs:278:9
4: core::ptr::drop_in_place<tokio::runtime::blocking::pool::BlockingPool>
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ptr/mod.rs:487:1
5: core::ptr::drop_in_place<tokio::runtime::runtime::Runtime>
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ptr/mod.rs:487:1
6: reqwest::blocking::wait::enter
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/wait.rs:76:21
7: reqwest::blocking::wait::timeout
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/wait.rs:13:5
8: reqwest::blocking::client::ClientHandle::new
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/client.rs:1038:15
9: reqwest::blocking::client::ClientBuilder::build
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/client.rs:103:9
10: crypto_contract_value::exchanges::utils::http_get
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/utils.rs:7:18
11: crypto_contract_value::exchanges::kucoin::fetch_linear_multipliers
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181:22
12: crypto_contract_value::exchanges::kucoin::LINEAR_CONTRACT_VALUES::{{closure}}
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:152:23
13: core::ops::function::FnOnce::call_once
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
14: core::ops::function::FnOnce::call_once
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
15: once_cell::sync::Lazy<T,F>::force::{{closure}}
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1275:28
16: once_cell::sync::OnceCell<T>::get_or_init::{{closure}}
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1085:57
17: once_cell::imp::OnceCell<T>::initialize::{{closure}}
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:82:23
18: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:297:13
19: once_cell::imp::initialize_or_wait
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:206:20
20: once_cell::imp::OnceCell<T>::initialize
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:78:9
21: once_cell::sync::OnceCell<T>::get_or_try_init
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1126:13
22: once_cell::sync::OnceCell<T>::get_or_init
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1085:19
23: once_cell::sync::Lazy<T,F>::force
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1274:13
24: <once_cell::sync::Lazy<T,F> as core::ops::deref::Deref>::deref
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1335:13
25: crypto_contract_value::exchanges::kucoin::get_contract_value
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:198:40
26: crypto_contract_value::get_contract_value
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/lib.rs:22:21
27: crypto_msg_parser::exchanges::utils::calc_quantity_and_volume
at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/utils.rs:41:9
28: crypto_msg_parser::exchanges::kucoin::kucoin_swap::parse_trade
at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/kucoin/kucoin_swap.rs:64:62
29: crypto_msg_parser::exchanges::kucoin::parse_trade
at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/kucoin/mod.rs:118:9
30: crypto_msg_parser::parse_trade
at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/lib.rs:97:21
31: qk::main::{{closure}}
at ./src/main.rs:23:13
32: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/future/mod.rs:91:19
33: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:63
34: tokio::runtime::coop::with_budget
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:102:5
35: tokio::runtime::coop::budget
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:68:5
36: tokio::runtime::park::CachedParkThread::block_on
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:31
37: tokio::runtime::context::BlockingRegionGuard::block_on
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/context.rs:295:13
38: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
39: tokio::runtime::runtime::Runtime::block_on
at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/runtime.rs:284:45
40: qk::main
at ./src/main.rs:20:5
41: core::ops::function::FnOnce::call_once
at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
PS. With the release build, i let it run for several days, but the websocket stopped getting new messages after 2 days. I rebuilt as a debug and ran into the panic above. Should I file a separate Issue on the stuck websocket (unsure how to debug that, strace showed waiting for some FUTEX)?
Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
Here you go
use crypto_crawler::{crawl_trade, MarketType};
#[tokio::main(flavor = "multi_thread")]
async fn main() {
let (tx, rx) = std::sync::mpsc::channel();
tokio::task::spawn(async move {
crawl_trade(
"kucoin",
MarketType::LinearSwap,
Some(&["SOLUSDTM".to_string()]),
/*
MarketType::Spot,
Some(&["SOL-USDT".to_string()]),
*/
tx,
)
.await
});
for msg in rx {
//https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
let trades =
crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
let trade = &trades[0];
println!("{:#?}", trade);
}
}
and
[package]
name = "qk"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
crypto-crawler = { git = "https://github.com/crypto-crawler/crypto-crawler-rs", branch = "main" }
crypto-msg-parser = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main" }
crypto-message = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main"}
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
Here you go
use crypto_crawler::{crawl_trade, MarketType}; #[tokio::main(flavor = "multi_thread")] async fn main() { let (tx, rx) = std::sync::mpsc::channel(); tokio::task::spawn(async move { crawl_trade( "kucoin", MarketType::LinearSwap, Some(&["SOLUSDTM".to_string()]), /* MarketType::Spot, Some(&["SOL-USDT".to_string()]), */ tx, ) .await }); for msg in rx { //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs let trades = crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap(); let trade = &trades[0]; println!("{:#?}", trade); } }and
[package] name = "qk" version = "0.1.0" edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] crypto-crawler = { git = "https://github.com/crypto-crawler/crypto-crawler-rs", branch = "main" } crypto-msg-parser = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main" } crypto-message = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main"} tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
I have the same issue.any resolutions? @panicfarm
I have the same issue.any resolutions? @panicfarm
@soulmachine I believe the issue is here:
https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L152
fn fetch_linear_multipliers() is blocking
https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L178 but your example that I used is async (because your websocket is async), so this function (and thus crypto_msg_parser::parse_trade()) cannot be called from an asynchronous context.
The function is blocking because inside it contains blocking reqwest: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/utils.rs#L7
You could make the entire crypto_msg_parser::parse_trade() async, although I think it's prefererrable to "instantly" parse the websocket messages, leaving this function sync, so maybe this blocking fetch_linear_multipliers reqwest can be avoided or asynchronously done before the websocket connection is established.
I have the same issue.any resolutions? @panicfarm
@soulmachine I believe the issue is here: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L152
fn fetch_linear_multipliers()is blocking https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L178 but your example that I used is async (because your websocket is async), so this function (and thuscrypto_msg_parser::parse_trade()) cannot be called from an asynchronous context.The function is blocking because inside it contains blocking reqwest: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/utils.rs#L7
You could make the entire
crypto_msg_parser::parse_trade()async, although I think it's prefererrable to "instantly" parse the websocket messages, leaving this function sync, so maybe this blockingfetch_linear_multipliersreqwest can be avoided or asynchronously done before the websocket connection is established.
I think whether fetch_linear_multipliers is blocking or async is not important because it's only called once when called. static LINEAR_CONTRACT_VALUES: Lazy<HashMap<String, f64>> = Lazy::new(|| only request once. I prefer the async resolution because the app may use the async functions.