Error when submitting narwhal transactions through `SubmitTransactionStream`

[palango]

Steps to Reproduce Issue

I'm playing around with narwhal and try to submit some transactions to it. The code is similar to the benchmark client, but I always run into the following error without an helpful message:

Failed to send transaction: status: Unimplemented, message: "", details: [], metadata: MetadataMap { headers: {"date": "Tue, 15 Nov 2022 15:24:59 GMT", "content-type": "application/grpc", "content-length": "0", "grpc-path-req": "/narwhal.Transactions/SubmitTransactionStream"} }

let mut client = TransactionsClient::connect(dsts[client].clone())
    .await
    .expect("Could not create TransactionsClient");
let stream = tokio_stream::iter(txs.into_iter())
    .map(move |tx| {
        println!("Resending tx {:?}", &tx);
        TransactionProto {
            transaction: tx.serialize(),
        }
});

if let Err(e) = client.submit_transaction_stream(stream).await {
    println!("Failed to send transaction: {e}");
    // FIXME: Not sure why this keeps happening, ignore for now
    // return Ok(());
}

Expected Result

Submitting should either work or provide a more helpful error message.

Actual Result

Error without useful message.

System Information

• OS: macOS 13.0.1
• Compiler: rustc 1.64.0

[huitseeker]

@palango can you confirm which revision of the code base you're using as this has changed recently? /cc @Andrew Schran that has recently looked at this

[palango]

@huitseeker This is on 0bc7d1984 (tag: devnet-0.14.1) from 3 weeks ago.

[aschran]

Are there any debug logs on worker side we could also look at?

[palango]

@aschran Here are logs of the above happening: logs.zip

In demo_client-1.log and demo_client-2.log you see the failures when sending transactions to the narwhal node. I didn't find any errors in the worker logs with a quick grep.

[aschran]

I don't see any changes in the gRPC server code recently. Did it used to work and now it broke, or have you never gotten this working before?

Based on lack of log errors I'd guess that the endpoint your client is connecting to is not actually providing the Transactions gRPC service and this is some default error gRPC generates for that case. Which server are you connecting to? If you manually run grpc_cli ls on it while it's running, what services does it show as available?

[palango]

Hi @aschran

to give you a bit more context: I wanted to learn about more about the provided API and decided to build a little end-to-end prototype including narwhal. I found the existing benchmark and demo infrastructure quite useful and changed it to my requirements. This means:

• the demo client was changed to run a simple blockchain that creates blocks from the transactions received from narwhal
• the benchmark client was changed to send transactions of my simple blockchain

Creating the transactions in the benchmark client works, so the SubmitTransactionStream endpoint seems to work. Also, in the demo client, usually a few transactions can be sent before the mentioned error appears. From that I'm assuming that the Transactions gRPC service is available.

The code used to generate the logs is available at https://github.com/palango/sui/tree/for-mysten . To run it, just

cd narwhal/benchmark/
fab demo

Hope this makes the problem clearer for you, I'm happy to answer any additional questions.