catapult-rest icon indicating copy to clipboard operation
catapult-rest copied to clipboard

Published 20 hours ago • verified publisher icon symbol

REST failed to submit transaction to server.

Open Wayonb opened this issue 4 years ago • 1 comments

REST fails to submit transaction to the server and there is no error in the server logs This does not seem to be an issue with server since Node/info and node/peers works and return data.

rest-gateway_1  | verbose: connecting to api-node:7900 {"timestamp":"2021-01-30T20:03:08.026Z"}
api-node_1      | 2021-01-30 20:03:08.030190 0x00007fbbd9d9b700: <debug> (ionet::PacketSocket.cpp@721) invoking user callback after successful async_accept (000000000000B791) 
api-node_1      | 2021-01-30 20:03:08.030318 0x00007fbbd3fff700: <info> (extensions::NetworkUtils.cpp@125) accept result to local node port 7900 from 172.20.0.10: Accepted 
api-node_1      | 2021-01-30 20:03:08.030361 0x00007fbbd3fff700: <debug> (net::PacketReaders.cpp@258) accepted connection from '172.20.0.10' as 4D347C6E95D6917BE76E408ED0E089D9CC6BA72CC0147C65C1DEC22FDE38CEA3 
api-node_1      | 2021-01-30 20:03:08.031108 0x00007fbbd959a700: <info> (ionet::PacketSocket.cpp@42) eof reading from socket: End of file 
api-node_1      | 2021-01-30 20:03:08.031129 0x00007fbbd959a700: <debug> (net::ChainedSocketReader.cpp@71) 4D347C6E95D6917BE76E408ED0E089D9CC6BA72CC0147C65C1DEC22FDE38CEA3 @ 172.20.0.10 read completed with error: Closed 
api-node_1      | 2021-01-30 20:03:08.031140 0x00007fbbd959a700: <debug> (net::PacketReaders.cpp@169) closing connection to 4D347C6E95D6917BE76E408ED0E089D9CC6BA72CC0147C65C1DEC22FDE38CEA3 @ 172.20.0.10 - 0 
api-node_1      | 2021-01-30 20:03:08.031198 0x00007fbbd959a700: <debug> (ionet::PacketSocket.cpp@603) socket close triggered by owner (000000000000B791), 3ms elapsed 
rest-gateway_1  | verbose: connecting to api-node:7900 {"timestamp":"2021-01-30T20:03:08.031Z"}
api-node_1      | 2021-01-30 20:03:08.035269 0x00007fbbd959a700: <debug> (ionet::PacketSocket.cpp@721) invoking user callback after successful async_accept (000000000000B792) 
api-node_1      | 2021-01-30 20:03:08.035415 0x00007fbbd959a700: <info> (extensions::NetworkUtils.cpp@125) accept result to local node port 7900 from 172.20.0.10: Accepted 
api-node_1      | 2021-01-30 20:03:08.035453 0x00007fbbd959a700: <debug> (net::PacketReaders.cpp@258) accepted connection from '172.20.0.10' as 4D347C6E95D6917BE76E408ED0E089D9CC6BA72CC0147C65C1DEC22FDE38CEA3 
api-node_1      | 2021-01-30 20:03:08.035989 0x00007fbbd8d99700: <info> (ionet::PacketSocket.cpp@42) eof reading from socket: End of file 
api-node_1      | 2021-01-30 20:03:08.036013 0x00007fbbd8d99700: <debug> (net::ChainedSocketReader.cpp@71) 4D347C6E95D6917BE76E408ED0E089D9CC6BA72CC0147C65C1DEC22FDE38CEA3 @ 172.20.0.10 read completed with error: Closed 
api-node_1      | 2021-01-30 20:03:08.036024 0x00007fbbd8d99700: <debug> (net::PacketReaders.cpp@169) closing connection to 4D347C6E95D6917BE76E408ED0E089D9CC6BA72CC0147C65C1DEC22FDE38CEA3 @ 172.20.0.10 - 0 
api-node_1      | 2021-01-30 20:03:08.036059 0x00007fbbd3fff700: <debug> (ionet::PacketSocket.cpp@603) socket close triggered by owner (000000000000B792), 2ms elapsed 
api-node_1      | 2021-01-30 20:03:08.655676 0x00007fbbd959a700: <debug> (extensions::PeersConnectionTasks.cpp@119) no add candidates for service 1346458196 
db_1            | {"t":{"$date":"2021-01-30T20:03:10.034+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn8","msg":"Slow query","attr":{"type":"command","ns":"catapult.namespaces","command":{"find":"namespaces","filter":{"$and":[{"namespace.alias.mosaicId":{"$in":[3239218328380109726]}},{"namespace.alias.type":1},{"$and":[{"meta.latest":true},{"$or":[{"namespace.endHeight":-1},{"namespace.endHeight":{"$gt":64347}}]}]}]},"returnKey":false,"showRecordId":false,"lsid":{"id":{"$uuid":"e2c7ba69-82ca-402a-a0f2-7be98559c032"}},"$db":"catapult"},"planSummary":"IXSCAN { meta.latest: -1, meta.index: 1, namespace.level0: 1, namespace.depth: 1 }","keysExamined":413639,"docsExamined":413639,"cursorExhausted":true,"numYields":427,"nreturned":1,"queryHash":"726729D0","planCacheKey":"B9C490C5","reslen":399,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":428}},"Global":{"acquireCount":{"r":428}},"Database":{"acquireCount":{"r":428}},"Collection":{"acquireCount":{"r":428}},"Mutex":{"acquireCount":{"r":1}}},"storage":{"data":{"bytesRead":115377231,"timeReadingMicros":497748}},"protocol":"op_msg","durationMillis":1896}}
rest-gateway_1  | error: caught error 503 connection failed {"jse_shortmsg":"connection failed","jse_info":{},"body":{"code":"ServiceUnavailable","message":"connection failed"},"stack":"ServiceUnavailableError: connection failed\n    at Object.createServiceUnavailableError (/app/catapult-rest/rest/_build/server/errors.js:56:44)\n    at TLSSocket.<anonymous> (/app/catapult-rest/rest/_build/connection/connectionService.js:104:20)\n    at TLSSocket.emit (events.js:327:22)\n    at net.js:674:12\n    at TCP.done (_tls_wrap.js:567:7)","timestamp":"2021-01-30T20:03:10.497Z"}
api-node_1      | 2021-01-30 20:03:11.315596 0x00007fbbd8d99700: <debug> (extensions::PeersConnectionTasks.cpp@119) no add candidates for service 1179209292 
rest-gateway_1  | verbose: connecting to api-node:7900 {"timestamp":"2021-01-30T20:03:28.098Z"}

docker ps -a
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS                  PORTS                    NAMES
8fcdf9397f75        symbolplatform/symbol-rest:2.3.1            "docker-entrypoint.s…"   2 days ago          Up 2 days               0.0.0.0:3000->3000/tcp   apiassembly_rest-gateway_1
3039bdccb8fa        mongo:4.4.3-bionic                          "docker-entrypoint.s…"   2 days ago          Exited (0) 2 days ago                            apiassembly_init-db_1
c4c24a9ca1d4        symbolplatform/symbol-server:gcc-0.10.0.5   "bash -c '/bin-mount…"   2 days ago          Up 2 days               0.0.0.0:7902->7902/tcp   apiassembly_api-broker_1
c033c0e3dd72        symbolplatform/symbol-server:gcc-0.10.0.5   "bash -c '/bin-mount…"   2 days ago          Up 2 days               0.0.0.0:7900->7900/tcp   apiassembly_api-node_1
95fa49ec2fc1        mongo:4.4.3-bionic                          "docker-entrypoint.s…"   2 days ago          Up 2 days               27017/tcp                apiassembly_db_1

Wayonb avatar Jan 30 '21 20:01 Wayonb

Leaving REST up, I restarted the api-node to see but REST still fails.

rest-gateway_1  | verbose: websocket MJSU4JL7X3WFU6Z2PRAH44CJDQAQOUPU: created /ws websocket connection {"timestamp":"2021-01-30T21:25:35.673Z"}
db_1            | {"t":{"$date":"2021-01-30T21:25:37.801+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn8","msg":"Slow query","attr":{"type":"command","ns":"catapult.namespaces","command":{"find":"namespaces","filter":{"$and":[{"namespace.alias.mosaicId":{"$in":[3239218328380109726]}},{"namespace.alias.type":1},{"$and":[{"meta.latest":true},{"$or":[{"namespace.endHeight":-1},{"namespace.endHeight":{"$gt":64512}}]}]}]},"returnKey":false,"showRecordId":false,"lsid":{"id":{"$uuid":"4991517e-4e8c-4165-b53f-40d063475dd9"}},"$db":"catapult"},"planSummary":"IXSCAN { meta.latest: -1, meta.index: 1, namespace.level0: 1, namespace.depth: 1 }","keysExamined":413639,"docsExamined":413639,"cursorExhausted":true,"numYields":414,"nreturned":1,"queryHash":"726729D0","planCacheKey":"B9C490C5","reslen":399,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":415}},"Global":{"acquireCount":{"r":415}},"Database":{"acquireCount":{"r":415}},"Collection":{"acquireCount":{"r":415}},"Mutex":{"acquireCount":{"r":1}}},"storage":{"data":{"bytesRead":115377231,"timeReadingMicros":38173}},"protocol":"op_msg","durationMillis":1262}}
rest-gateway_1  | error: caught error 503 connection failed {"jse_shortmsg":"connection failed","jse_info":{},"body":{"code":"ServiceUnavailable","message":"connection failed"},"stack":"ServiceUnavailableError: connection failed\n    at Object.createServiceUnavailableError (/app/catapult-rest/rest/_build/server/errors.js:56:44)\n    at TLSSocket.<anonymous> (/app/catapult-rest/rest/_build/connection/connectionService.js:104:20)\n    at TLSSocket.emit (events.js:327:22)\n    at net.js:674:12\n    at TCP.done (_tls_wrap.js:567:7)","timestamp":"2021-01-30T21:25:38.265Z"}
api-node_1      | 2021-01-30 21:25:47.421385 0x00007f2371326700: <info> (ionet::PacketSocket.cpp@850) connected to beacon-01.eu-central-1.testnet.symboldev.network [35.158.140.1:7900] (00000000000000DE)

Wayonb avatar Jan 30 '21 21:01 Wayonb