crawlab
crawlab copied to clipboard
Published
20 hours ago •
crawlab-team
crawlab-team
Pro Version random GRPC connection errors in the middle of task execution.
Describe the bug
While executing a task that runs for quite a long time (>1 day) docker logs tend to show various errors related to GRPC connection such as rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp <MASTER_NODE_IP_ADDRESS>:<GRPC_PORT>: connect: connection refused or connection reset by peer. Both worker node on which task runs and master node machines show continuous, uninterrupted availability network and hardware wise. Sometimes it is accompanied by errors related to failure of verifying license error verify license error: Post \"https://license.crawlab.cn/release/license/verify\": net/http: TLS handshake timeout. retry in 5 seconds.
Expected behavior Nodes don't lose GRPC connection randomly
Below there are logs from master node from most recent such occurence. Logs are timestamped between 1 minute before it happened to 1 minute after. Crawlab container did rebuild itself at exactly 2023-09-13T16:30:47.886726955Z. Around the same time, as we can see on line 15 of the logs, panic: runtime error: index out of range [243] with length 62 error happened. Afterwards service did reset.
Logs
2023-09-13T16:30:00.679442141Z [GIN] 2023/09/14 - 00:30:00 | 200 | 2.745628ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:02.539992260Z [GIN] 2023/09/14 - 00:30:02 | 200 | 371.551614ms | 127.0.0.1 | GET "/metrics" 2023-09-13T16:30:05.679892343Z [GIN] 2023/09/14 - 00:30:05 | 200 | 2.698525ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:10.682191045Z [GIN] 2023/09/14 - 00:30:10 | 200 | 2.474229ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:15.700731204Z [GIN] 2023/09/14 - 00:30:15 | 200 | 2.06508ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:17.530844865Z [GIN] 2023/09/14 - 00:30:17 | 200 | 363.273375ms | 127.0.0.1 | GET "/metrics" 2023-09-13T16:30:20.683237154Z [GIN] 2023/09/14 - 00:30:20 | 200 | 2.886519ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:25.693064357Z [GIN] 2023/09/14 - 00:30:25 | 200 | 3.350451ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:30.683004760Z [GIN] 2023/09/14 - 00:30:30 | 200 | 2.76009ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:32.540234105Z [GIN] 2023/09/14 - 00:30:32 | 200 | 372.425448ms | 127.0.0.1 | GET "/metrics" 2023-09-13T16:30:35.683029499Z [GIN] 2023/09/14 - 00:30:35 | 200 | 2.692134ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:40.683991587Z [GIN] 2023/09/14 - 00:30:40 | 200 | 2.399953ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:45.703194226Z [GIN] 2023/09/14 - 00:30:45 | 200 | 2.537904ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" 2023-09-13T16:30:47.499802207Z panic: runtime error: index out of range [243] with length 62 2023-09-13T16:30:47.499842700Z 2023-09-13T16:30:47.499846920Z goroutine 2559365 [running]: 2023-09-13T16:30:47.499850850Z github.com/emirpasic/gods/lists/arraylist.(*List).Add(...) 2023-09-13T16:30:47.499854241Z /go/pkg/mod/github.com/emirpasic/[email protected]/lists/arraylist/arraylist.go:47 2023-09-13T16:30:47.499858001Z github.com/crawlab-team/crawlab-pro/core/metric.(*Service).getAllNodeMetrics.func1.1({0xc0024f2000?, 0xd4, 0xc001f65e70?}) 2023-09-13T16:30:47.499861631Z /go/src/app/core/metric/service.go:226 +0x22d 2023-09-13T16:30:47.499864811Z github.com/crawlab-team/crawlab-pro/core/metric.(*Service).requestRemoteNodeMetrics(0xc000a1b3b0, {0x229e148, 0xc001077500}, {0x22aaca0, 0xc001f71ba0}, 0xc0000fc870?, 0xc00119bee8) 2023-09-13T16:30:47.499868422Z /go/src/app/core/metric/service.go:334 +0x4da 2023-09-13T16:30:47.499871842Z github.com/crawlab-team/crawlab-pro/core/metric.(*Service).getAllNodeMetrics.func1({{0x64, 0xcb, 0x50, 0xd5, 0x59, 0xbf, 0x44, 0x6b, 0x80, 0x9c, ...}, ...}) 2023-09-13T16:30:47.499875282Z /go/src/app/core/metric/service.go:224 +0xfa 2023-09-13T16:30:47.499878852Z created by github.com/crawlab-team/crawlab-pro/core/metric.(*Service).getAllNodeMetrics 2023-09-13T16:30:47.499892744Z /go/src/app/core/metric/service.go:223 +0x40c 2023-09-13T16:30:48.028047289Z 2023/09/14 00:30:48 info [NotificationService] start handling events 2023-09-13T16:30:48.028150446Z 2023/09/14 00:30:48 info [NotificationService] attempt to obtain grpc stream... 2023-09-13T16:30:48.035790910Z 2023/09/14 00:30:48 info initialized controllers successfully 2023-09-13T16:30:48.035805620Z 2023/09/14 00:30:48 info initialized middlewares successfully 2023-09-13T16:30:48.036093351Z 2023/09/14 00:30:48 info initialized routes successfully 2023-09-13T16:30:48.050108956Z 2023/09/14 00:30:48 info current node type: master 2023-09-13T16:30:48.050120687Z 2023/09/14 00:30:48 info running in docker container 2023-09-13T16:30:48.056160645Z 2023/09/14 00:30:48 info process created 2023-09-13T16:30:48.073410993Z 2023/09/14 00:30:48 info process started 2023-09-13T16:30:48.094045077Z [nginx] * Starting nginx nginx 2023-09-13T16:30:48.112639794Z [nginx] ...done. 2023-09-13T16:30:48.124166549Z ts=2023-09-13T16:30:48.124Z caller=main.go:535 level=info msg="Starting Prometheus Server" mode=server version="(version=2.37.0, branch=HEAD, revision=b41e0750abf5cc18d8233161560731de05199330)" 2023-09-13T16:30:48.124265346Z ts=2023-09-13T16:30:48.124Z caller=main.go:540 level=info build_context="(go=go1.18.4, user=root@0ebb6827e27f, date=20220714-15:13:18)" 2023-09-13T16:30:48.124347572Z ts=2023-09-13T16:30:48.124Z caller=main.go:541 level=info host_details="(Linux 6.2.0-1011-aws #11~22.04.1-Ubuntu SMP Mon Aug 21 16:27:59 UTC 2023 x86_64 d94895de835b (none))" 2023-09-13T16:30:48.124430108Z ts=2023-09-13T16:30:48.124Z caller=main.go:542 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-09-13T16:30:48.124506303Z ts=2023-09-13T16:30:48.124Z caller=main.go:543 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-09-13T16:30:48.128543475Z ts=2023-09-13T16:30:48.128Z caller=web.go:553 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 2023-09-13T16:30:48.128969857Z ts=2023-09-13T16:30:48.128Z caller=main.go:972 level=info msg="Starting TSDB ..." 2023-09-13T16:30:48.129318592Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1691863202169 maxt=1692057600000 ulid=01H7VS25N0N02EQSXQHX1HX3MY 2023-09-13T16:30:48.129418209Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692057602167 maxt=1692252000000 ulid=01H81JERK0MNXN208KY4CD1QSE 2023-09-13T16:30:48.129508896Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692252002167 maxt=1692446400000 ulid=01H87BTXQNTQ1QK6A23D9J1X4E 2023-09-13T16:30:48.129593812Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692446402167 maxt=1692640800000 ulid=01H8D57Q3452QTS4CKV817MFP2 2023-09-13T16:30:48.129682358Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692640802172 maxt=1692835200000 ulid=01H8JYMEHHPCFY0KJYECPSEMB7 2023-09-13T16:30:48.129793816Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692835200000 maxt=1693029600000 ulid=01H8RH4RZFEZNPNV2E0706WPSM 2023-09-13T16:30:48.129895274Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693029600000 maxt=1693224000000 ulid=01H8YAHCSRB190BC7GX09ZCERJ 2023-09-13T16:30:48.129972059Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693224002168 maxt=1693418400000 ulid=01H94AT8JEBMTGPG395XDB4D9Q 2023-09-13T16:30:48.130060405Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693418402167 maxt=1693612800000 ulid=01H9A46Y5GEB9MBYYR73R4ZNAF 2023-09-13T16:30:48.130138811Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693612802167 maxt=1693807200000 ulid=01H9FXKFJ7S9M4HZZGPCNHHW27 2023-09-13T16:30:48.130234048Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693807202167 maxt=1694001600000 ulid=01H9NQ01NRA4TJJYRQ80KV6FRH 2023-09-13T16:30:48.130321995Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694001602169 maxt=1694196000000 ulid=01H9VGCN18QM0GXZJ7099R5PA2 2023-09-13T16:30:48.130407651Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694196002172 maxt=1694390400000 ulid=01HA19S8CB3XBZ3S86GBDE1Q02 2023-09-13T16:30:48.130492657Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694584802167 maxt=1694592000000 ulid=01HA6WA4QYSWDD6EGQQ5R97QPW 2023-09-13T16:30:48.130559062Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694592000000 maxt=1694599200000 ulid=01HA735DBF53ZW8F1HKW2DFFD0 2023-09-13T16:30:48.130673510Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694390402167 maxt=1694584800000 ulid=01HA735FM7S727FVXTC0B7EVKH 2023-09-13T16:30:48.130732974Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694599202167 maxt=1694606400000 ulid=01HA7A1K7ZS8676T3HXJC9Y6QW 2023-09-13T16:30:48.130818120Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694606402167 maxt=1694613600000 ulid=01HA7GXAGEA7C2EHET3NAAK42M 2023-09-13T16:30:48.130988453Z ts=2023-09-13T16:30:48.130Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/data/prometheus/lock 2023-09-13T16:30:48.131119902Z ts=2023-09-13T16:30:48.131Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2023-09-13T16:30:48.164358649Z 2023/09/14 00:30:48 info grpc server listens to 0.0.0.0:9666 2023-09-13T16:30:48.164842774Z 2023/09/14 00:30:48 info master[4abd561c-3138-11ee-a3dd-0242ac120002] exists in db 2023-09-13T16:30:48.166868181Z 2023/09/14 00:30:48 info updated master[4abd561c-3138-11ee-a3dd-0242ac120002] in db. id: 64cb50d559bf446b809c6a62 2023-09-13T16:30:48.166956257Z 2023/09/14 00:30:48 info master[4abd561c-3138-11ee-a3dd-0242ac120002] monitoring started 2023-09-13T16:30:48.169513262Z 2023/09/14 00:30:48 info [GrpcClient] grpc client connected to localhost:9666 2023-09-13T16:30:48.169524753Z 2023/09/14 00:30:48 info [GrpcClient] grpc client registered client services 2023-09-13T16:30:48.169592698Z 2023/09/14 00:30:48 info cron: start 2023-09-13T16:30:48.170579110Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.170493093 +0800 CST entry 1 next 2023-09-15 09:00:00 +0800 CST 2023-09-13T16:30:48.170662996Z 2023/09/14 00:30:48 info [GrpcClient] grpc client connected to localhost:9666 2023-09-13T16:30:48.170671566Z 2023/09/14 00:30:48 info [GrpcClient] grpc client registered client services 2023-09-13T16:30:48.175231716Z 2023/09/14 00:30:48 info [GrpcClient] grpc client subscribed to remote server 2023-09-13T16:30:48.175360726Z 2023/09/14 00:30:48 info subscribe events 2023-09-13T16:30:48.175465383Z 2023/09/14 00:30:48 info [GrpcClient] grpc client subscribed to remote server 2023-09-13T16:30:48.175553690Z 2023/09/14 00:30:48 info [GrpcClient] start handling stream message... 2023-09-13T16:30:48.175621995Z 2023/09/14 00:30:48 info [GrpcClient] start handling stream message... 2023-09-13T16:30:48.175852781Z 2023/09/14 00:30:48 info [NodeServer] master received subscribe request from node[4abd561c-3138-11ee-a3dd-0242ac120002] 2023-09-13T16:30:48.175861732Z 2023/09/14 00:30:48 info [NodeServer] master subscribed node[4abd561c-3138-11ee-a3dd-0242ac120002] 2023-09-13T16:30:48.194122604Z 2023/09/14 00:30:48 info [NodeServer] master received subscribe request from node[4abd561c-3138-11ee-a3dd-0242ac120002] 2023-09-13T16:30:48.194135145Z 2023/09/14 00:30:48 info [NodeServer] master subscribed node[4abd561c-3138-11ee-a3dd-0242ac120002] 2023-09-13T16:30:48.194996458Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[4abd561c-3138-11ee-a3dd-0242ac120002], key: plugin:notification:4abd561c-3138-11ee-a3dd-0242ac120002 2023-09-13T16:30:48.211657454Z mongo: no documents in result 2023-09-13T16:30:48.211668385Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.211672266Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:48.211675766Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:48.211678956Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:48.211682256Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:48.211685387Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:48.211688507Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:48.211691657Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:48.211694857Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:48.211698057Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:48.211700938Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:48.211703838Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:48.211716639Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.234837623Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:48.234851094Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.234855574Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:48.234859325Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:48.234862935Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:48.234866855Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:48.234870885Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:48.234874686Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:48.234911768Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.238151413Z grpc error: stream not found 2023-09-13T16:30:48.238162364Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.238166834Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.238170654Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.238174324Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.238177514Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.238180725Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.238190115Z 2023/09/14 00:30:48 error cannot subscribe worker node[dbfb8369-3c07-11ee-be05-0242ac130002]: grpc error: stream not found 2023-09-13T16:30:48.239289515Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.239197668 +0800 CST entry 2 next 2023-09-15 09:00:00 +0800 CST 2023-09-13T16:30:48.246282942Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[4abd561c-3138-11ee-a3dd-0242ac120002], key: plugin:dependency:4abd561c-3138-11ee-a3dd-0242ac120002 2023-09-13T16:30:48.258706171Z grpc error: stream not found 2023-09-13T16:30:48.258717372Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.258721532Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.258724873Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.258728193Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.258731293Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.258734453Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.258823160Z grpc error: stream not found 2023-09-13T16:30:48.258831140Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.258835210Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.258839331Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.258843071Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.258846921Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.258850722Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.258869503Z 2023/09/14 00:30:48 error cannot subscribe worker node[0f85d06d-3c08-11ee-8713-0242ac130002]: grpc error: stream not found 2023-09-13T16:30:48.259326676Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.259267142 +0800 CST entry 3 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.261264696Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.261208792 +0800 CST entry 4 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.261341982Z grpc error: stream not found 2023-09-13T16:30:48.261349122Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.261353263Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.261357133Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.261368774Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.261373334Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.261377284Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.261427718Z grpc error: stream not found 2023-09-13T16:30:48.261434058Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.261437919Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.261441779Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.261446029Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.261449739Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.261453690Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.261618682Z 2023/09/14 00:30:48 error cannot subscribe worker node[f1c6d46d-3cd5-11ee-9861-0242ac130002]: grpc error: stream not found 2023-09-13T16:30:48.263339686Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.263284292 +0800 CST entry 5 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.269075362Z grpc error: stream not found 2023-09-13T16:30:48.269085343Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.269089203Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.269092663Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.269095944Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.269099204Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.269109774Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.269186280Z grpc error: stream not found 2023-09-13T16:30:48.269191990Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.269195231Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.269198281Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.269201561Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.269204611Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.269207661Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.269246914Z 2023/09/14 00:30:48 error cannot subscribe worker node[9cef89e1-508b-11ee-a134-0242ac130002]: grpc error: stream not found 2023-09-13T16:30:48.270702010Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.270644596 +0800 CST entry 6 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.271406231Z grpc error: stream not found 2023-09-13T16:30:48.271416941Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.271421572Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() 2023-09-13T16:30:48.271426312Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() 2023-09-13T16:30:48.271430152Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.271433533Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.271436803Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.271458524Z node error: monitor error 2023-09-13T16:30:48.271463805Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.271467425Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:236 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.271471445Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.271483646Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.271616576Z node error: monitor error 2023-09-13T16:30:48.271624767Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.271627927Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:236 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() 2023-09-13T16:30:48.271631227Z /go/pkg/mod/github.com/crawlab-team/[email protected]/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() 2023-09-13T16:30:48.271634317Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.272588666Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.272536603 +0800 CST entry 7 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.274388327Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.274329453 +0800 CST entry 8 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.276172956Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.276112632 +0800 CST entry 9 next 2023-09-16 09:00:00 +0800 CST 2023-09-13T16:30:48.277968866Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.277911702 +0800 CST entry 10 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.279803769Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.279744484 +0800 CST entry 11 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.281613940Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.281555846 +0800 CST entry 12 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.283546860Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.283487825 +0800 CST entry 13 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.285316878Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.285259664 +0800 CST entry 14 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.287169462Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.287080446 +0800 CST entry 15 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.289052438Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.288971663 +0800 CST entry 16 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.290922164Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.29086329 +0800 CST entry 17 next 2023-09-16 15:00:00 +0800 CST 2023-09-13T16:30:48.292724124Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.29266683 +0800 CST entry 18 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.294465891Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.294409167 +0800 CST entry 19 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.296359038Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.296303924 +0800 CST entry 20 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.298385725Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.29832902 +0800 CST entry 21 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.300268811Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.300216007 +0800 CST entry 22 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.302334860Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.302275956 +0800 CST entry 23 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.304317414Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.304257159 +0800 CST entry 24 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.306258495Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.30620042 +0800 CST entry 25 next 2023-09-17 09:00:00 +0800 CST 2023-09-13T16:30:48.311538557Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.311486213 +0800 CST entry 26 next 2023-09-17 15:00:00 +0800 CST 2023-09-13T16:30:48.316619995Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.31655729 +0800 CST entry 27 next 2023-09-17 15:00:00 +0800 CST 2023-09-13T16:30:48.319157229Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.319054161 +0800 CST entry 28 next 2023-09-17 15:00:00 +0800 CST 2023-09-13T16:30:48.323228483Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.323135586 +0800 CST entry 29 next 2023-09-19 09:00:00 +0800 CST 2023-09-13T16:30:48.326576826Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.326512721 +0800 CST entry 30 next 2023-09-15 16:00:00 +0800 CST 2023-09-13T16:30:48.336786435Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.3367168 +0800 CST entry 31 next 2023-09-19 10:00:00 +0800 CST 2023-09-13T16:30:48.338617938Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.338559374 +0800 CST entry 32 next 2023-09-19 10:00:00 +0800 CST 2023-09-13T16:30:48.340433219Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.340369525 +0800 CST entry 33 next 2023-09-19 10:00:00 +0800 CST 2023-09-13T16:30:48.342137293Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.342081329 +0800 CST entry 34 next 2023-09-19 10:00:00 +0800 CST 2023-09-13T16:30:48.343908811Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.343845176 +0800 CST entry 35 next 2023-09-19 10:00:00 +0800 CST 2023-09-13T16:30:48.345669669Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.345602994 +0800 CST entry 36 next 2023-10-02 09:00:00 +0800 CST 2023-09-13T16:30:48.347481690Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.347416715 +0800 CST entry 37 next 2023-10-03 09:00:00 +0800 CST 2023-09-13T16:30:48.349201034Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.34913614 +0800 CST entry 38 next 2023-10-02 16:00:00 +0800 CST 2023-09-13T16:30:48.350922719Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.350867145 +0800 CST entry 39 next 2023-09-19 13:00:00 +0800 CST 2023-09-13T16:30:48.352625532Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.352568008 +0800 CST entry 40 next 2023-09-19 13:00:00 +0800 CST 2023-09-13T16:30:48.354303654Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.354246139 +0800 CST entry 41 next 2023-09-19 13:00:00 +0800 CST 2023-09-13T16:30:48.355980365Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.355922581 +0800 CST entry 42 next 2023-09-19 13:00:00 +0800 CST 2023-09-13T16:30:48.364167488Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.364100293 +0800 CST entry 43 next 2023-09-19 13:00:00 +0800 CST 2023-09-13T16:30:48.365823988Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.365763603 +0800 CST entry 44 next 2023-09-19 16:00:00 +0800 CST 2023-09-13T16:30:48.367579845Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.36751782 +0800 CST entry 45 next 2023-09-19 16:00:00 +0800 CST 2023-09-13T16:30:48.369297279Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.369237985 +0800 CST entry 46 next 2023-09-19 16:00:00 +0800 CST 2023-09-13T16:30:48.370967450Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.370910826 +0800 CST entry 47 next 2023-09-18 09:00:00 +0800 CST 2023-09-13T16:30:48.372716277Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.372650622 +0800 CST entry 48 next 2023-10-05 09:00:00 +0800 CST 2023-09-13T16:30:48.374416750Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.374349725 +0800 CST entry 49 next 2023-10-06 19:00:00 +0800 CST 2023-09-13T16:30:48.376166837Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.376099792 +0800 CST entry 50 next 2023-10-06 19:00:00 +0800 CST 2023-09-13T16:30:48.377973608Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.377914203 +0800 CST entry 51 next 2023-10-06 19:00:00 +0800 CST 2023-09-13T16:30:48.379894037Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.379817621 +0800 CST entry 52 next 2023-10-06 19:00:00 +0800 CST 2023-09-13T16:30:48.381571658Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.381505673 +0800 CST entry 53 next 2023-10-06 19:00:00 +0800 CST 2023-09-13T16:30:48.383283142Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.383218447 +0800 CST entry 54 next 2023-10-07 03:00:00 +0800 CST 2023-09-13T16:30:48.385002346Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.384939622 +0800 CST entry 55 next 2023-10-07 03:00:00 +0800 CST 2023-09-13T16:30:48.386824198Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.386765214 +0800 CST entry 56 next 2023-10-07 03:00:00 +0800 CST 2023-09-13T16:30:48.388849745Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.388791781 +0800 CST entry 57 next 2023-10-07 03:00:00 +0800 CST 2023-09-13T16:30:48.390883252Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.390821998 +0800 CST entry 58 next 2023-10-07 03:00:00 +0800 CST 2023-09-13T16:30:48.392864636Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.392806252 +0800 CST entry 59 next 2023-10-07 10:00:00 +0800 CST 2023-09-13T16:30:48.394832358Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.394774094 +0800 CST entry 60 next 2023-10-07 10:00:00 +0800 CST 2023-09-13T16:30:48.396829313Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.396769179 +0800 CST entry 61 next 2023-10-07 10:00:00 +0800 CST 2023-09-13T16:30:48.400890897Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.40079376 +0800 CST entry 62 next 2023-10-05 16:00:00 +0800 CST 2023-09-13T16:30:48.407137769Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.407066024 +0800 CST entry 63 next 2023-09-17 00:00:00 +0800 CST 2023-09-13T16:30:48.409093171Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.409034047 +0800 CST entry 64 next 2023-09-17 00:10:00 +0800 CST 2023-09-13T16:30:48.410776713Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.410716749 +0800 CST entry 65 next 2023-09-17 00:20:00 +0800 CST 2023-09-13T16:30:48.412455405Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.41239779 +0800 CST entry 66 next 2023-09-19 03:00:00 +0800 CST 2023-09-13T16:30:48.414206571Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.414149237 +0800 CST entry 67 next 2023-09-19 03:10:00 +0800 CST 2023-09-13T16:30:48.416003121Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.415944257 +0800 CST entry 68 next 2023-09-19 03:20:00 +0800 CST 2023-09-13T16:30:48.417769439Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.417710335 +0800 CST entry 69 next 2023-09-19 03:30:00 +0800 CST 2023-09-13T16:30:48.419579280Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.419524866 +0800 CST entry 70 next 2023-09-19 03:40:00 +0800 CST 2023-09-13T16:30:48.421472447Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.421418394 +0800 CST entry 71 next 2023-09-19 03:50:00 +0800 CST 2023-09-13T16:30:48.423543088Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.423492984 +0800 CST entry 72 next 2023-09-19 04:00:00 +0800 CST 2023-09-13T16:30:48.425464476Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.425415463 +0800 CST entry 73 next 2023-09-19 04:10:00 +0800 CST 2023-09-13T16:30:48.427472282Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.427421058 +0800 CST entry 74 next 2023-09-19 04:20:00 +0800 CST 2023-09-13T16:30:48.429523250Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.429471677 +0800 CST entry 75 next 2023-09-19 04:30:00 +0800 CST 2023-09-13T16:30:48.431584180Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.431533386 +0800 CST entry 76 next 2023-09-20 12:00:00 +0800 CST 2023-09-13T16:30:48.433670461Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.433611617 +0800 CST entry 77 next 2023-10-02 08:00:00 +0800 CST 2023-09-13T16:30:48.435718919Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.435661165 +0800 CST entry 78 next 2023-10-05 08:00:00 +0800 CST 2023-09-13T16:30:48.437821861Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.437761857 +0800 CST entry 79 next 2023-10-06 20:00:00 +0800 CST 2023-09-13T16:30:48.439831377Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.439773043 +0800 CST entry 80 next 2023-10-06 20:00:00 +0800 CST 2023-09-13T16:30:48.441888976Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.441830612 +0800 CST entry 81 next 2023-10-06 20:00:00 +0800 CST 2023-09-13T16:30:48.456727700Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.456643184 +0800 CST entry 82 next 2023-10-06 20:00:00 +0800 CST 2023-09-13T16:30:48.459072970Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.458999365 +0800 CST entry 83 next 2023-10-06 20:00:00 +0800 CST 2023-09-13T16:30:48.460879591Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.460804575 +0800 CST entry 84 next 2023-10-06 20:00:00 +0800 CST 2023-09-13T16:30:48.462746306Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.462677541 +0800 CST entry 85 next 2023-10-07 04:00:00 +0800 CST 2023-09-13T16:30:48.464530156Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.46446041 +0800 CST entry 86 next 2023-10-07 04:00:00 +0800 CST 2023-09-13T16:30:48.466201666Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.466135572 +0800 CST entry 87 next 2023-10-07 04:00:00 +0800 CST 2023-09-13T16:30:48.467935222Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.467855396 +0800 CST entry 88 next 2023-10-07 04:00:00 +0800 CST 2023-09-13T16:30:48.469661507Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.469591202 +0800 CST entry 89 next 2023-10-07 04:00:00 +0800 CST 2023-09-13T16:30:48.471407703Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.471335958 +0800 CST entry 90 next 2023-10-07 04:00:00 +0800 CST 2023-09-13T16:30:48.473188602Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.473130798 +0800 CST entry 91 next 2023-09-17 00:10:00 +0800 CST 2023-09-13T16:30:48.474946260Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.474889495 +0800 CST entry 92 next 2023-09-18 00:10:00 +0800 CST 2023-09-13T16:30:48.476723078Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.476662274 +0800 CST entry 93 next 2023-09-18 00:40:00 +0800 CST 2023-09-13T16:30:48.478456494Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.47840208 +0800 CST entry 94 next 2023-09-15 08:00:00 +0800 CST 2023-09-13T16:30:48.480227672Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.480159697 +0800 CST entry 95 next 2023-10-12 03:00:00 +0800 CST 2023-09-13T16:30:48.481984820Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.481917195 +0800 CST entry 96 next 2023-10-12 03:00:00 +0800 CST 2023-09-13T16:30:48.483730656Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.483662961 +0800 CST entry 97 next 2023-10-12 03:00:00 +0800 CST 2023-09-13T16:30:48.485460671Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.485394366 +0800 CST entry 98 next 2023-10-12 03:00:00 +0800 CST 2023-09-13T16:30:48.487238190Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.487170855 +0800 CST entry 99 next 2023-10-12 06:00:00 +0800 CST 2023-09-13T16:30:48.489009338Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.488941773 +0800 CST entry 100 next 2023-10-12 06:00:00 +0800 CST 2023-09-13T16:30:48.490747003Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.490680199 +0800 CST entry 101 next 2023-10-12 06:00:00 +0800 CST 2023-09-13T16:30:48.492567766Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.4924972 +0800 CST entry 102 next 2023-10-12 06:00:00 +0800 CST 2023-09-13T16:30:48.494323243Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.494270439 +0800 CST entry 103 next 2023-09-14 01:00:00 +0800 CST 2023-09-13T16:30:48.496157876Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.496103572 +0800 CST entry 104 next 2023-09-14 01:00:00 +0800 CST 2023-09-13T16:30:48.498083635Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.49802129 +0800 CST entry 105 next 2023-09-14 01:00:00 +0800 CST 2023-09-13T16:30:48.500041877Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.499988243 +0800 CST entry 106 next 2023-09-14 04:00:00 +0800 CST 2023-09-13T16:30:48.501919772Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.501855618 +0800 CST entry 107 next 2023-09-14 04:00:00 +0800 CST 2023-09-13T16:30:48.507186574Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[9cef89e1-508b-11ee-a134-0242ac130002], key: plugin:dependency:9cef89e1-508b-11ee-a134-0242ac130002 2023-09-13T16:30:48.507313743Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[9cef89e1-508b-11ee-a134-0242ac130002], key: metric:9cef89e1-508b-11ee-a134-0242ac130002 2023-09-13T16:30:48.507406250Z 2023/09/14 00:30:48 info [NodeServer] master received subscribe request from node[9cef89e1-508b-11ee-a134-0242ac130002] 2023-09-13T16:30:48.507447313Z 2023/09/14 00:30:48 info [NodeServer] master subscribed node[9cef89e1-508b-11ee-a134-0242ac130002] 2023-09-13T16:30:48.507718253Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.507673829 +0800 CST entry 108 next 2023-09-14 04:00:00 +0800 CST 2023-09-13T16:30:48.511682350Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.511632456 +0800 CST entry 109 next 2023-09-14 07:00:00 +0800 CST 2023-09-13T16:30:48.513652393Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.513598349 +0800 CST entry 110 next 2023-09-14 07:00:00 +0800 CST 2023-09-13T16:30:48.515647877Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.515601054 +0800 CST entry 111 next 2023-09-14 07:00:00 +0800 CST 2023-09-13T16:30:48.579386273Z ts=2023-09-13T16:30:48.579Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-09-13T16:30:48.580266966Z ts=2023-09-13T16:30:48.580Z caller=head.go:536 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=888.794µs 2023-09-13T16:30:48.580279887Z ts=2023-09-13T16:30:48.580Z caller=head.go:542 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-09-13T16:30:48.704699707Z 2023/09/14 00:30:48 info subscribe events 2023-09-13T16:30:48.705826868Z mongo: no documents in result 2023-09-13T16:30:48.705837899Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.705842569Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:48.705846360Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:48.705849610Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:48.705852780Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:48.705868561Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:48.705872401Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:48.705875842Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:48.705879312Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:48.705882922Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:48.705886202Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:48.705892433Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:48.705896743Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:48.706077136Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:48.706085917Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:48.706089757Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:48.706093248Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:48.706096398Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:48.706099728Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:48.706103348Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:48.706107078Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:48.706111319Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:49.028973368Z 2023/09/14 00:30:49 info [NotificationService] obtained grpc stream, start receiving messages... 2023-09-13T16:30:49.101417084Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002] 2023-09-13T16:30:49.101586436Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[f1c6d46d-3cd5-11ee-9861-0242ac130002] 2023-09-13T16:30:49.151207909Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[0f85d06d-3c08-11ee-8713-0242ac130002], key: metric:0f85d06d-3c08-11ee-8713-0242ac130002 2023-09-13T16:30:49.207353705Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002], key: metric:f1c6d46d-3cd5-11ee-9861-0242ac130002 2023-09-13T16:30:49.220476805Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002], key: plugin:dependency:f1c6d46d-3cd5-11ee-9861-0242ac130002 2023-09-13T16:30:49.246155205Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[dbfb8369-3c07-11ee-be05-0242ac130002], key: plugin:dependency:dbfb8369-3c07-11ee-be05-0242ac130002 2023-09-13T16:30:49.251243753Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002] 2023-09-13T16:30:49.251327519Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[f1c6d46d-3cd5-11ee-9861-0242ac130002] 2023-09-13T16:30:49.400263954Z 2023/09/14 00:30:49 info subscribe events 2023-09-13T16:30:49.401600281Z mongo: no documents in result 2023-09-13T16:30:49.401615312Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:49.401619832Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:49.401623842Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:49.401628293Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:49.401632913Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:49.401637594Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:49.401641704Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:49.401645764Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:49.401649744Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:49.401654095Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:49.401657995Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:49.401674066Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:49.401678337Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:49.401811316Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:49.401820267Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:49.401823857Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:49.401827527Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:49.401830588Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:49.401833648Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:49.401837208Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:49.401840928Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:49.401844629Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:49.425924592Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[dbfb8369-3c07-11ee-be05-0242ac130002] 2023-09-13T16:30:49.425945734Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[dbfb8369-3c07-11ee-be05-0242ac130002] 2023-09-13T16:30:49.536712495Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[0f85d06d-3c08-11ee-8713-0242ac130002] 2023-09-13T16:30:49.536843734Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[0f85d06d-3c08-11ee-8713-0242ac130002] 2023-09-13T16:30:49.538279638Z 2023/09/14 00:30:49 info license activated 2023-09-13T16:30:49.550948655Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002] 2023-09-13T16:30:49.551122128Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[f1c6d46d-3cd5-11ee-9861-0242ac130002] 2023-09-13T16:30:49.685311155Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[dbfb8369-3c07-11ee-be05-0242ac130002], key: metric:dbfb8369-3c07-11ee-be05-0242ac130002 2023-09-13T16:30:49.934377751Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[0f85d06d-3c08-11ee-8713-0242ac130002], key: plugin:dependency:0f85d06d-3c08-11ee-8713-0242ac130002 2023-09-13T16:30:50.643634660Z ts=2023-09-13T16:30:50.643Z caller=head.go:578 level=info component=tsdb msg="WAL checkpoint loaded" 2023-09-13T16:30:50.703857920Z ts=2023-09-13T16:30:50.703Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=619 maxSegment=622 2023-09-13T16:30:50.737192114Z 2023/09/14 00:30:50 info subscribe events 2023-09-13T16:30:50.738325386Z mongo: no documents in result 2023-09-13T16:30:50.738336077Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:50.738340797Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:50.738344898Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:50.738348268Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:50.738351908Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:50.738355958Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:50.738380070Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:50.738384400Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:50.738388481Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:50.738392661Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:50.738396152Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:50.738399532Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:50.738403222Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:50.738624378Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:50.738632829Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:50.738636329Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:50.738640019Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:50.738643529Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:50.738659060Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:50.738662311Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:50.738679422Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:50.738682772Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:50.788570685Z ts=2023-09-13T16:30:50.788Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=620 maxSegment=622 2023-09-13T16:30:50.938028827Z ts=2023-09-13T16:30:50.937Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=621 maxSegment=622 2023-09-13T16:30:50.938108033Z ts=2023-09-13T16:30:50.938Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=622 maxSegment=622 2023-09-13T16:30:50.938167167Z ts=2023-09-13T16:30:50.938Z caller=head.go:619 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=2.06331245s wal_replay_duration=294.550479ms total_replay_duration=2.358784145s 2023-09-13T16:30:51.560316738Z ts=2023-09-13T16:30:51.560Z caller=main.go:993 level=info fs_type=EXT4_SUPER_MAGIC 2023-09-13T16:30:51.560362692Z ts=2023-09-13T16:30:51.560Z caller=main.go:996 level=info msg="TSDB started" 2023-09-13T16:30:51.560367342Z ts=2023-09-13T16:30:51.560Z caller=main.go:1177 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-09-13T16:30:51.560761441Z ts=2023-09-13T16:30:51.560Z caller=main.go:1214 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=453.832µs db_storage=1.431µs remote_storage=1.59µs web_handler=490ns query_engine=1µs scrape=141.42µs scrape_sd=27.352µs notify=1.07µs notify_sd=4.001µs rules=1.75µs tracing=7.3µs 2023-09-13T16:30:51.560796573Z ts=2023-09-13T16:30:51.560Z caller=main.go:957 level=info msg="Server is ready to receive web requests." 2023-09-13T16:30:51.560802744Z ts=2023-09-13T16:30:51.560Z caller=manager.go:941 level=info component="rule manager" msg="Starting rule manager..." 2023-09-13T16:30:51.846940566Z 2023/09/14 00:30:51 info subscribe events 2023-09-13T16:30:51.848286533Z mongo: no documents in result 2023-09-13T16:30:51.848322036Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:51.848326926Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:51.848330616Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:51.848334257Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:51.848352808Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:51.848356888Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:51.848360889Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:51.848364599Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:51.848368219Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:51.848372230Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:51.848376090Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:51.848406982Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:51.848415913Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:51.848417863Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:51.848419623Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:51.848421423Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:51.848423323Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:51.848425253Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:51.848427043Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:51.848428843Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:51.848430754Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:51.848432674Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:53.360018842Z 2023/09/14 00:30:53 info subscribe events 2023-09-13T16:30:53.360941059Z mongo: no documents in result 2023-09-13T16:30:53.360968881Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:53.360973191Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:53.360976742Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:53.360980152Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:53.360984012Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:53.361005914Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:53.361010804Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:53.361014174Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:53.361017435Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:53.361020665Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:53.361024165Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:53.361027895Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:53.361032496Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:53.361057978Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:53.361063258Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:53.361066838Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:53.361070429Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:53.361090340Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:53.361094580Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:53.361103681Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:53.361105731Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:53.361107601Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:56.402736142Z 2023/09/14 00:30:56 info subscribe events 2023-09-13T16:30:56.403781127Z mongo: no documents in result 2023-09-13T16:30:56.403815970Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:56.403820720Z /go/pkg/mod/github.com/crawlab-team/[email protected]/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() 2023-09-13T16:30:56.403824681Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() 2023-09-13T16:30:56.403828341Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:56.403831711Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() 2023-09-13T16:30:56.403834881Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() 2023-09-13T16:30:56.403838131Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() 2023-09-13T16:30:56.403841402Z /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() 2023-09-13T16:30:56.403844502Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() 2023-09-13T16:30:56.403848352Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() 2023-09-13T16:30:56.403851932Z /go/pkg/mod/google.golang.org/[email protected]/server.go:1616 google.golang.org/grpc.(*Server).handleStream() 2023-09-13T16:30:56.403854863Z /go/pkg/mod/google.golang.org/[email protected]/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() 2023-09-13T16:30:56.403859013Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() 2023-09-13T16:30:56.403920337Z rpc error: code = Unknown desc = mongo: no documents in result 2023-09-13T16:30:56.403927018Z /go/pkg/mod/github.com/crawlab-team/[email protected]/trace.go:13 github.com/crawlab-team/go-trace.TraceError() 2023-09-13T16:30:56.403948689Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() 2023-09-13T16:30:56.403953040Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() 2023-09-13T16:30:56.403957000Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() 2023-09-13T16:30:56.403960960Z /go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28 github.com/cenkalti/backoff/v4.Retry() 2023-09-13T16:30:56.403964531Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() 2023-09-13T16:30:56.403968441Z /go/pkg/mod/github.com/crawlab-team/[email protected]/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() 2023-09-13T16:30:56.403972001Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit()```
@tikazyq It is the same error on the pro version, and it happens frequently
I probably managed to pinpoint that it tends to happen while API call to /metrics endpoint is being made. Shortly after it tends to crash the container and rebuild it again with errors akin to the logs provided in my previous comment.
You can switch to the latest version of docker image to resolve this issue.
@tikazyq hey, we've updated images to crawlab-pro:latest but still we have this abnormal issue.
I've noticed that the last modifications on crawlab-pro:latest was made 5 month ago and on crawlab-pro:develop 4 month ago. Can you please verify that fix of abnormal issue is on crawlab-pro:latest and not on crawlab-pro:develop?
@tikazyq Hi , could you tell how to disable the metrics flag
We have observed that there is some kind of co-relation between the metrics and the abnormal status. As long as the metrics are off there is no issue, but some how it gets turned on automatically and all of a sudden the abnormal status starts popping up.
It would be really helpful to know of a way to disable it.
