temporal icon indicating copy to clipboard operation
temporal copied to clipboard
verified publisher icon temporalio

Upcoming runs and Interval discrepancies in the UI

Open bodinsamuel opened this issue 2 years ago • 5 comments

Expected Behavior

Listing schedules and their dedicated page should output the same "Upcoming runs" and "Interval"

Actual Behavior

When you list schedules /namespaces/default/schedules you can get some values for the upcoming and interval. But you can get different values when you click on a specific schedule.

https://github.com/temporalio/temporal/assets/1637651/d1e88996-e212-42fc-a6da-0f24bb0a9272

Note: the schedule seems to be respected so I just think it's just a matter of desync between two data storage

Steps to Reproduce the Problem

Main

  • Create a schedule
  • Check schedules homepage
  • Click on a schedule

Other

  • Modify a schedule with
const scheduleHandle = client.schedule.getHandle(schedule_id);
await scheduleHandle.update((scheduleDescription: ScheduleDescription) => {
            scheduleDescription.spec = {
                intervals: [
                    {
                        every: "86400",
                        offset
                    }
                ]
            };
            return scheduleDescription;
        });

Specifications

  • Version: 2.22.2
  • Platform: Docker image

bodinsamuel avatar Jan 24 '24 14:01 bodinsamuel

Do you also see incorrect values from temporal schedule list using the CLI?

dnr avatar Jan 27 '24 00:01 dnr

Hey @dnr same issue on my end The CLI returns the wrong schedules too

e.g: here it says "28 minutes from now" even though the interval is 1day and next run is in > 12hours Screenshot 2024-02-05 at 11 18 59

bodinsamuel avatar Feb 05 '24 10:02 bodinsamuel

What visibility store are you using?

It does look like the update to the memo is not making it to the visibility store. Do you see any errors in the server logs, specifically for the history or worker services, specifically about visibility or memos?

dnr avatar Feb 07 '24 01:02 dnr

I don't see much errors unfortunately, I'm using everything straight up from Docker

version: '3.9'
services:
    foobar-db:
        image: postgres
        container_name: foobar-db
        environment:
            POSTGRES_PASSWORD: foobar
            POSTGRES_USER: foobar
            POSTGRES_DB: foobar
        ports:
            - '${DB_PORT:-5432}:5432'
        volumes:
            - ./foobar-data:/var/lib/postgresql/data

    temporal:
        image: temporalio/auto-setup
        container_name: temporal
        depends_on:
            - foobar-db
        environment:
            - DB=postgresql
            - DB_PORT=5432
            - POSTGRES_USER=foobar
            - POSTGRES_PWD=foobar
            - POSTGRES_SEEDS=foobar-db
        ports:
            - 7233:7233

Startup logs

TEMPORAL_ADDRESS is not set, setting it to 172.18.0.4:7233
Waiting for PostgreSQL to startup.
PostgreSQL started.
Setup PostgreSQL schema.
2024-02-07T14:07:36.261Z	INFO	Starting schema setup	{"config": {"SchemaFilePath":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "setuptask.go:58"}
2024-02-07T14:07:36.261Z	DEBUG	Setting up version tables	{"logging-call-at": "setuptask.go:68"}
2024-02-07T14:07:36.271Z	DEBUG	Current database schema version 1.10 is greater than initial schema version 0.0. Skip version upgrade	{"logging-call-at": "setuptask.go:117"}
2024-02-07T14:07:36.271Z	INFO	Schema setup complete	{"logging-call-at": "setuptask.go:132"}
2024-02-07T14:07:36.283Z	INFO	UpdateSchemeTask started	{"config": {"DBName":"","TargetVersion":"","SchemaDir":"/etc/temporal/schema/postgresql/v96/temporal/versioned","IsDryRun":false}, "logging-call-at": "updatetask.go:97"}
2024-02-07T14:07:36.284Z	DEBUG	Schema Dirs: []	{"logging-call-at": "updatetask.go:195"}
2024-02-07T14:07:36.284Z	DEBUG	found zero updates from current version 1.10	{"logging-call-at": "updatetask.go:127"}
2024-02-07T14:07:36.284Z	INFO	UpdateSchemeTask done	{"logging-call-at": "updatetask.go:120"}
2024-02-07T14:07:36.319Z	INFO	Starting schema setup	{"config": {"SchemaFilePath":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "setuptask.go:58"}
2024-02-07T14:07:36.320Z	DEBUG	Setting up version tables	{"logging-call-at": "setuptask.go:68"}
2024-02-07T14:07:36.332Z	DEBUG	Current database schema version 1.1 is greater than initial schema version 0.0. Skip version upgrade	{"logging-call-at": "setuptask.go:117"}
2024-02-07T14:07:36.332Z	INFO	Schema setup complete	{"logging-call-at": "setuptask.go:132"}
2024-02-07T14:07:36.343Z	INFO	UpdateSchemeTask started	{"config": {"DBName":"","TargetVersion":"","SchemaDir":"/etc/temporal/schema/postgresql/v96/visibility/versioned","IsDryRun":false}, "logging-call-at": "updatetask.go:97"}
2024-02-07T14:07:36.344Z	DEBUG	Schema Dirs: []	{"logging-call-at": "updatetask.go:195"}
2024-02-07T14:07:36.344Z	DEBUG	found zero updates from current version 1.1	{"logging-call-at": "updatetask.go:127"}
2024-02-07T14:07:36.344Z	INFO	UpdateSchemeTask done	{"logging-call-at": "updatetask.go:120"}
Temporal CLI address: 172.18.0.4:7233.
2024/02/07 14:07:36 Loading config; env=docker,zone=,configDir=config
2024/02/07 14:07:36 Loading config files=[config/docker.yaml]
{"level":"info","ts":"2024-02-07T14:07:36.362Z","msg":"Build info.","git-time":"2024-01-12T20:57:30.000Z","git-revision":"ca74b18c7257fdb8a87a27afc89827901e152d15","git-modified":true,"go-arch":"arm64","go-os":"linux","go-version":"go1.20.10","cgo-enabled":false,"server-version":"1.22.4","debug-mode":false,"logging-call-at":"main.go:148"}
{"level":"info","ts":"2024-02-07T14:07:36.362Z","msg":"Updated dynamic config","logging-call-at":"file_based_client.go:195"}
{"level":"warn","ts":"2024-02-07T14:07:36.362Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"main.go:178"}
Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: connection error: desc = "transport: Error while dialing: dial tcp 172.18.0.4:7233: connect: connection refused"
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
Waiting for Temporal server to start...
{"level":"info","ts":"2024-02-07T14:07:36.379Z","msg":"Use rpc address 127.0.0.1:7233 for cluster active.","component":"metadata-initializer","logging-call-at":"fx.go:732"}
{"level":"info","ts":"2024-02-07T14:07:36.385Z","msg":"historyClient: ownership caching disabled","service":"history","logging-call-at":"client.go:82"}
{"level":"info","ts":"2024-02-07T14:07:36.392Z","msg":"Created gRPC listener","service":"history","address":"172.18.0.4:7234","logging-call-at":"rpc.go:152"}
{"level":"info","ts":"2024-02-07T14:07:36.398Z","msg":"Created gRPC listener","service":"matching","address":"172.18.0.4:7235","logging-call-at":"rpc.go:152"}
{"level":"info","ts":"2024-02-07T14:07:36.399Z","msg":"historyClient: ownership caching disabled","service":"matching","logging-call-at":"client.go:82"}
{"level":"info","ts":"2024-02-07T14:07:36.408Z","msg":"historyClient: ownership caching disabled","service":"frontend","logging-call-at":"client.go:82"}
{"level":"info","ts":"2024-02-07T14:07:36.408Z","msg":"Created gRPC listener","service":"frontend","address":"172.18.0.4:7233","logging-call-at":"rpc.go:152"}
{"level":"info","ts":"2024-02-07T14:07:36.412Z","msg":"Service is not requested, skipping initialization.","service":"internal-frontend","logging-call-at":"fx.go:477"}
{"level":"info","ts":"2024-02-07T14:07:36.420Z","msg":"historyClient: ownership caching disabled","service":"worker","logging-call-at":"client.go:82"}
{"level":"info","ts":"2024-02-07T14:07:36.423Z","msg":"PProf not started due to port not set","logging-call-at":"pprof.go:67"}
{"level":"info","ts":"2024-02-07T14:07:36.423Z","msg":"Starting server for services","value":{"frontend":{},"history":{},"matching":{},"worker":{}},"logging-call-at":"server_impl.go:94"}
{"level":"info","ts":"2024-02-07T14:07:36.432Z","msg":"fifo scheduler started","service":"history","logging-call-at":"fifo_scheduler.go:96"}
{"level":"info","ts":"2024-02-07T14:07:36.432Z","msg":"interleaved weighted round robin task scheduler started","service":"history","logging-call-at":"interleaved_weighted_round_robin.go:197"}
{"level":"info","ts":"2024-02-07T14:07:36.432Z","msg":"fifo scheduler started","service":"history","component":"memory-scheduled-queue-processor","logging-call-at":"fifo_scheduler.go:96"}
{"level":"info","ts":"2024-02-07T14:07:36.433Z","msg":"fifo scheduler started","service":"history","logging-call-at":"fifo_scheduler.go:96"}
{"level":"info","ts":"2024-02-07T14:07:36.433Z","msg":"interleaved weighted round robin task scheduler started","service":"history","logging-call-at":"interleaved_weighted_round_robin.go:197"}
{"level":"info","ts":"2024-02-07T14:07:36.433Z","msg":"fifo scheduler started","service":"history","logging-call-at":"fifo_scheduler.go:96"}
{"level":"info","ts":"2024-02-07T14:07:36.433Z","msg":"interleaved weighted round robin task scheduler started","service":"history","logging-call-at":"interleaved_weighted_round_robin.go:197"}
{"level":"info","ts":"2024-02-07T14:07:36.433Z","msg":"fifo scheduler started","service":"history","logging-call-at":"fifo_scheduler.go:96"}
{"level":"info","ts":"2024-02-07T14:07:36.434Z","msg":"interleaved weighted round robin task scheduler started","service":"history","logging-call-at":"interleaved_weighted_round_robin.go:197"}
{"level":"info","ts":"2024-02-07T14:07:36.434Z","msg":"RuntimeMetricsReporter started","service":"worker","logging-call-at":"runtime.go:138"}
{"level":"info","ts":"2024-02-07T14:07:36.434Z","msg":"worker starting","service":"worker","component":"worker","logging-call-at":"service.go:379"}
{"level":"info","ts":"2024-02-07T14:07:36.434Z","msg":"RuntimeMetricsReporter started","service":"frontend","logging-call-at":"runtime.go:138"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"frontend starting","service":"frontend","logging-call-at":"service.go:336"}
{"level":"info","ts":"2024-02-07T14:07:36.435Z","msg":"RuntimeMetricsReporter started","service":"matching","logging-call-at":"runtime.go:138"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"matching starting","service":"matching","logging-call-at":"service.go:91"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"RuntimeMetricsReporter started","service":"history","logging-call-at":"runtime.go:138"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"history starting","service":"history","logging-call-at":"service.go:91"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"Starting to serve on matching listener","service":"matching","logging-call-at":"service.go:104"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"Starting to serve on frontend listener","service":"frontend","logging-call-at":"service.go:355"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"Replication task fetchers started.","logging-call-at":"task_fetcher.go:142"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"none","component":"shard-controller","address":"172.18.0.4:7234","lifecycle":"Started","logging-call-at":"controller_impl.go:136"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"Starting to serve on history listener","service":"history","logging-call-at":"service.go:103"}
{"level":"info","ts":"2024-02-07T14:07:36.436Z","msg":"sequential scheduler started","logging-call-at":"sequential_scheduler.go:96"}
{"level":"info","ts":"2024-02-07T14:07:36.443Z","msg":"Membership heartbeat upserted successfully","address":"172.18.0.4","port":6939,"hostId":"3edf1f1c-c5c2-11ee-8c45-0242ac120004","logging-call-at":"monitor.go:256"}
{"level":"info","ts":"2024-02-07T14:07:36.443Z","msg":"Membership heartbeat upserted successfully","address":"172.18.0.4","port":6933,"hostId":"3edd43a4-c5c2-11ee-8c45-0242ac120004","logging-call-at":"monitor.go:256"}
{"level":"info","ts":"2024-02-07T14:07:36.443Z","msg":"Membership heartbeat upserted successfully","address":"172.18.0.4","port":6935,"hostId":"3edbdd6c-c5c2-11ee-8c45-0242ac120004","logging-call-at":"monitor.go:256"}
{"level":"info","ts":"2024-02-07T14:07:36.443Z","msg":"Membership heartbeat upserted successfully","address":"172.18.0.4","port":6934,"hostId":"3ed9d271-c5c2-11ee-8c45-0242ac120004","logging-call-at":"monitor.go:256"}
{"level":"info","ts":"2024-02-07T14:07:36.444Z","msg":"bootstrap hosts fetched","bootstrap-hostports":"172.18.0.4:6934,172.18.0.4:6935,172.18.0.4:6933,172.18.0.4:6939","logging-call-at":"monitor.go:298"}
{"level":"info","ts":"2024-02-07T14:07:36.444Z","msg":"bootstrap hosts fetched","bootstrap-hostports":"172.18.0.4:6934,172.18.0.4:6935,172.18.0.4:6933,172.18.0.4:6939","logging-call-at":"monitor.go:298"}
{"level":"info","ts":"2024-02-07T14:07:36.444Z","msg":"bootstrap hosts fetched","bootstrap-hostports":"172.18.0.4:6934,172.18.0.4:6935,172.18.0.4:6933,172.18.0.4:6939","logging-call-at":"monitor.go:298"}
{"level":"info","ts":"2024-02-07T14:07:36.444Z","msg":"bootstrap hosts fetched","bootstrap-hostports":"172.18.0.4:6933,172.18.0.4:6939,172.18.0.4:6934,172.18.0.4:6935","logging-call-at":"monitor.go:298"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.18.0.4:7234"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","component":"shard-controller","address":"172.18.0.4:7234","component":"shard-controller","address":"172.18.0.4:7234","shard-update":"RingMembershipChangedEvent","number-processed":1,"number-deleted":0,"logging-call-at":"ownership.go:116"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-context","logging-call-at":"context_impl.go:1482"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","component":"shard-controller","address":"172.18.0.4:7234","numShards":1,"logging-call-at":"controller_impl.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-context","logging-call-at":"context_impl.go:1482"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","component":"shard-controller","address":"172.18.0.4:7234","numShards":2,"logging-call-at":"controller_impl.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-context","logging-call-at":"context_impl.go:1482"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","component":"shard-controller","address":"172.18.0.4:7234","numShards":3,"logging-call-at":"controller_impl.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-context","logging-call-at":"context_impl.go:1482"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"none","component":"shard-controller","address":"172.18.0.4:7234","numShards":4,"logging-call-at":"controller_impl.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.448Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.18.0.4:7239"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.449Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["172.18.0.4:7233"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.449Z","msg":"Frontend is now healthy","service":"frontend","logging-call-at":"workflow_handler.go:219"}
{"level":"info","ts":"2024-02-07T14:07:36.450Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["172.18.0.4:7235"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.450Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["172.18.0.4:7233"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.453Z","msg":"Range updated for shardID","shard-id":2,"address":"172.18.0.4:7234","shard-range-id":4,"previous-shard-range-id":3,"number":0,"next-number":0,"logging-call-at":"context_impl.go:1147"}
{"level":"info","ts":"2024-02-07T14:07:36.453Z","msg":"Acquired shard","shard-id":2,"address":"172.18.0.4:7234","logging-call-at":"context_impl.go:1839"}
{"level":"info","ts":"2024-02-07T14:07:36.453Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","lifecycle":"Starting","component":"shard-engine","logging-call-at":"context_impl.go:1358"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Starting","logging-call-at":"history_engine.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"Task rescheduler started.","shard-id":2,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Starting","logging-call-at":"memory_scheduled_queue.go:103"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Started","logging-call-at":"memory_scheduled_queue.go:108"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"Task rescheduler started.","shard-id":2,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"Task rescheduler started.","shard-id":2,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"Task rescheduler started.","shard-id":2,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Started","logging-call-at":"history_engine.go:295"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"none","shard-id":2,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-engine","logging-call-at":"context_impl.go:1361"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"queue reader started","shard-id":2,"address":"172.18.0.4:7234","component":"transfer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"queue reader started","shard-id":2,"address":"172.18.0.4:7234","component":"visibility-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"queue reader started","shard-id":2,"address":"172.18.0.4:7234","component":"timer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"queue reader started","shard-id":2,"address":"172.18.0.4:7234","component":"archival-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"history client encountered error","service":"frontend","error":"Not enough hosts to serve the request","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:104"}
{"level":"info","ts":"2024-02-07T14:07:36.454Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.18.0.4:7234"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"Range updated for shardID","shard-id":4,"address":"172.18.0.4:7234","shard-range-id":5,"previous-shard-range-id":4,"number":0,"next-number":0,"logging-call-at":"context_impl.go:1147"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"Acquired shard","shard-id":4,"address":"172.18.0.4:7234","logging-call-at":"context_impl.go:1839"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","lifecycle":"Starting","component":"shard-engine","logging-call-at":"context_impl.go:1358"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Starting","logging-call-at":"history_engine.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"Task rescheduler started.","shard-id":4,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"Task rescheduler started.","shard-id":4,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Starting","logging-call-at":"memory_scheduled_queue.go:103"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Started","logging-call-at":"memory_scheduled_queue.go:108"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"Task rescheduler started.","shard-id":4,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"Task rescheduler started.","shard-id":4,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Started","logging-call-at":"history_engine.go:295"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"none","shard-id":4,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-engine","logging-call-at":"context_impl.go:1361"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"queue reader started","shard-id":4,"address":"172.18.0.4:7234","component":"archival-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"queue reader started","shard-id":4,"address":"172.18.0.4:7234","component":"transfer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"queue reader started","shard-id":4,"address":"172.18.0.4:7234","component":"visibility-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.459Z","msg":"queue reader started","shard-id":4,"address":"172.18.0.4:7234","component":"timer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"Range updated for shardID","shard-id":3,"address":"172.18.0.4:7234","shard-range-id":25,"previous-shard-range-id":24,"number":0,"next-number":0,"logging-call-at":"context_impl.go:1147"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"Range updated for shardID","shard-id":1,"address":"172.18.0.4:7234","shard-range-id":4,"previous-shard-range-id":3,"number":0,"next-number":0,"logging-call-at":"context_impl.go:1147"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"Acquired shard","shard-id":3,"address":"172.18.0.4:7234","logging-call-at":"context_impl.go:1839"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","lifecycle":"Starting","component":"shard-engine","logging-call-at":"context_impl.go:1358"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Starting","logging-call-at":"history_engine.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"Task rescheduler started.","shard-id":3,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":3,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Starting","logging-call-at":"memory_scheduled_queue.go:103"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Started","logging-call-at":"memory_scheduled_queue.go:108"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.460Z","msg":"Acquired shard","shard-id":1,"address":"172.18.0.4:7234","logging-call-at":"context_impl.go:1839"}
{"level":"error","ts":"2024-02-07T14:07:36.460Z","msg":"Unable to process new range","shard-id":3,"address":"172.18.0.4:7234","component":"archival-queue-processor","error":"shard status unknown","logging-call-at":"queue_base.go:316","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/history/queues.(*queueBase).processNewRange\n\t/home/builder/temporal/service/history/queues/queue_base.go:316\ngo.temporal.io/server/service/history/queues.(*scheduledQueue).processEventLoop\n\t/home/builder/temporal/service/history/queues/queue_scheduled.go:218"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":3,"address":"172.18.0.4:7234","component":"transfer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","lifecycle":"Starting","component":"shard-engine","logging-call-at":"context_impl.go:1358"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":3,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":3,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Started","logging-call-at":"history_engine.go:295"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":3,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-engine","logging-call-at":"context_impl.go:1361"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":3,"address":"172.18.0.4:7234","component":"visibility-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":3,"address":"172.18.0.4:7234","component":"timer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Starting","logging-call-at":"history_engine.go:286"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":1,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"visibility-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":1,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"timer-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Starting","logging-call-at":"queue_scheduled.go:154"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":1,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"archival-queue-processor","lifecycle":"Started","logging-call-at":"queue_scheduled.go:163"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Starting","logging-call-at":"queue_immediate.go:114"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"Task rescheduler started.","shard-id":1,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"rescheduler.go:124"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"transfer-queue-processor","lifecycle":"Started","logging-call-at":"queue_immediate.go:123"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Starting","logging-call-at":"memory_scheduled_queue.go:103"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","service":"history","component":"memory-scheduled-queue-processor","lifecycle":"Started","logging-call-at":"memory_scheduled_queue.go:108"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","component":"history-engine","lifecycle":"Started","logging-call-at":"history_engine.go:295"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"none","shard-id":1,"address":"172.18.0.4:7234","lifecycle":"Started","component":"shard-engine","logging-call-at":"context_impl.go:1361"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":3,"address":"172.18.0.4:7234","component":"archival-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":1,"address":"172.18.0.4:7234","component":"archival-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":1,"address":"172.18.0.4:7234","component":"timer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":1,"address":"172.18.0.4:7234","component":"transfer-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.461Z","msg":"queue reader started","shard-id":1,"address":"172.18.0.4:7234","component":"visibility-queue-processor","queue-reader-id":0,"lifecycle":"Started","logging-call-at":"reader.go:182"}
{"level":"info","ts":"2024-02-07T14:07:36.486Z","msg":"temporal-sys-history-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"scanner.go:292"}
{"level":"info","ts":"2024-02-07T14:07:36.501Z","msg":"temporal-sys-tq-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"scanner.go:292"}
{"level":"info","ts":"2024-02-07T14:07:36.818Z","msg":"matching client encountered error","service":"frontend","error":"Not enough hosts to serve the request","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2024-02-07T14:07:36.818Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["172.18.0.4:7235"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:36.819Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-tq-scanner-taskqueue-0","WorkerID":"1@b03267cd203a@","logging-call-at":"scanner.go:239"}
{"level":"info","ts":"2024-02-07T14:07:36.820Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:f3b35e8f-0d28-4767-9623-7717a040eba8","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.820Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.820Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-history-scanner-taskqueue-0","WorkerID":"1@b03267cd203a@","logging-call-at":"scanner.go:239"}
{"level":"info","ts":"2024-02-07T14:07:36.822Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.823Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:3ac1559a-8301-4a71-bd0d-08e2dca1a7d6","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.823Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.824Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.824Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.824Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.826Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.827Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.827Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:45597881-e322-4026-8db6-bb7be9caee4b","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.827Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-archival-tq","WorkerID":"1@b03267cd203a@","logging-call-at":"client_worker.go:137"}
{"level":"info","ts":"2024-02-07T14:07:36.828Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-archival-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.828Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-archival-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.828Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:c5e98077-bb55-4b7a-9efe-2f1887ec4d61","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.828Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-processor-parent-close-policy","WorkerID":"1@b03267cd203a@","logging-call-at":"processor.go:98"}
{"level":"info","ts":"2024-02-07T14:07:36.829Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.829Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-archival-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.829Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.829Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.830Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-batcher-taskqueue","WorkerID":"1@b03267cd203a@","logging-call-at":"batcher.go:98"}
{"level":"info","ts":"2024-02-07T14:07:36.831Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:39e46baa-727a-4f29-83f5-ab4d5a7f9db4","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.831Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-batcher-taskqueue","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.831Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.832Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.832Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-batcher-taskqueue","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.832Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"default-worker-tq","WorkerID":"1@b03267cd203a@","logging-call-at":"worker.go:101"}
{"level":"info","ts":"2024-02-07T14:07:36.832Z","msg":"none","component":"worker-manager","lifecycle":"Started","logging-call-at":"worker.go:106"}
{"level":"info","ts":"2024-02-07T14:07:36.832Z","msg":"none","component":"perns-worker-manager","lifecycle":"Starting","logging-call-at":"pernamespaceworker.go:166"}
{"level":"info","ts":"2024-02-07T14:07:36.832Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.833Z","msg":"none","component":"perns-worker-manager","lifecycle":"Started","logging-call-at":"pernamespaceworker.go:177"}
{"level":"info","ts":"2024-02-07T14:07:36.833Z","msg":"worker service started","service":"worker","component":"worker","address":"172.18.0.4:7239","logging-call-at":"service.go:418"}
{"level":"info","ts":"2024-02-07T14:07:36.833Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:08a24322-0c44-4105-a485-4e3582ac172d","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.834Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.834Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.834Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.835Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.835Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:c0fbfdff-ff3f-4ce5-948d-49d5c3a7d0ed","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.835Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.835Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"b03267cd203a:77c51f4a-17b4-488f-a49b-5e6aa10ee3e0","wf-task-queue-type":"Workflow","wf-namespace":"default","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.835Z","msg":"Started Worker","service":"worker","Namespace":"default","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"server-worker@1@b03267cd203a@default","logging-call-at":"pernamespaceworker.go:483"}
{"level":"info","ts":"2024-02-07T14:07:36.835Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"server-worker@1@b03267cd203a@temporal-system","logging-call-at":"pernamespaceworker.go:483"}
{"level":"info","ts":"2024-02-07T14:07:36.836Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"default","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.836Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.836Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
{"level":"info","ts":"2024-02-07T14:07:36.852Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","lifecycle":"Started","logging-call-at":"task_queue_manager.go:331"}
Temporal server started.
Registering default namespace: default.
  NamespaceInfo.Name                    default
  NamespaceInfo.Id                      283870ae-962a-4047-ba77-77dabaf55f47
  NamespaceInfo.Description             Default namespace for Temporal
                                        Server.
  NamespaceInfo.OwnerEmail
  NamespaceInfo.State                   Registered
  NamespaceInfo.Data                    map[]
  Config.WorkflowExecutionRetentionTtl  24h0m0s
  ReplicationConfig.ActiveClusterName   active
  ReplicationConfig.Clusters            [&ClusterReplicationConfig{ClusterName:active,}]
  Config.HistoryArchivalState           Disabled
  Config.VisibilityArchivalState        Disabled
  IsGlobalNamespace                     false
  FailoverVersion                                                                      0
  FailoverHistory                       []

Default namespace default already registered.
             Name                Type
  BatcherNamespace            Keyword
  BatcherUser                 Keyword
  BinaryChecksums             KeywordList
  BuildIds                    KeywordList
  CloseTime                   Datetime
  CustomBoolField             Bool
  CustomDatetimeField         Datetime
  CustomDoubleField           Double
  CustomIntField              Int
  CustomKeywordField          Keyword
  CustomStringField           Text
  CustomTextField             Text
  ExecutionDuration           Int
  ExecutionStatus             Keyword
  ExecutionTime               Datetime
  HistoryLength               Int
  HistorySizeBytes            Int
  RunId                       Keyword
  StartTime                   Datetime
  StateTransitionCount        Int
  TaskQueue                   Keyword
  TemporalChangeVersion       KeywordList
  TemporalNamespaceDivision   Keyword
  TemporalSchedulePaused      Bool
  TemporalScheduledById       Keyword
  TemporalScheduledStartTime  Datetime
  WorkflowId                  Keyword
  WorkflowType                Keyword
Namespace cache refreshed.
Adding Custom*Field search attributes.
Search attributes already exist
{"level":"info","ts":"2024-02-07T14:07:46.449Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.18.0.4:7239"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.450Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["172.18.0.4:7233"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.450Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.18.0.4:7239"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.451Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["172.18.0.4:7239"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.451Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.18.0.4:7234"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.451Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["172.18.0.4:7235"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.451Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["172.18.0.4:7234"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.451Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["172.18.0.4:7233"],"logging-call-at":"service_resolver.go:279"}
{"level":"info","ts":"2024-02-07T14:07:46.451Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["172.18.0.4:7235"],"logging-call-at":"service_resolver.go:279"}

bodinsamuel avatar Feb 07 '24 14:02 bodinsamuel

It looks like you're using standard visibility on postgres. Schedule metadata in list results isn't supported with standard visibility. You should configure advanced visibility on postgres (or elasticsearch if you prefer). There's more information here: https://docs.temporal.io/visibility https://github.com/temporalio/temporal/releases/tag/v1.20.0

dnr avatar Feb 09 '24 01:02 dnr