Add task queue query

[mfateev]

Is your feature request related to a problem? Please describe. Currently, there is no way to query a specific worker for information as queries are always related to workflow execution. For example it is not possible to query a worker for activity types it supports.

Describe the solution you'd like Add task queue query concept. Such a query specifies a task queue name to send the query to. Then a worker picks it up and replies using a new API like RespondTaskQueueQueryCompleted.

Additional context This would support the ability to query for registered activity and workflow types, worker configuration options as well as execute activities synchronously.

https://community.temporal.io/t/list-registered-activities/1927

[ilmn-aeolus]

We are exploring a design where third parties would provide activity workers by running our activity daemon that communicates to a pre-configured, party-specific namespace. This feature would allow our central service to monitor whether third-parties have active workers.

[ilmn-aeolus]

Also, does this ticket service https://community.temporal.io/t/ability-to-inspect-task-queues/1765 ?

[swyxio]

discussing with Samar an alternative name for this is "worker queries"

• "interceptor for starting a workflow" for input validation before starting a workflow
• we think we can implement worker queries first, THEN synchronous start https://github.com/temporalio/temporal/issues/804

[jlegrone]

This would support the ability to query for registered activity and workflow types

We do something like this internally, inspired by the GRPC server reflection protocol. Our "reflection" service is just a Temporal workflow type that each of our workers exposes. The reflection workflow is automatically registered and implemented via our worker package, but if this were moved upstream each language SDK could probably do the same.

Internally we also use protobuf to define workflow/activity/signal/query interfaces, so we expose the message descriptors for those payloads as well. This is the full reflection interface:

message MessageDescriptor {
    // Serialized message descriptor. We avoid taking a dependency on
    // descriptor.proto, which uses proto2 only features, by making this opaque
    // bytes instead.
    //
    // Source: https://github.com/protocolbuffers/protobuf/blob/c57ee04cf32274681f45a0c0acb35e8605fff9e7/src/google/protobuf/descriptor.proto#L93-L126
    bytes message_descriptor_proto = 1;
}

message SignalType {
    string name = 1;
    // A description of the signal.
    string doc = 2;
    MessageDescriptor payload_type = 3;
}

message QueryType {
    string name = 1;
    // A description of the query.
    string doc = 2;
    MessageDescriptor request_type = 3;
    MessageDescriptor response_type = 4;
}

message WorkflowType {
    string name = 1;
    // A description of the workflow.
    string doc = 2;
    temporal.StartWorkflowOptions default_options = 3;
    MessageDescriptor request_type = 4;
    MessageDescriptor response_type = 5;
    repeated SignalType signals = 6;
    repeated QueryType queries = 7;
}

message ActivityType {
    string name = 1;
    // A description of the activity.
    string doc = 2;
    temporal.ActivityOptions default_options = 3;
    MessageDescriptor request_type = 4;
    MessageDescriptor response_type = 5;
}

message ReflectResponse {
    repeated WorkflowType workflows = 1;
    repeated ActivityType activities = 2;
}

// The `Reflection` interface is exposed by every worker on its own task queue.
// This allows external systems to query the list of workflows, activities,
// signals, and queries supported on any task queue.
//
// To use the Reflection service, the client should send a request to execute
// the Reflect workflow on the target task queue. If the worker listening on
// the task queue has enabled Reflection, it will respond with a list of
// available workflows and activities.
service Reflection {
    // Reflect returns a list of workflows and activities registered on the current task queue.
    rpc Reflect(google.protobuf.Empty) returns (ReflectResponse) {
        option (temporal.workflow).config = {};
    };
}

[jlegrone]

Add task queue query concept. Such a query specifies a task queue name to send the query to. Then a worker picks it up and replies using a new API like RespondTaskQueueQueryCompleted.

What does a new API provide vs. a pre-defined workflow type?

[bergundy]

We should consider supporting querying each worker separately so we can compare that they're all the same version and contain the same set of workflows and activities.

[mdross95]

Any update with this?

[JacobJae]

Any update on this?

[skyf0cker]

Any update on this?