Per task queue ratelimiting should only count valid activity tasks

[yycptt]

Expected Behavior

Only valid (started) tasks should count towards per task queue ratelimiting. (when user specifies TaskQueueActivitiesPerSecond in worker config)

Actual Behavior

https://github.com/temporalio/temporal/blob/master/service/matching/matcher.go#L228

All tasks, regardless of whether its expired or failed to start or successfully started will consume token from the rate limiter, since we perform the rate limiting step when attempting a match.

For example when there's a backlog of expired tasks in the task buffer, all tasks will consume the rate limiting token and throttle how fast the backlog can be drained. However, from user/worker perspective, there's no dispatched tasks at all.

Steps to Reproduce the Problem

Specifications

• Version:
• Platform: