GCP PubSub Source: wont resume fetching messages if queue was empty for a while (vector needs to be restarted)

[vmm-afonso]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We are using Vector (v.0.37.0 of helm chart running in gke) to fetch events from a pubsub topic to an elasticsearch (running inside a gke cluster) sink. Problem here is that our pubsub topic only receives messages when a CloudRun instance runs, which is not always and not necessarily at fixed times and it seems that Vector stops retrying to process events from this source if the queue is empty for a while. I know I might be able to get around this problem by changing the retention time on the topic and scheduling restarts to the vector pod but that is the costly option. Ideally vector would constantly retry connecting to this source every x minutes in case new messages were added to the queue.

Configuration

api:

  address: 127.0.0.1:8686
  enabled: true
  playground: false

data_dir: /vector-data-dir

sinks:

  elasticsearch:
    api_version: auto
    auth: <auth>
    bulk:
      index: <index_name>-%Y-%m-%d
    compression: none
    doc_type: _doc
    endpoints:
    - <endpoint>
    healthcheck: true
    inputs:
    - transform_json
    mode: bulk
    suppress_type_name: true
    type: elasticsearch

sources:

  pubsub:
    endpoint: https://pubsub.googleapis.com
    project: <gcp_project>
    subscription: <pubsub_subscription>
    type: gcp_pubsub

transforms:

  transform_json:
    type: remap
    inputs:
      - elasticsearch
    drop_on_abort: true
    source: |
       . = parse_json!(.message)

Version

CHART: vector-0.32.0 APP VERSION: 0.37.0-distroless-libc

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response