Pub/Sub Exactly Once integration test fails

[dbolduc]

• GCB: https://console.cloud.google.com/cloud-build/builds;region=us-east1/57f16414-f9fe-4c7d-adf6-d5bd26fc1fe9;tab=detail?project=cloud-cpp-testing-resources
• Raw: https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/9617/d1e22d3c499626f863299794a1a7274a85dca516/fedora-36-integration-production/log-57f16414-f9fe-4c7d-adf6-d5bd26fc1fe9.txt

Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.

[ RUN      ] SubscriberIntegrationTest.ExactlyOnce
google/cloud/pubsub/integration_tests/subscriber_integration_test.cc:512: Failure
Value of: status
Expected: code is equal to OK and message is anything
  Actual: INVALID_ARGUMENT: Permanent error in AckMessage: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed. (of type google::cloud::v2_1_0::Status), with a code that isn't equal to OK, but a message that is anything
 ack() failed for id=5248797734140914

2022-08-02T15:44:08.840525444Z [DEBUG] <140050853546240> AsyncStreamingPull(65) << request=google.pubsub.v1.StreamingPullRequest {
...
2022-08-02T15:44:13.491764151Z [DEBUG] <140050853546240> Cancel(65) << (google/cloud/pubsub/internal/subscriber_logging.cc:212)
2022-08-02T15:44:13.492032123Z [DEBUG] <140050853546240> Cancel(65) >> (google/cloud/pubsub/internal/subscriber_logging.cc:214)
2022-08-02T15:44:13.492075615Z [DEBUG] <140047340631616> Read(65) >> response={} (google/cloud/pubsub/internal/subscriber_logging.cc:237)
2022-08-02T15:44:13.492075320Z [DEBUG] <140047323846208> AsyncAcknowledge(72) >> future_status=timeout (./google/cloud/internal/log_wrapper.h:228)
2022-08-02T15:44:13.492097065Z [DEBUG] <140047340631616> Finish(65) << (google/cloud/pubsub/internal/subscriber_logging.cc:278)
2022-08-02T15:44:13.492414891Z [DEBUG] <140047432951360> Finish(65) >> status=CANCELLED: CANCELLED (google/cloud/pubsub/internal/subscriber_logging.cc:281)
2022-08-02T15:44:14.129745384Z [DEBUG] <140047374202432> AsyncAcknowledge(69) >> response=OK (./google/cloud/internal/log_wrapper.h:233)
2022-08-02T15:44:14.555002196Z [DEBUG] <140047365809728> AsyncModifyAckDeadline(71) >> response=OK (./google/cloud/internal/log_wrapper.h:233)
2022-08-02T15:44:14.598757905Z [DEBUG] <140047390987840> AsyncAcknowledge(70) >> response=OK (./google/cloud/internal/log_wrapper.h:233)
2022-08-02T15:44:14.668759772Z [DEBUG] <140047416165952> AsyncAcknowledge(72) >> response=INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed. error_info={reason=EXACTLY_ONCE_ACKID_FAILURE, domain=pubsub.googleapis.com, metadata={RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIBAGd4OBtGGg9vM1x1B1ENGHp_ZCNpCxtSAEFZf1VbMz1woeDgoGcNKVBdZHduUhQIBUJQel5fGwRpWHS7hs7V1OzkWG9WYu69l-JlXsKJnNVZZiE9WhJLLD5-MyJFQV5AEkw4F0RJUytDCypYEU4EISE-MD4=PERMANENT_FAILURE_INVALID_ACK_ID}} (./google/cloud/internal/log_wrapper.h:233)