Add bigtable-attempt to request metadata when the RPC is a retry attempt

[igorbernstein2]

Thanks for stopping by to let us know something could be better!

Is your feature request related to a problem? Please describe. Improve observability of network issues from the bigtable server side.

Describe the solution you'd like When the initial RPC fails, the retry attempts that occur afterwards should populate bigtable-attempt request metadata. The first retry attempt should set the value to 1. If that fails, the 2nd attempt should set it to 2, etc. Please note that the initial RPC should not set the header. This will increase feature parity with java-bigtable.

Describe alternatives you've considered Adding clientside metrics - would give us similar info, but will take a lot longer to implement and will have gaps due to possible connectivity issues with stackdriver

Logging - logging failed attempts can alert customers to retries occuring. However during support cases the customer would have to be aware of this

[bshaffer]

Thank you Igor! I think we can add support for this without too much trouble by modifying the readAll in the ResumableStream class

https://github.com/googleapis/google-cloud-php/blob/7f1ceafc2867125e55ca08f5b9c98ff422909d4a/Bigtable/src/ResumableStream.php#L139-L142

Something like this should work:

// Send in "bigtable-attempt" header on each request
$headers = $this->callOptions['headers'] ?? [];
$headers['bigtable-attempt'] = ++$attempt;

// make RPC call
$stream = call_user_func_array(
    [$this->gapicClient, $this->method],
    [$this->request, ['headers' => $headers] + $this->callOptions]
);

Could you provide a link to the PR or documentation for the Java implementation so that we can compare and make sure they're consistent?

EDIT: I found the Java implementation here: https://github.com/googleapis/java-bigtable/pull/935

[bshaffer]

This has been added for ResumableStream in #7414