[Bug] context deadline exceeded in integration tests

[jpds]

Describe the bug

I've created an integration test for Temporal for NixOS over at: https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/temporal.nix

This creates a fresh Temporal server in a VM, and then in the testScript section uses the cli to query that server about cluster status and workflows.

For some reason, the CLI calls will randomly fail with, whilst succeeding with a simple reattempt:

waiting for success: temporal workflow list --output json
... level=ERROR msg="failed reaching server: context deadline exceeded"

I've also tried setting --connect-timeout 120s, and it happens with that too.

This has also been encountered in NixOS's CI system: https://hydra.nixos.org/build/306058770/log

Minimal Reproduction

1. Install Nix
2. git clone https://github.com/NixOS/nixpkgs # Large download
3. cd nixpkgs; nix-build -A nixosTests.temporal --check

Environment/Versions

• OS and processor: Linux
• Temporal Version: [e.g. 1.14.0?] and/or SDK version: 1.28.1
• Are you using Docker or Kubernetes or building Temporal from source? Source