Optimize the `Integration Test Suite` to build a reusable Vector image

[pront]

The IT workflow

https://github.com/vectordotdev/vector/blob/master/.github/workflows/integration.yml

Recently I made these tests runs in parallel (https://github.com/vectordotdev/vector/pull/22205) which resulted in significant speedup (~3x). We can do even better here by introducing a job that builds Vector first with --features all-integration-tests and reuse this image!

Status

Partially done. We building and re-using layers (see new job here) but we are still compiling:

Executing: AWS_ACCESS_KEY_ID="dummy" AWS_SECRET_ACCESS_KEY="dummy" CLOUDWATCH_ADDRESS="http://mock-localstack:4566" CONFIG_VECTOR_IMAGE="vector-test-runner-1.91:latest" CONFIG_VERSION="latest" EC2_METADATA_ADDRESS="http://mock-ec2-metadata:1338" ECS_ADDRESS="http://mock-ecs" ELASTICSEARCH_ADDRESS="http://mock-localstack:4566" KINESIS_ADDRESS="http://mock-localstack:4566" KMS_ADDRESS="http://mock-localstack:4566" S3_ADDRESS="http://mock-localstack:4566" SNS_ADDRESS="http://mock-localstack:4566" SQS_ADDRESS="http://mock-localstack:4566" VECTOR_LOG="info" "docker" "exec" "--env" "RUST_BACKTRACE=1" "--env" "CARGO_BUILD_TARGET_DIR=/home/target" "--env" "AWS_ACCESS_KEY_ID" "--env" "AWS_SECRET_ACCESS_KEY" "--env" "CLOUDWATCH_ADDRESS" "--env" "CONFIG_VECTOR_IMAGE" "--env" "CONFIG_VERSION" "--env" "EC2_METADATA_ADDRESS" "--env" "ECS_ADDRESS" "--env" "ELASTICSEARCH_ADDRESS" "--env" "KINESIS_ADDRESS" "--env" "KMS_ADDRESS" "--env" "S3_ADDRESS" "--env" "SNS_ADDRESS" "--env" "SQS_ADDRESS" "--env" "VECTOR_LOG" "vector-test-runner-1.91" "cargo" "nextest" "run" "--no-fail-fast" "--no-default-features" "--features" "all-integration-tests" "--lib" "::aws_" "--no-capture" "--retries" "2"
    $AWS_ACCESS_KEY_ID="dummy"
    $AWS_SECRET_ACCESS_KEY="dummy"
    $CLOUDWATCH_ADDRESS="http://mock-localstack:4566"
    $CONFIG_VECTOR_IMAGE="vector-test-runner-1.91:latest"
    $CONFIG_VERSION="latest"
    $EC2_METADATA_ADDRESS="http://mock-ec2-metadata:1338"
    $ECS_ADDRESS="http://mock-ecs"
    $ELASTICSEARCH_ADDRESS="http://mock-localstack:4566"
    $KINESIS_ADDRESS="http://mock-localstack:4566"
    $KMS_ADDRESS="http://mock-localstack:4566"
    $S3_ADDRESS="http://mock-localstack:4566"
    $SNS_ADDRESS="http://mock-localstack:4566"
    $SQS_ADDRESS="http://mock-localstack:4566"
    $VECTOR_LOG="info"
      Updating git repository `[https://github.com/vectordotdev/nix.git`](https://github.com/vectordotdev/nix.git%60)
      Updating git repository `[https://github.com/MSxDOS/ntapi.git`](https://github.com/MSxDOS/ntapi.git%60)
      Updating git repository `[https://github.com/vectordotdev/tokio`](https://github.com/vectordotdev/tokio%60)
      Updating crates.io index
      Updating git repository `[https://github.com/GreptimeTeam/greptimedb-ingester-rust`](https://github.com/GreptimeTeam/greptimedb-ingester-rust%60)
      Updating git repository `[https://github.com/vectordotdev/heim.git`](https://github.com/vectordotdev/heim.git%60)
      Updating git repository `[https://github.com/tokio-rs/tracing`](https://github.com/tokio-rs/tracing%60)
      Updating git repository `[https://github.com/vectordotdev/vrl.git`](https://github.com/vectordotdev/vrl.git%60)
      Updating git repository `[https://github.com/GreptimeTeam/greptime-proto.git`](https://github.com/GreptimeTeam/greptime-proto.git%60)
   Downloading crates ...
   ...
    Downloaded Inflector v0.11.4
     Compiling proc-macro2 v1.0.101

[pront]

There was a lot of progress in this area. We are now building and reusing an image for the IT suite. Also, there were serval vdev related PRs in the past couple of months.

However, when I experimented with a single unified image for both IT and E2e tests, the experiment hit a roadblock: https://github.com/vectordotdev/vector/pull/24164

I consider this partially done and the motivation to chase this further is not there since the IT and E2E suites now run significantly faster.