Option for outputting service logs during execution

[KevinGrandon]

The problem: Services which are spun up with depends_on will not display log output during execution.

We have a docker-compose service list as follows:

version: '2.1'
services:
  my-server:
    build: .
    command: yarn start
    expose:
      - 8080
    ports:
      - "8080:8080"
    network_mode: "host"
    healthcheck:
      test: ["CMD-SHELL", "curl -H \"Accept: text/html\" -f http://localhost:8080 || exit 1"]
      interval: 5s
      timeout: 10s
      retries: 5

  server-healthy:
    build: .
    network_mode: "host"
    depends_on:
      my-server:
        condition: service_healthy

  run-service:
    build: .
    network_mode: "host"
    depends_on:
      - server-healthy
    command: ./test.sh

It seems that this is either not documented, or it's not supported, but right now it seems impossible to get the log output of my-server with this plugin. (Docker-compose eats depends_on service log output by default).

Solution for local development: Developers can manually view output in another terminal tab, via a command like: docker-compose logs -f my-service

Possible solutions for this plugin:

• One possible solution would be a new configuration option allowing developers to tail docker-compose output of selected services. An example of the pipeline implementation:

  - name: 'run tests'
    command: ./test.sh
    plugins:
      'docker-compose#v1.8.0':
        run: run-service
        logs:
          - my-server
          - my-other-service

• It might also be nice if we could have multiple UI tabs for output of different services in the Buildkite frontend, but that would likely require changes in the frontend product, and is not nearly as important as being able to see the logs in the first place.
• If we can't have background tabs during execution, perhaps running docker-compose logs -f <services> & before the docker-compose run script would be sufficient.

[KevinGrandon]

Perhaps there would be some way to do this in userland if we had https://github.com/buildkite-plugins/docker-compose-buildkite-plugin/issues/47, though multiple logs per job would be even more amazing.

[lox]

Sorry for the delay responding @KevinGrandon. I like the suggestion of the top-level logs block!

[lox]

Currently we capture logs of dependant services as artifacts on failure, is this not adequate @KevinGrandon? The more I look at this, the gnarlier it is to implement.

[KevinGrandon]

Thanks for checking in. The captures artifacts are better than noting, but a way to configure live output of service logs would be really welcome. If there was something that wasn’t a crazy amount of effort but provided live service output that would be really useful.

Maybe just an option to background tail all logs before execution?

[lox]

Maybe just an option to background tail all logs before execution?

I'll look into this, but the issues is that docker-compose run creates ephemeral services that can't easily be docker-compose log -f'd. I'll look for a workaround.

[lox]

Also docker-compose logs and docker logs will exit if the containers or services aren't running yet.

[toote]

While I do believe that there are situations when something like this can be helpful, as mentioned back in 2018, logs as artifacts uploaded after the fact appear to have been enough.

I've been investigating a way to implement this but I can't think of a way to do this while still allowing the plugin to continue to run according to the main application container and shut down services afterwards with the current plugin structure. Add to that the fact that output for the main application as well as all other containers will probably be interleaved and would make it a nightmare to differentiate between them live.

Closing for now