Insert initial data into conductor in Docker compose [feature]

[unikzforce]

I've managed to change a docker-compose in a way to Insert my arbitrary Workflows and Taskdefs in startup:

# Docker compose for netflix conductor + elasticsearch + dynomite
version: '2'
services:
  conductor-server:
    environment:
      - CONFIG_PROP=config.properties
    image: conductor:server
    build:
      context: ../
      dockerfile: docker/server/Dockerfile
    ports:
      - 8080:8080
    links:
      - elasticsearch:es
      - dynomite:dyno1
    healthcheck:
      test: curl -f http://localhost:8080/api/metadata/taskdefs/task_0/ || false
      interval: 10s
      timeout: 2s
      retries: 7

  conductor-ui:
    environment:
      - WF_SERVER=http://conductor-server:8080/api/
    image: conductor:ui
    build:
      context: ../
      dockerfile: docker/ui/Dockerfile
    ports:
      - 5000:5000
    links:
      - conductor-server

  dynomite:
    image: v1r3n/dynomite

  elasticsearch:
    image: elasticsearch:5
    command: "-Etransport.host=0.0.0.0"

  init-curl-tasks-and-workflows:
    image: spotify/alpine
    entrypoint: bash /opt/conductor/init.sh
    volumes:
      - ./init-script/init.sh:/opt/conductor/init.sh
    depends_on:
      conductor-server:
        condition: service_healthy
    links:
      - conductor-server:conductor-server

in this docker-compose i added healthcheck to conductor-server and after conductor server state become healthy, i've just started init-curl-tasks-and-workflows in which i've curled my initial taskdefs and workflows into conductor-server.

in the init-curl-tasks-and-workflows container i needed curl to be available so i've used spotify/alpine docker image which had curl into itself.

there was a problem with the healthcheck in conductor-server because curl was not available in it's docker image so i changed it's docker file to add curl inside it:

#
# conductor:server - Netflix conductor server
#
FROM openjdk:8-jre-alpine

MAINTAINER Netflix OSS <[email protected]>

RUN apk update && apk add curl && rm -rf /var/cache/apk/*

# Make app folders
RUN mkdir -p /app/config /app/logs /app/libs

# Copy the project directly onto the image
COPY ./docker/server/bin /app
COPY ./docker/server/config /app/config
COPY ./server/build/libs/conductor-server-*-all.jar /app/libs

# Copy the files for the server into the app folders
RUN chmod +x /app/startup.sh

EXPOSE 8080
EXPOSE 8090

CMD [ "/app/startup.sh" ]
ENTRYPOINT [ "/bin/sh"]

and now i was able to add my arbitrary tasks using init.sh script using curl:

curl -X POST \
  http://conductor-server:8080/api/metadata/taskdefs \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '[
  {
    "name": "t_em_register_user",
    "retryCount": 3,
    "timeoutSeconds": 1200,
    "inputKeys": [
      "studentId",
      "educationalInstituteId",
      "majorName",
      "degree",
      "startSemester",
      "person"
    ],
    "outputKeys": [
      "responseCode",
      "responseStatus"
    ],
    "timeoutPolicy": "TIME_OUT_WF",
    "retryLogic": "FIXED",
    "retryDelaySeconds": 600,
    "responseTimeoutSeconds": 3600
  },
  {
    "name": "t_bm_register_cardholder",
    "retryCount": 3,
    "timeoutSeconds": 1200,
    "inputKeys": [
      "studentId",
      "educationalInstituteId",
      "majorName",
      "degree",
      "startSemester",
      "person"
    ],
    "outputKeys": [
      "responseCode",
      "responseStatus"
    ],
    "timeoutPolicy": "TIME_OUT_WF",
    "retryLogic": "FIXED",
    "retryDelaySeconds": 600,
    "responseTimeoutSeconds": 3600
  }
]'&&
curl -X POST \
  http://conductor-server:8080/api/metadata/workflow \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -d '{
  "name": "w_em_register_student",
  "description": "A workflow for saving student with 2 tasks",
  "version": 1,
  "tasks": [
    {
      "name": "t_em_register_user",
      "taskReferenceName": "t_em_register_user",
      "type": "SIMPLE",
      "startDelay": 0,
      "optional": false
    },
    {
      "name": "t_bm_register_cardholder",
      "taskReferenceName": "t_bm_register_cardholder",
      "type": "SIMPLE",
      "startDelay": 0,
      "optional": false
    }
  ],
  "schemaVersion": 2,
  "restartable": true
}'

should i create a PR for it? and from which branches should i start with?

[kishorebanala]

@unikzforce It's been a while, but this looks good to me. Please send a PR if you're still interested. Also, please make sure to rebase with latest Docker set-up. Thank you.

[unikzforce]

i've just rebased and wanted to test my solution again, but there is a problem with the healthcheck. during my healthcheck interval "kitchensink" example cannot be added to the system. i mean the conductor-server will start and after it started it'll try to add kitchensink, if there was no healthcheck it'll add kitchensink, but if it has a healthcheck it will stuck like this

and it will continue only after healthcheck has been failed. first i thought that my healthcheck test is interrupting. it was like this (in conductor-server service):

version: '2.3'

services:
  conductor-server:
    environment:
      - CONFIG_PROP=config.properties
    image: conductor:server
    build:
      context: ../
      dockerfile: docker/server/Dockerfile
    networks:
      - internal
    ports:
      - 8080:8080
    links:
      - elasticsearch:es
      - dynomite:dyno1
    depends_on:
      elasticsearch:
        condition: service_healthy
      dynomite:
        condition: service_healthy
    # logging:
    #   driver: "json-file"
    #   options:
    #     max-size: "1k"
    #     max-file: "3"
    healthcheck:
      test: curl -f http://localhost:8080/api/metadata/taskdefs/task_0/ || false
      interval: 10s
      timeout: 2s
      retries: 50

  conductor-ui:
    environment:
      - WF_SERVER=http://conductor-server:8080/api/
    image: conductor:ui
    build:
      context: ../
      dockerfile: docker/ui/Dockerfile
    networks:
      - internal
    ports:
      - 5000:5000
    links:
      - conductor-server

  dynomite:
    image: v1r3n/dynomite
    networks:
      - internal
    ports:
      - 8102:8102
    healthcheck:
      test: timeout 5 bash -c 'cat < /dev/null > /dev/tcp/localhost/8102'
      interval: 5s
      timeout: 5s
      retries: 12
    # logging:
    #   driver: "json-file"
    #   options:
    #     max-size: "1k"
    #     max-file: "3"

  # https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.8
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - transport.host=0.0.0.0
      - discovery.type=single-node
      - xpack.security.enabled=false
    networks:
      - internal
    ports:
      - 9200:9200
      - 9300:9300
    healthcheck:
      test: timeout 5 bash -c 'cat < /dev/null > /dev/tcp/localhost/9300'
      interval: 5s
      timeout: 5s
      retries: 12
    logging:
      driver: "json-file"
      options:
        max-size: "1k"
        max-file: "3"

  init-curl-tasks-and-workflows:
    image: spotify/alpine
    networks:
      - internal
    entrypoint: bash /opt/conductor/init.sh
    volumes:
      - ./init-script/init.sh:/opt/conductor/init.sh
    depends_on:
      conductor-server:
        condition: service_healthy
    links:
      - conductor-server:conductor-server

networks:
  internal:

so i changed it from test: curl -f http://localhost:8080/api/metadata/taskdefs/task_0/ || false to test: timeout 2 bash -c 'cat < /dev/null > /dev/tcp/localhost/8080', but still the same reaction happen. do you know how to create a proper healthcheck?

[unikzforce]

after the healthcheck fails:

kitchensink example addition will continue:

I don't want to depend healthcheck on the existence of kitchensink but even the test: timeout 2 bash -c 'cat < /dev/null > /dev/tcp/localhost/8080' will fail anyway.

you can find my code in this fork in docker-healthcheck-and-initial-data-insertion branch.

[kishorebanala]

Hard to tell why the health checks have failed, would you mind posting full error logs?

[unikzforce]

Of course ,i'll send more information ASAP.