Multi container deployments

[duartemvix]

I'm running a simple, Flask python script that's working locally, but by using Kamal 2 conventions I haven't been able to deploy and build an ephemeral container that the script uses for running a few asynchronous jobs. The container should work a job and shut down after each job.

Given the Kamal setup, I'm able to only run a unique Dockerfile for each deployment, but this application runs by using a docker-compose.yml configuration with docker compose up -d --build. I really like Kamal, and it seems better than Capistrano, for Ruby apps, but how do I actually deploy this app, not by building containers with docker build command.

The actual issue I'm runing into is when kamal setup asks me to choose a Dockerfile, but my app doens't have only 1 dockerfile. I build it with docker compose build

Would appreciate any tips, or inputs.

A few snippets:

# docker-compose.yml
services:
  listener:
    container_name: crawler-listener
    build:
      context: .
      dockerfile: Dockerfile.listener
    ports:
      - "8000:8000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock    
    networks:
      - crawler-network
    env_file:
      - .env
    
  job:
    container_name: crawler-job
    build:
      context: .
      dockerfile: Dockerfile.job
    networks:
      - crawler-network
    deploy:
      replicas: 0 # This ensures it doesn't start automatically with `docker compose up`
    env_file:
      - .env

networks:
  crawler-network:
    name: crawler-network
    driver: bridge

Thank you!

[djmb]

kamal app exec is designed to run ephemeral containers. Can you call that from your script?

A Kamal deployment only works with a single image though, so you'd either need separate deployments for your listener and your crawler, or else create a single image that can run both the listener and the crawler job.

[Jainam-17-18]

@duartemvix how did you create multiple instances of the same application and handle requests?

[duartemvix]

@duartemvix how did you create multiple instances of the same application and handle requests?

I guess that you mean how to run multiple containers for each job? All I was doing was: building an image for a standard container that will run the jobs (make sure all dependencies are installed in the image), then I'd use the docker SDK and import the libraries to spin up a container from inside my code, at my /crawl route of the API. The image for containers like that just take a script that you wrote and run it once; shutting down the container afterwards since it isn't meant to run like a server.

# Dockerfile.job
# syntax=docker/dockerfile:1.4

# Use TARGETARG to detect architecture
ARG TARGETARCH=amd64

FROM python:3.10-slim

WORKDIR /app

RUN if [ "$TARGETARCH" = "arm64" ]; then \
    echo "🦾 Installing ARM-specific optimizations"; \
      apt-get update && apt-get install -y --no-install-recommends \
      libopenblas-dev \
      && rm -rf /var/lib/apt/lists/*; \
    elsif \
      apt-get update && apt-get upgrade -y \
      && echo "Running on $TARGETARCH (AMD64 or others)"; \
    fi

RUN pip install --no-cache-dir requests openai crawl4ai \
&& crawl4ai-setup

COPY app/ app/

# Set the default command
CMD ["python", "-um", "app.crawler"]

In the app/ directory from the root folder I have the script that runs the job at app/crawler.py

# main.py
@app.route('/crawl', methods=['POST'])
def crawl():
    api_key = request.headers.get('X-API-Key')
    if api_key != os.getenv('API_KEY'):
        abort(401)

    data = request.get_json()
    urls = data.get('urls', [])
    webhook_url = data.get('webhook_url')

    # Validate URLs
    if not isinstance(urls, list) or not urls:
        return jsonify({"error": "Field 'urls' must be a non-empty list."}), 400
    if len(urls) > MAX_URLS:
        return jsonify({"error": f"Maximum {MAX_URLS} URLs allowed."}), 400
    if not all(isinstance(url, str) and is_valid_url(url) for url in urls):
        return jsonify({"error": "All items in 'urls' must be valid URLs."}), 400

    # Validate webhook_url
    if not webhook_url or not is_valid_url(webhook_url):
        return jsonify({"error": "Valid 'webhook_url' is required."}), 400

    # Generate job ID
    job_id = str(uuid.uuid4())

    try:
        container =  docker_client.containers.run(
            image=IMAGE_NAME,
            command=["python", "app/crawler.py", job_id, webhook_url] + urls,
            detach=True,
            network=DEFAULT_NETWORK,
            auto_remove=True,
            name=f"crawler-job-{job_id}",
            environment={
                "OPENROUTER_API_KEY":os.getenv('OPENROUTER_API_KEY')
            }          
        )

        return jsonify({
            "status": "started",
            "job_id": job_id,
            "container_id": container.id[:12]
        }), 202

    except Exception as e:
        logging.error(f"Error running/starting the container: {str(e)}")
        return jsonify({"error": "Failed to start crawl job."}), 500

The /crawl route is in a separate file at main.py and running as a server to listen to calls. Only to call the containers to run the jobs when requested.

[imWildCat]

second this.

currently we'll have to maintain multiple kamal deploy.yml files for multiple images.