[Bug] “sendgrid+api” dsn scheme is not supported after installing Symfony SendGrid mailer

[Insannik]

Describe the bug

I'm trying to install the Symfony SendGrid mailer plugin as described here.

We run Mautic v5.2.5 in Docker and since the documentation for docker-mautic states the following:

Currently this image has no easy way to extend Mautic (e.g. adding extra composer dependencies or installing extra plugins or themes). This is an ongoing effort we hope to support in an upcoming 5.x release. For now, please build your own images based on the official ones to add the needed dependencies, plugins and themes.

I've tried to slightly extend the Dockerfile in order to get composer and SendGrid mailer to work.

After trying to configure Mautic to use the SendGrid API, I get the following error in the UI:

Steps to reproduce

Extend the Dockerfile as shown below.

RUN cd /opt && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer create-project mautic/recommended-project:${MAUTIC_VERSION} mautic --no-interaction && \
    rm -rf /opt/mautic/var/cache/js && \
    find /opt/mautic/node_modules -mindepth 1 -maxdepth 1 -not \( -name 'jquery' -or -name 'vimeo-froogaloop2' \) | xargs rm -rf

Becomes:

# Install Mautic using composer, install Symfony Sengrid Mailer and clean cache
RUN cd /opt && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer create-project mautic/recommended-project:${MAUTIC_VERSION} mautic --no-interaction && \
    cd /opt/mautic && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer require symfony/sendgrid-mailer:* && \
    rm -rf /opt/mautic/var/cache/js && \
    find /opt/mautic/node_modules -mindepth 1 -maxdepth 1 -not \( -name 'jquery' -or -name 'vimeo-froogaloop2' \) | xargs rm -rf

This builds without errors. Inspecting the files by running an interactive shell shows that the composer.json file now references the sendgrid-mailer plugin and the vendor folder contains the necessary files. The full Dockerfile can be found below.

Deploy the container as normal and try to configure the SendGrid API using the documentation here.

Expected behavior

I expect to be able to use the sendgrid+api scheme when configuring Email Settings via the UI.

Actual behavior

After configuring the Email Settings to use the sendgrid+api scheme as described here and saving the configuration, the frontend shows an error message saying “sendgrid+api” dsn scheme is not supported.

Docker Image Tag

Based on: mautic/mautic:5.2.5-apache

Server Architecture

AMD64 (Intel or AMD processors)

Docker Image Digest (Optional but helpful)

No response

Dockerfile (if applicable)

FROM php:8.1-apache as builder

LABEL vendor="Mautic"
LABEL maintainer="Mautic core team <>"

# Install PHP extensions
RUN apt-get update && apt-get install --no-install-recommends -y \
    ca-certificates \
    build-essential  \
    git \
    curl \
    libcurl4-gnutls-dev \
    libc-client-dev \
    libkrb5-dev \
    libmcrypt-dev \
    libssl-dev \
    libxml2-dev \
    libzip-dev \
    libjpeg-dev \
    libmagickwand-dev \
    libpng-dev \
    libgif-dev \
    libtiff-dev \
    libz-dev \
    libpq-dev \
    imagemagick \
    graphicsmagick \
    libwebp-dev \
    libjpeg62-turbo-dev \
    libxpm-dev \
    libaprutil1-dev \
    libicu-dev \
    libfreetype6-dev \
    libonig-dev \
    librabbitmq-dev \
    unzip \
    nodejs \
    npm

RUN curl -L -o /tmp/amqp.tar.gz "https://github.com/php-amqp/php-amqp/archive/refs/tags/v2.1.1.tar.gz" \
    && mkdir -p /usr/src/php/ext/amqp \
    && tar -C /usr/src/php/ext/amqp -zxvf /tmp/amqp.tar.gz --strip 1 \
    && rm /tmp/amqp.tar.gz

RUN docker-php-ext-configure gd --with-freetype --with-jpeg \
    && docker-php-ext-configure imap --with-kerberos --with-imap-ssl \
    && docker-php-ext-configure opcache --enable-opcache \
    && docker-php-ext-install intl mbstring mysqli curl pdo_mysql zip bcmath sockets exif amqp gd imap opcache \
    && docker-php-ext-enable intl mbstring mysqli curl pdo_mysql zip bcmath sockets exif amqp gd imap opcache

# Install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer

RUN echo "memory_limit = -1" > /usr/local/etc/php/php.ini

# Define Mautic version by package tag
ARG MAUTIC_VERSION=5.2.5

# Install Mautic using composer, install Symfony Sengrid Mailer and clean cache
RUN cd /opt && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer create-project mautic/recommended-project:${MAUTIC_VERSION} mautic --no-interaction && \
    cd /opt/mautic && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer require symfony/sendgrid-mailer:* && \
    rm -rf /opt/mautic/var/cache/js && \
    find /opt/mautic/node_modules -mindepth 1 -maxdepth 1 -not \( -name 'jquery' -or -name 'vimeo-froogaloop2' \) | xargs rm -rf

FROM php:8.1-apache

COPY --from=builder /usr/local/lib/php/extensions /usr/local/lib/php/extensions
COPY --from=builder /usr/local/etc/php/conf.d/ /usr/local/etc/php/conf.d/

COPY --from=builder --chown=www-data:www-data /opt/mautic /var/www/html

# Install PHP extensions requirements and other dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
    unzip libwebp-dev libzip-dev libfreetype6-dev libjpeg62-turbo-dev libpng-dev libc-client-dev librabbitmq4 \
    mariadb-client supervisor cron \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
    && rm -rf /var/lib/apt/lists/* \
    && rm /etc/cron.daily/*

# Setting PHP properties
ENV PHP_INI_VALUE_DATE_TIMEZONE='UTC' \
    PHP_INI_VALUE_MEMORY_LIMIT=512M \
    PHP_INI_VALUE_UPLOAD_MAX_FILESIZE=512M \
    PHP_INI_VALUE_POST_MAX_FILESIZE=512M \
    PHP_INI_VALUE_MAX_EXECUTION_TIME=300

COPY ./common/php.ini /usr/local/etc/php/php.ini

ENV APACHE_DOCUMENT_ROOT=/var/www/html/docroot

RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf

# Enable Apache Rewrite Module
RUN a2enmod rewrite

COPY ./common/docker-entrypoint.sh /entrypoint.sh
COPY ./common/entrypoint_mautic_web.sh /entrypoint_mautic_web.sh
COPY ./common/entrypoint_mautic_cron.sh /entrypoint_mautic_cron.sh
COPY ./common/entrypoint_mautic_worker.sh /entrypoint_mautic_worker.sh

# Apply necessary permissions
RUN ["chmod", "+x", "/entrypoint.sh", "/entrypoint_mautic_web.sh", "/entrypoint_mautic_cron.sh", "/entrypoint_mautic_worker.sh"]

# Setting worker env vars
ENV DOCKER_MAUTIC_WORKERS_CONSUME_EMAIL=2 \
    DOCKER_MAUTIC_WORKERS_CONSUME_HIT=2 \
    DOCKER_MAUTIC_WORKERS_CONSUME_FAILED=2

COPY ./common/supervisord.conf /etc/supervisor/conf.d/supervisord.conf

# Define Mautic volumes to persist data
VOLUME /var/www/html/config
VOLUME /var/www/html/var/logs
VOLUME /var/www/html/docroot/media

WORKDIR /var/www/html/docroot

ENV DOCKER_MAUTIC_ROLE=mautic_web \
    DOCKER_MAUTIC_RUN_MIGRATIONS=false \
    DOCKER_MAUTIC_LOAD_TEST_DATA=false

ENTRYPOINT ["/entrypoint.sh"]

CMD ["apache2-foreground"]

Docker Compose (if applicable)

We do not use Docker Compose. Instead, the 3 docker containers (web, worker, cron) are deployed using Ansible. Our Mautic setup has been functional for a few years now and has recently been updated to v5.2.5 without issues.

Additional context (Optional)

Please note that I'm not 100% sure this is a bug even though I suspect it is.

It could very well be I'm taking the wrong approach trying to expand the docker image to support the SendGrid API.

Any help or advice is appreciated.

Acknowledgements

• [x] I agree to follow this project's Code of Conduct
• [x] This issue is NOT a security vulnerability. (Security vulnerabilities should be reported via the Security tab.)

[cibero42]

Hey @Insannik ,

Sorry for the delay - could you try doing your Dockerfile based on the official container? Just so that we can start diag this.

Example

[Insannik]

Hey @cibero42,

I've taken a look at your proposed solution but I don't think the use of the Sendgrid API is classified as installing "plugin". According to the docs here, it requires the installation of a "package" using composer.

If I understand correctly, "plugins" are the ones you can install through the frontend (integrations -> plugins). However, the symfony/sendgrid-mailer package needs to be installed in a default installation (not using Docker) by using SSH to connect to the server and running a few commands.

So, I could be wrong but I don't think it can be downloaded and added as a "plugin" in the container. It has to be installed using composer like I tried initially.

Thanks again for the help.

Kind regards, Insannik

[cibero42]

Hello @Insannik ,

Yes, you're correct. I'll have a better look at your issue next week - I'll check if it is viable to provide this support by default.

[Insannik]

Hi again @cibero42,

I hope we're able to figure something out. In any case, thanks in advance for the help.

Kind regards, Insannik

[Insannik]

Hi,

I think we may have figured it out. The issue was that the cache wasn't being cleared as I suspected it would've been when a container is launched from scratch. Apparently the cache is stored as part of the file system and is copied to the final container from the builder container because of the following line in the Dockerfile:

COPY --from=builder --chown=www-data:www-data /opt/mautic /var/www/html

I've now changed the Dockerfile I posted above to include the php bin/console cache:clear and php bin/console cache:warmup command after the installation of the sendgrid-mailer package as follows:

# Install Mautic using composer, install Symfony Sengrid Mailer and clean cache. Note that the cache:clear command is required to load the newly installed sendgrid-mailer package.
RUN cd /opt && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer create-project mautic/recommended-project:${MAUTIC_VERSION} mautic --no-interaction && \
    cd /opt/mautic && \
    COMPOSER_ALLOW_SUPERUSER=1 COMPOSER_PROCESS_TIMEOUT=10000 composer require symfony/sendgrid-mailer:* && \
    rm -rf /opt/mautic/var/cache/js && \
    find /opt/mautic/node_modules -mindepth 1 -maxdepth 1 -not \( -name 'jquery' -or -name 'vimeo-froogaloop2' \) | xargs rm -rf && \
    php bin/console cache:clear && \
    php bin/console cache:warmup

After launching the newly built container, I'm able to set the email scheme to sendgrid+api without errors.

I've now requested the team to test their findings. I'll update this issue as soon as I know more but I suspect this will be the fix to our issue.

Kind regards, Insannik

[cibero42]

Hey @Insannik ,

I'm sorry, but I ended up not having time last week to focus on this.

I've now requested the team to test their findings. I'll update this issue as soon as I know more but I suspect this will be the fix to our issue.

Did they validate it?

[Insannik]

Hey @cibero42,

I haven't heard back from them yet but from testing myself I can only conclude that it seems to be working. I'll update here if I hear anything but it might take a while because of the summer holidays and all that.

[andres-asm]

This is what I did for the same usecase: I created my own dockerfile referencing the official one

FROM mautic/mautic:6-fpm
WORKDIR /var/www/html/
# Add the SendGrid API transport; ignore platform reqs and skip scripts (no npm needed)
RUN sh -c ls -la
RUN cat composer.json
RUN COMPOSER_ALLOW_SUPERUSER=1 composer require symfony/sendgrid-mailer --no-interaction --no-progress --ignore-platform-reqs --no-scripts
WORKDIR /var/www/html
RUN php bin/console cache:clear

And I also added environment variables to reference the DSN, this is required because otherwise the worker and/or cron containers will just use defaults and not the DSN from the config file

MAUTIC_MAILER_DSN=sendgrid+api://SG.----------------------------------@default

The I reference that in docker-compose.yml

x-mautic-volumes: &mautic-volumes
  - /srv/${COMPOSE_PROJECT_NAME}/config:/var/www/html/config:z
  - /srv/${COMPOSE_PROJECT_NAME}/logs:/var/www/html/var/logs:z
  - /srv/${COMPOSE_PROJECT_NAME}/media/files:/var/www/html/docroot/media/files:z
  - /srv/${COMPOSE_PROJECT_NAME}/media/images:/var/www/html/docroot/media/images:z
  #  - ./cron:/opt/mautic/cron:z # Uncomment this line to customize your cron configuration
  - mautic-docroot:/var/www/html/docroot:z
services:
  db:
    image: mysql:lts
    env_file:
      - stack.env
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
    volumes:
      - mysql-data:/var/lib/mysql
    healthcheck:
      test: mysqladmin --user=$$MYSQL_USER --password=$$MYSQL_PASSWORD ping
      start_period: 5s
      interval: 5s
      timeout: 5s
      retries: 10
    networks:
      - default
  nginx:
    image: nginx
    volumes:
      - /srv/${COMPOSE_PROJECT_NAME}/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
      - mautic-docroot:/var/www/html/docroot:z
      - /srv/${COMPOSE_PROJECT_NAME}/media/files:/var/www/html/docroot/media/files:z
      - /srv/${COMPOSE_PROJECT_NAME}/media/images:/var/www/html/docroot/media/images:z
    depends_on:
      - mautic_web
    ports:
      - 8080:80
    networks:
      default:
      proxy:
        aliases:
          - ${COMPOSE_PROJECT_NAME}
  mautic_web:
    image: andresasm/mautic:6-fpm-sendgrid
    links:
      - db:mysql
    volumes: *mautic-volumes
    environment:
      - DOCKER_MAUTIC_LOAD_TEST_DATA=${DOCKER_MAUTIC_LOAD_TEST_DATA}
      - DOCKER_MAUTIC_RUN_MIGRATIONS=${DOCKER_MAUTIC_RUN_MIGRATIONS}
    env_file:
      - stack.env
    healthcheck:
      test: cgi-fcgi -bind -connect 127.0.0.1:9000
      start_period: 5s
      interval: 5s
      timeout: 5s
      retries: 100
    depends_on:
      db:
        condition: service_healthy
    networks:
      - default
  mautic_cron:
    image: andresasm/mautic:6-fpm-sendgrid
    links:
      - db:mysql
    volumes: *mautic-volumes
    environment:
      - DOCKER_MAUTIC_ROLE=mautic_cron
    env_file:
      - stack.env
    depends_on:
      mautic_web:
        condition: service_healthy
    networks:
      - default
  mautic_worker:
    image: andresasm/mautic:6-fpm-sendgrid
    links:
      - db:mysql
    volumes: *mautic-volumes
    environment:
      - DOCKER_MAUTIC_ROLE=mautic_worker
    env_file:
      - stack.env
    depends_on:
      mautic_web:
        condition: service_healthy
    networks:
      - default
volumes:
  mysql-data:
  mautic-docroot:
networks:
  default:
    name: ${COMPOSE_PROJECT_NAME}-docker
  proxy:
    external: true

[Insannik]

Hi again,

I just heard back from the team. Everything seems to work as expected.

Thanks to @andres-asm, this issue now contains 2 valid solutions to the problem in case anyone is looking for this in the future.

As far as I'm concerned, this issue can be closed.

Thanks!

[martinounap]

This is what I did for the same usecase: I created my own dockerfile referencing the official one

FROM mautic/mautic:6-fpm
WORKDIR /var/www/html/
# Add the SendGrid API transport; ignore platform reqs and skip scripts (no npm needed)
RUN sh -c ls -la
RUN cat composer.json
RUN COMPOSER_ALLOW_SUPERUSER=1 composer require symfony/sendgrid-mailer --no-interaction --no-progress --ignore-platform-reqs --no-scripts
WORKDIR /var/www/html
RUN php bin/console cache:clear

I needed to add to the end of the dockerfile: RUN chown -R www-data:www-data /var/www/html/var/

Otherwise some folders would have wrong owner:

root@1578f28fa843:/var/www/html/var# ls -lah cache/
total 16K
drwxr-xr-x. 1 www-data www-data 4.0K Aug 22 17:55 .
drwxr-xr-x. 1 www-data www-data 4.0K Aug 18 14:31 ..
drwxr-xr-x. 1 www-data www-data 4.0K Aug 18 14:31 js
drwxr-xr-x. 1 root     root     4.0K Aug 22 17:55 prod

Error would look like something like this:

mautic_worker-1  | In Filesystem.php line 320:
mautic_worker-1  |                                                                                
mautic_worker-1  |   Cannot rename "/tmp/catalogue.en_GB.XxfB26D.phpg4nSor" to "/var/www/html/va  
mautic_worker-1  |   r/cache/prod/translations/catalogue.en_GB.XxfB26D.php": rename(/tmp/catalog  
mautic_worker-1  |   ue.en_GB.XxfB26D.phpg4nSor,/var/www/html/var/cache/prod/translations/catalo  
mautic_worker-1  |   gue.en_GB.XxfB26D.php): Permission denied

After adding, everything with SendGrid API works! ;)

[andres-asm]

Thank you I'll fix that on my end too

[martinounap]

If someone requires it, I have published an Apache version 6 Docker image on Docker Hub. This image should be used solely for testing and proof of concept purposes. I cannot guarantee that I will update it whenever a new version becomes available. If I have the time, I intend to create a GitHub public repository and automate the update process using actions.

Additionally, I hope we can find a way to publish a container under the Mautic repository, as SendGrid is a widely used service.

Docker Hub: martinounap/mautic:6-apache-sendgrid (https://hub.docker.com/r/martinounap/mautic)

[cibero42]

If someone requires it, I have published an Apache version 6 Docker image on Docker Hub. This image should be used solely for testing and proof of concept purposes. I cannot guarantee that I will update it whenever a new version becomes available. If I have the time, I intend to create a GitHub public repository and automate the update process using actions.

Additionally, I hope we can find a way to publish a container under the Mautic repository, as SendGrid is a widely used service.

Docker Hub: martinounap/mautic:6-apache-sendgrid (https://hub.docker.com/r/martinounap/mautic)

Sorry for the big big delay

Thanks for your contribution! Perhaps we could create a Docker compose that automatically builds this image based on the official base container. What do you all think?

[martinounap]

I updated image to 6.0.6. Enjoy. I did not manage to test it myself yet, I will do it soon. But if there is any kind of problem, please ping me!

[martinounap]

I’ve created a GitHub repository: https://github.com/martinounap/mautic-apache-sendgrid.

A scheduled job runs daily at 04:00 UTC to check whether the mautic-6 Docker Hub tag has changed. If the digest is different, it kicks off a multi-architecture build and updates the image here: https://hub.docker.com/r/martinounap/mautic.

Until SendGrid is included in the official Mautic Docker Hub container, this image can be used as an alternative. If you notice anything that should be adjusted, please let me know.

@cibero42 Is this what you had in mind with your last comment?

[martinounap]

I updated Docker file on my repo. I needed to change back WORKDIR, otherwise fresh deploy was not possible. So I added to the end of the file (https://github.com/martinounap/mautic-apache-sendgrid/blob/ab25c2493a1a696d41e98f6a60a14d8f82d5e1ed/Dockerfile#L9C1-L9C30) WORKDIR /var/www/html/docroot

[cibero42]

Hello,

My bad for not answering you back.

@cibero42 Is this what you had in mind with your last comment?

I mean something similar to what you did on this repo.

If you would like to contribute to main, I'd advise that we integrate your solution as a Docker Compose such as this one. This way, we can keep the base image smaller and avoid augmenting the attack surface for those who don't use it.

What do you think?

[martinounap]

I would love to do it. I will start reading your contribution guide and if I can handle it, I will do fork and pull request. But it takes little time from my side.