Getting `[Errno 111] Connection refused` whenever trying to send out emails

[hwittenborn]

Self-Hosted Version

22.8.0

CPU Architecture

x86_64

Docker Version

20.10.12

Docker Compose Version

2.10.2

Steps to Reproduce

Here's my config files (values in {{ env.* }} are replaced when deployed to my server):

config.yml

mail.backend: 'smtp'
mail.host: 'mailcow.hunterwittenborn.com'
mail.port: 465
mail.username: '[email protected]'
mail.password: '{{ env.sentry_mail_password }}'
mail.use-tls: false
mail.use-ssl: true
mail.enable-replies: true
mail.from: 'Sentry - Makedeb <[email protected]>'

# When email-replies are enabled, this value is used in the Reply-To header
mail.reply-hostname: '[email protected]'

###################
# System Settings #
###################

# If this file ever becomes compromised, it's important to generate a new key.
# Changing this value will result in all current sessions being invalidated.
# A new key can be generated with `$ sentry config generate-secret-key`
system.secret-key: '{{ env.sentry_secret_key }}'

################
# File storage #
################

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'
dsym.cache-path: '/data/dsym-cache'
releasefile.cache-path: '/data/releasefile-cache'

system.url-prefix: 'https://sentry.makedeb.org'
system.internal-url-prefix: 'http://web:9000'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

transaction-events.force-disable-internal-project: true

######################
# GitHub Integration #
######################

github-login.extended-permissions: ['repo']
github-app.id: {{ env.sentry_github_app_id }}
github-app.name: 'Sentry - makedeb'
github-app.webhook-secret: '{{ env.sentry_github_app_webhook_secret }}'
github-app.client-id: '{{ env.sentry_github_app_client_id }}'
github-app.client-secret: '{{ env.sentry_github_app_client_secret }}'
github-app.private-key: |
{{ env.sentry_github_app_private_key }}

sentry.conf.py

from sentry.conf.server import *  # NOQA


def get_internal_network():
    import ctypes
    import fcntl
    import math
    import socket
    import struct

    iface = b"eth0"
    sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifreq = struct.pack(b"16sH14s", iface, socket.AF_INET, b"\x00" * 14)

    try:
        ip = struct.unpack(
            b"!I", struct.unpack(b"16sH2x4s8x", fcntl.ioctl(sockfd, 0x8915, ifreq))[2]
        )[0]
        netmask = socket.ntohl(
            struct.unpack(b"16sH2xI8x", fcntl.ioctl(sockfd, 0x891B, ifreq))[2]
        )
    except IOError:
        return ()
    base = socket.inet_ntoa(struct.pack(b"!I", ip & netmask))
    netmask_bits = 32 - int(round(math.log(ctypes.c_uint32(~netmask).value + 1, 2), 1))
    return "{0:s}/{1:d}".format(base, netmask_bits)


INTERNAL_SYSTEM_IPS = (get_internal_network(),)


DATABASES = {
    "default": {
        "ENGINE": "sentry.db.postgres",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "",
        "HOST": "postgres",
        "PORT": "",
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = False

SENTRY_OPTIONS["system.event-retention-days"] = int(
    env("SENTRY_EVENT_RETENTION_DAYS", "90")
)

#########
# Redis #
#########

# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB

SENTRY_OPTIONS["redis.clusters"] = {
    "default": {
        "hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
    }
}

#########
# Queue #
#########

# See https://develop.sentry.dev/services/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

rabbitmq_host = None
if rabbitmq_host:
    BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
        username="guest", password="guest", host=rabbitmq_host, vhost="/"
    )
else:
    BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
        **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
    )


#########
# Cache #
#########

# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
        "LOCATION": ["memcached:11211"],
        "TIMEOUT": 3600,
    }
}

# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"

DEFAULT_KAFKA_OPTIONS = {
    "bootstrap.servers": "kafka:9092",
    "message.max.bytes": 50000000,
    "socket.timeout.ms": 1000,
}

SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}

KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS

###############
# Rate Limits #
###############

# Rate limits apply to notification handlers and are enforced per-project
# automatically.

SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"

##################
# Update Buffers #
##################

# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)

SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"

##########
# Quotas #
##########

# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.

SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"

########
# TSDB #
########

# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.

SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"

#########
# SNUBA #
#########

SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}

###########
# Digests #
###########

# The digest backend powers notification summaries.

SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed in order to prevent https://github.com/getsentry/sentry/blob/c6f9660e37fcd9c1bbda8ff4af1dcfd0442f5155/src/sentry/services/http.py#L70
    "uwsgi-socket": None,
    "so-keepalive": True,
    # Keep this between 15s-75s as that's what Relay supports
    "http-keepalive": 15,
    "http-chunked-input": True,
    # the number of web workers
    "workers": 3,
    "threads": 4,
    "memory-report": False,
    # Some stuff so uwsgi will cycle workers sensibly
    "max-requests": 100000,
    "max-requests-delta": 500,
    "max-worker-lifetime": 86400,
    # Duplicate options from sentry default just so we don't get
    # bit by sentry changing a default value that we depend on.
    "thunder-lock": True,
    "log-x-forwarded-for": False,
    "buffer-size": 32768,
    "limit-post": 209715200,
    "disable-logging": True,
    "reload-on-rss": 600,
    "ignore-sigpipe": True,
    "ignore-write-errors": True,
    "disable-write-exception": True,
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
#CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

# End of SSL/TLS settings

########
# Mail #
########

SENTRY_OPTIONS["mail.list-namespace"] = env('SENTRY_MAIL_HOST', 'localhost')
SENTRY_OPTIONS["mail.from"] = "[email protected]"

############
# Features #
############

SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            "organizations:discover",
            "organizations:events",
            "organizations:global-views",
            "organizations:incidents",
            "organizations:integrations-issue-basic",
            "organizations:integrations-issue-sync",
            "organizations:invite-members",
            "organizations:metric-alert-builder-aggregate",
            "organizations:sso-basic",
            "organizations:sso-rippling",
            "organizations:sso-saml2",
            "organizations:performance-view",
            "organizations:advanced-search",
            "projects:custom-inbound-filters",
            "projects:data-forwarding",
            "projects:discard-groups",
            "projects:plugins",
            "projects:rate-limits",
            "projects:servicehooks",
        )
    }
)

#######################
# MaxMind Integration #
#######################

GEOIP_PATH_MMDB = '/geoip/GeoLite2-City.mmdb'

######################
# GitHub Integration #
######################
GITHUB_APP_ID = "{{ env.sentry_github_app_id }}"
GITHUB_API_SECRET = "{{ env.sentry_github_app_client_secret }}"
GITHUB_REQUIRE_VERIFIED_EMAIL = True

.env

COMPOSE_PROJECT_NAME=sentry-self-hosted
SENTRY_EVENT_RETENTION_DAYS=90
# You can either use a port number or an IP:PORT combo for SENTRY_BIND
# See https://docs.docker.com/compose/compose-file/#ports for more
SENTRY_BIND=127.0.0.1:5080
# Set SENTRY_MAIL_HOST to a valid FQDN (host/domain name) to be able to send emails!
SENTRY_MAIL_HOST=mailcow.hunterwittenborn.com
SENTRY_IMAGE=getsentry/sentry:22.8.0
SNUBA_IMAGE=getsentry/snuba:22.8.0
RELAY_IMAGE=getsentry/relay:22.8.0
SYMBOLICATOR_IMAGE=getsentry/symbolicator:0.5.1
WAL2JSON_VERSION=latest
HEALTHCHECK_INTERVAL=30s
HEALTHCHECK_TIMEOUT=60s
HEALTHCHECK_RETRIES=5

And my entire setup is in front of an NGINX config:

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  
  server_name sentry.makedeb.org;

  location / {

    proxy_pass http://127.0.0.1:5080/;

    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    client_max_body_size 0;
  }
  
  ssl_certificate /etc/letsencrypt/live/homelab/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/homelab/privkey.pem;
}

Expected Result

Emails get sent fine

Actual Result

They don't, as seen in this screenshot:

The end of my server logs also show such (IPs and User Agents have been stripped and replaced with XXX.XX.XX.XXX):

06:25:23 [INFO] sentry.superuser: superuser.request (url='http://sentry.makedeb.org/api/0/internal/mail/' method='POST' ip_address='XXX.XX.XX.XXX' user_id=1)
06:25:23 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.internal.mail.InternalMailEndpoint' response=500 user_id='1' is_app='None' token_type='None' is_frontend_request='True' organization_id='None' auth_id='None' path='/api/0/internal/mail/' caller_ip='XXX.XX.XX.XXX' user_agent='XXX.XX.XX.XXX' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.031336307525634766 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
06:25:23 [ERROR] django.request: Internal Server Error: /api/0/internal/mail/ (status_code=500 request=<WSGIRequest: POST '/api/0/internal/mail/'>)

[hwittenborn]

I also looked through the thread at #1312 but nothing mentioned there helped.

[smoehrle]

Try change your .env to:

# Was
SENTRY_MAIL_HOST=mailcow.hunterwittenborn.com
# Should be
# SENTRY_MAIL_HOST=example.com

I just had the same "Connection refused" error and fixed it by not setting this env variable. It's also mentioned here: https://github.com/getsentry/self-hosted/issues/1312#issuecomment-1084591311

[hwittenborn]

Try change your .env

That did the trick.

Looking at the linked issue though @chadwhitacre mentioned this:

Is there something we could change in the docs to make this clearer?

I think it probably could, the only purpose I really got for the variable from the docs is that it functioned the same as the mail.host config option - I'm still a bit lost on what it does, I just know now that commenting it out fixed the issue.

[hubertdeng123]

Are you running your instance on localhost? See this line in sentry.conf.py

SENTRY_OPTIONS["mail.list-namespace"] = env('SENTRY_MAIL_HOST', 'localhost')

Perhaps it would be useful to make this more clear in the docs and the differences between SENTRY_MAIL_HOST and mail.host

[hubertdeng123]

As per documentation here, SENTRY_MAIL_HOST is the FQDN (fully qualified domain name). I guess mail.host could be more clearly be indicated as the smtp address

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀