Docker-Compose Redis check failing with message "Authentication required."

[AttilaTheFun]

trafficstars

I'm trying to integrate Redis with Datadog in my docker compose application.

Per your instructions, I am using this compose file for my Datadog instance:

version: "3"
services:
  datadog:
    container_name: datadog
    build:
      context: [directory where Dockerfile is and my conf.d folder is stored]
      dockerfile: ./Dockerfile
      args:
        - DD_API_KEY
        - DD_SITE
    restart: on-failure
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc/:/host/proc/:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
    links:
     - redis

And this is the Dockerfile for my Datadog service:

FROM datadog/agent:latest

# Resolve the runtime arguments:
ARG DD_API_KEY
ARG DD_SITE

# Set the environment variables:
ENV DD_API_KEY ${DD_API_KEY}
ENV DD_SITE ${DD_SITE}

ADD conf.d/redisdb.d/conf.yaml /etc/datadog-agent/conf.d/redisdb.d/conf.yaml
ADD conf.d/redisdb.d/conf.yaml /etc/datadog-agent/conf.d/redisdb.yaml

(I have seen conflicting information about whether the configuration file should be in conf.d/redisdb.yaml or conf.d/redisdb.d/conf.yaml. I tried both but neither worked.)

My Redis conf.yaml is:

init_config:

instances:
  - host: redis
    port: 6379
    password: [password to my Redis instance]

I have verified that the Redis instance is working and I can connect with that password. I have also ssh'd into the container and verified that the yaml files copied over correctly.

However when I pull the logs from the Datadog container I see this:

2021-01-09 01:14:19 UTC | CORE | INFO | (cmd/agent/app/run.go:218 in StartAgent) | Starting Datadog Agent v7.24.1
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/util/log/log.go:526 in func1) | Collector docker successfully detected
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/util/log/log.go:526 in func1) | Using collector docker
2021-01-09 01:14:19 UTC | PROCESS | WARN | (pkg/util/log/log.go:541 in func1) | failed to get configuration value for key "process_agent_config.host_ips": unable to cast <nil> of type <nil> to []string
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/util/log/log.go:526 in func1) | could not deduce host IP from source config: no hostIPs were configured
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/util/log/log.go:526 in func1) | could not deduce host IP from source ec2 metadata endpoint: unable to fetch EC2 API, status code 404 trying to fetch http://169.254.169.254/latest/meta-data/local-ipv4
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/util/log/log.go:526 in func1) | overriding API key from env DD_API_KEY value
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/process/config/config.go:297 in mergeConfigIfExists) | no config exists at /etc/datadog-agent/system-probe.yaml, ignoring...
2021-01-09 01:14:19 UTC | PROCESS | INFO | (pkg/process/config/config.go:456 in loadEnvVariables) | overriding API key from env DD_API_KEY value
2021-01-09 01:14:19 UTC | SYS-PROBE | INFO | (pkg/util/log/log.go:526 in func1) | no config exists at /etc/datadog-agent/system-probe.yaml, ignoring...
2021-01-09 01:14:19 UTC | SYS-PROBE | INFO | (pkg/util/log/log.go:526 in func1) | overriding API key from env DD_API_KEY value
2021-01-09 01:14:19 UTC | SYS-PROBE | INFO | (cmd/system-probe/main.go:84 in runAgent) | system probe not enabled. exiting.
2021-01-09 01:14:19 UTC | SECURITY | INFO | (app/app.go:168 in start) | All security-agent components are deactivated, exiting
2021-01-09 01:14:20 UTC | TRACE | INFO | (pkg/util/log/log.go:526 in func1) | Loaded configuration: /etc/datadog-agent/datadog.yaml
2021-01-09 01:14:20 UTC | TRACE | INFO | (pkg/tagger/tagger.go:156 in tryCollectors) | docker tag collector successfully started
2021-01-09 01:14:20 UTC | CORE | INFO | (cmd/agent/app/run.go:262 in StartAgent) | Hostname is: production
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/api/security/security.go:145 in fetchAuthToken) | Saved a new authentication token to /etc/datadog-agent/auth_token
2021-01-09 01:14:20 UTC | CORE | INFO | (cmd/agent/app/run.go:289 in StartAgent) | GUI server port -1 specified: not starting the GUI.
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/forwarder/forwarder.go:266 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://7-24-1-app.agent.datadoghq.com" (1 api key(s))
2021-01-09 01:14:20 UTC | CORE | INFO | (cmd/agent/app/run.go:343 in StartAgent) | logs-agent disabled
2021-01-09 01:14:20 UTC | CORE | INFO | (cmd/agent/app/run.go:347 in StartAgent) | System probe config not found, disabling pulling system probe info in the status page: open /etc/datadog-agent/system-probe.yaml: no such file or directory
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/util/version_history.go:43 in logVersionHistoryToFile) | Cannot read file: /opt/datadog-agent/run/version-history.json, will create a new one. open /opt/datadog-agent/run/version-history.json: no such file or directory
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/tagger/tagger.go:156 in tryCollectors) | docker tag collector successfully started
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/dogstatsd/listeners/udp.go:97 in Listen) | dogstatsd-udp: starting to listen on 127.0.0.1:8125
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/collector/runner/runner.go:92 in NewRunner) | Runner started with 4 workers.
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/collector/python/init.go:311 in Initialize) | Initializing rtloader with python3 /opt/datadog-agent/embedded
2021-01-09 01:14:20 UTC | PROCESS | INFO | (main_common.go:108 in runAgent) | running on platform: linux-4.4.0-169-generic-x86_64-with-glibc2.2.5
2021-01-09 01:14:20 UTC | PROCESS | INFO | (main_common.go:111 in runAgent) | running version: Version: 7.24.1, Git hash: cf2a7e6, Git branch: HEAD, Build date: 2020-12-11T15:10:53, Go Version: go version go1.14.7 linux/amd64, 
2021-01-09 01:14:20 UTC | PROCESS | INFO | (pkg/tagger/tagger.go:156 in tryCollectors) | docker tag collector successfully started
2021-01-09 01:14:20 UTC | CORE | INFO | (pkg/util/cloudprovider.go:54 in DetectCloudProvider) | No cloud provider detected
2021-01-09 01:14:21 UTC | TRACE | INFO | (pkg/trace/agent/run.go:131 in Run) | Trace agent running on host production
2021-01-09 01:14:21 UTC | TRACE | INFO | (pkg/trace/api/api.go:125 in Start) | Listening for traces at http://0.0.0.0:8126

...

2021-01-09 01:14:24 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | check:redisdb | Running check
2021-01-09 01:14:24 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | check:ntp | Running check
2021-01-09 01:14:24 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check redisdb: [{"message": "Authentication required.", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 863, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/redisdb/redisdb.py\", line 500, in check\n    self._check_db()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/redisdb/redisdb.py\", line 194, in _check_db\n    info = conn.info()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/client.py\", line 1304, in info\n    return self.execute_command('INFO')\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/client.py\", line 901, in execute_command\n    return self.parse_response(conn, command_name, **options)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/client.py\", line 915, in parse_response\n    response = connection.read_response()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 730, in read_response\n    response = self._parser.read_response()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 337, in read_response\n    raise error\nredis.exceptions.AuthenticationError: Authentication required.\n"}]
2021-01-09 01:14:24 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | check:redisdb | Done running check

I'm not really sure where I can go from here? Are my services configured correctly? Is there anything else I can try?

[ChristineTChen]

Hey @AttilaTheFun ,

(I have seen conflicting information about whether the configuration file should be in conf.d/redisdb.yaml or conf.d/redisdb.d/conf.yaml. I tried both but neither worked.)

Either of these will work with the agent. As long as the config directory or the config file is named the same as the integration (redisdb in this case). It looks like the config file is fine since the check is being run. I believe the main issue is authentication.

The error is occurring when the redis integration attempts to try connecting to your instance: https://github.com/DataDog/integrations-core/blob/6e5779072c333bcacaa47575810f8b8a6967212b/redisdb/datadog_checks/redisdb/redisdb.py#L194

Does your redis instance require a username by any chance?

[AttilaTheFun]

It shouldn't - I'm using Redis 5.0. Since I posted this I managed to get the Autodiscovery approach working with this docker-compose file:

version: "3"
services:
  datadog:
    container_name: datadog
    build:
      context: ~/Developer/Go/github.com/attilathefun/datadog
      dockerfile: ./Dockerfile
      args:
        - DD_API_KEY
        - DD_SITE
        - REDIS_PASSWORD
        - DD_MYSQL_PASSWORD
    restart: on-failure
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc/:/host/proc/:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
    links:
     - redis

And the Redis file:

version: "3"
services:
  redis:
    container_name: redis
    build:
      context: ~/Developer/Go/github.com/attilathefun/redis
      dockerfile: ./Dockerfile
      args:
        - REDIS_PASSWORD
    restart: on-failure
    volumes:
      - redis:/var/lib/redis
    ports:
      - "6379:6379"
    labels:
      com.datadoghq.ad.check_names: '["redisdb"]'
      com.datadoghq.ad.init_configs: '[{}]'
      com.datadoghq.ad.instances: '[{"host": "%%hostname%%","port":"6379","password":"%%env_REDIS_PASSWORD%%"}]'
volumes:
  redis:

I'm not sure why the manual config wasn't working though :/

[ChristineTChen]

With the manual approach, could you verify that there is a conf.yaml for redis in the /etc/datadog-agent/conf.d path of the container?

The recommended method to include a config is to mount the directory: https://docs.datadoghq.com/agent/docker/integrations/?tab=file The main difference between using ADD vs VOLUME would be that volume is accessible at run-time.

[dziedrius]

I'm facing the same issue, even though I'm using volume to attach the config file and I'm sure that config file is seen by the datadog redis checker - as I've changed port in redis config file to 8888 and see in datadog agent logs that it tries to use that port and fails. I've started with latest redis image, then downgraded to 5.0.10, that did not helped.

Docker compose file:

version: '3.4'

services:
  redis:
    container_name: redis_cache2
    image: 'bitnami/redis:5.0.10'
    environment:
      - REDIS_PASSWORD=Qwerty@123
    ports:
      - "6379:6379"

  datadog-agent:
    container_name: datadog-agent2
    image: gcr.io/datadoghq/agent:latest
    environment:
      - DD_API_KEY=mykey
      - DD_ENV=local
      - DD_SITE=datadoghq.com
      - DD_TAGS=datadog.agent
      - DD_APM_ENABLED=true
      - DD_LOGS_ENABLED=true
      - DD_PROCESS_AGENT_ENABLED=true
      - DD_APM_NON_LOCAL_TRAFFIC=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_CONTAINER_EXCLUDE=true

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
      - ./conf.d:/conf.d

    links:
     - redis

Redis config file, sitting in relative path to docker compose file ./conf.d/redisdb.d/conf.yaml

init_config:

instances:
  - host: redis
    port: 6379
    password: Qwerty@123

[julian-perge]

+1 to this, as I'm running into the exact same issue using helm. I'm putting the output of the redis helm chart user/pass into the datadog helm file via terraform, but no matter what configuration I try, it either says Authentication Required or invalid username-password pair or user is disabled.

I've tried setting the username to default, the output from the redis helm chart, and even root. None of which work, and I have no idea why.

My other services who use the redis service for access have zero issues connecting within my Kubernetes cluster, for context. I've even put Datadog in the same namespace as my redis service no dice.