Agent installation fails on pulling certs

[rockaut]

We already have a case (Nr. 1022982) open for this one.

We have a lot of servers which aren't allow to have direct access to "the interwebs". For package installations we have intermediate repositories to work around for this which works great in combination with the role. But not for the certificates. Currently the role always tries to pull the certs from Amazon S3 (or keys.datadog.com which is still S3) - and it's just not viable to whitelist those addresses!

We thought that setting the variables:

datadog_yum_repo_gpgcheck: false
datadog_yum_gpgcheck: false

would help but the just the the yum/dnf repo options to false. The role will still try to fetch the certs which in turn then fails and the whole role/installation fails.

So it would be great if the role might skip the cert-fetch maybe with a new variable or respecting the already present variables.

Or even better: if the role would not only accept URLs from where to pull the certs but instead:

• pull the certs only on the internet allowed ansible controller ...
• copying the certs from the controller to the hosts and using those then.

Edit: the problem for us as we currently use RHEL lies in the four tasks starting from (https://github.com/DataDog/ansible-datadog/blob/3bb392d16558f7249c61163cce21a9fd14bf2677/tasks/pkg-redhat.yml#L28).

[rockaut]

So instead of get_url on each host we think it would be a nice feature to be able to delegate_to: localhost those tasks and then ansible.builtin.copy: dest=/tmp/DATADOG_RPM_KEY_CURRENT.public for example.

This would be a relatively quick and also backward compatible way, easily manageable with a new variable or even with an fallback block scenario.

Edit: for example our modified pkg-redhat.yml TBC, this has problems as the repo still add the URL to the file. It's just our current workaround.

---
- name: Fail early if Python 3 is used on CentOS / RHEL < 8 with old Ansible
  fail:
    msg: "The installation of the Agent on RedHat family systems using yum is not compatible with Python 3 with older Ansible versions.
          To run this role, use a Python 2 interpreter on hosts running CentOS / RHEL < 8 or upgrade Ansible to version 2.11+"
  when: (not datadog_ignore_old_centos_python3_error)
        and (ansible_version.full is version("2.11", operator="lt", strict=True))
        and (ansible_pkg_mgr == "yum")
        and (ansible_facts.python.version.major | int >= 3)

- name: Find out whether to set repo_gpgcheck or not
  # We turn off repo_gpgcheck on custom repos and on RHEL/CentOS 8.1 because
  # of https://bugzilla.redhat.com/show_bug.cgi?id=1792506
  set_fact:
    do_yum_repo_gpgcheck: >-
      {{ datadog_yum_repo_gpgcheck if datadog_yum_repo_gpgcheck != '' else (
        'no' if (
          ansible_facts.distribution_version.startswith('8.1.') or ansible_facts.distribution_version == '8.1' or
          datadog_yum_repo != ''
        ) else 'yes'
      ) }}

- name: Pull the certs on remote host
  block:
    - name: Download current RPM key
      get_url:
        url: "{{ datadog_yum_gpgkey_current }}"
        dest: /tmp/DATADOG_RPM_KEY_CURRENT.public
        force: yes

    - name: Download new RPM key (Expires in 2022)
      get_url:
        url: "{{ datadog_yum_gpgkey_e09422b3 }}"
        dest: /tmp/DATADOG_RPM_KEY_E09422B3.public
        checksum: "sha256:{{ datadog_yum_gpgkey_e09422b3_sha256sum }}"

    - name: Download new RPM key (Expires in 2024)
      get_url:
        url: "{{ datadog_yum_gpgkey_20200908 }}"
        dest: /tmp/DATADOG_RPM_KEY_20200908.public
        checksum: "sha256:{{ datadog_yum_gpgkey_20200908_sha256sum }}"
  when: datadog_certs_remote_pull | default(true) and not ansible_check_mode

- name: Pull the certs on localhost an push to remote
  block:
    - name: Download current RPM key
      delegate_to: "datadog_certs_delegate | default(localhost)"
      run_once: true
      get_url:
        url: "{{ datadog_yum_gpgkey_current }}"
        dest: /tmp/DATADOG_RPM_KEY_CURRENT.public
        force: yes

    - ansible.builtin.copy:
        dest: /tmp/DATADOG_RPM_KEY_CURRENT.public
        src: /tmp/DATADOG_RPM_KEY_CURRENT.public

    - name: Download new RPM key (Expires in 2022)
      delegate_to: "datadog_certs_delegate | default(localhost)"
      run_once: true
      get_url:
        url: "{{ datadog_yum_gpgkey_e09422b3 }}"
        dest: /tmp/DATADOG_RPM_KEY_E09422B3.public
        checksum: "sha256:{{ datadog_yum_gpgkey_e09422b3_sha256sum }}"

    - ansible.builtin.copy:
        dest: /tmp/DATADOG_RPM_KEY_E09422B3.public
        src: /tmp/DATADOG_RPM_KEY_E09422B3.public

    - name: Download new RPM key (Expires in 2024)
      delegate_to: "datadog_certs_delegate | default(localhost)"
      run_once: true
      get_url:
        url: "{{ datadog_yum_gpgkey_20200908 }}"
        dest: /tmp/DATADOG_RPM_KEY_20200908.public
        checksum: "sha256:{{ datadog_yum_gpgkey_20200908_sha256sum }}"

    - ansible.builtin.copy:
        dest: /tmp/DATADOG_RPM_KEY_20200908.public
        src: /tmp/DATADOG_RPM_KEY_E09422B3.public

  when: not datadog_certs_remote_pull | default(false) and not ansible_check_mode

- name: Import new RPM key (Expires in 2024)
  rpm_key:
    key: /tmp/DATADOG_RPM_KEY_20200908.public
    state: present
  when: not ansible_check_mode and do_yum_repo_gpgcheck # modified

- name: Import current RPM key
  rpm_key:
    key: /tmp/DATADOG_RPM_KEY_CURRENT.public
    state: present
  when: not ansible_check_mode and do_yum_repo_gpgcheck # modified

- name: Import new RPM key (Expires in 2022)
  rpm_key:
    key: /tmp/DATADOG_RPM_KEY_E09422B3.public
    state: present
  when: not ansible_check_mode and do_yum_repo_gpgcheck # modified

- name: Install Datadog Agent 5 yum repo
  yum_repository:
    name: datadog
    description: Datadog, Inc.
    baseurl: "{{ datadog_agent5_yum_repo }}"
    enabled: yes
    repo_gpgcheck: no  # we don't sign Agent 5 repodata
    gpgcheck: "{{ datadog_yum_gpgcheck }}"
    gpgkey: [
      "{{ datadog_yum_gpgkey_current }}",
      "{{ datadog_yum_gpgkey_20200908 }}",
      "{{ datadog_yum_gpgkey_e09422b3 }}",
      "{{ datadog_yum_gpgkey }}",
    ]
  register: repofile5
  when: (datadog_agent_major_version|int == 5) and (datadog_yum_repo | length == 0) and (not ansible_check_mode)

- name: Install Datadog Agent 6 yum repo
  yum_repository:
    name: datadog
    description: Datadog, Inc.
    baseurl: "{{ datadog_agent6_yum_repo }}"
    enabled: yes
    repo_gpgcheck: "{{ do_yum_repo_gpgcheck }}"
    gpgcheck: "{{ datadog_yum_gpgcheck }}"
    gpgkey: [
      "{{ datadog_yum_gpgkey_current }}",
      "{{ datadog_yum_gpgkey_20200908 }}",
      "{{ datadog_yum_gpgkey_e09422b3 }}",
      "{{ datadog_yum_gpgkey }}",
    ]
  register: repofile6
  when: (datadog_agent_major_version|int == 6) and (datadog_yum_repo | length == 0) and (not ansible_check_mode)

- name: Install Datadog Agent 7 yum repo
  yum_repository:
    name: datadog
    description: Datadog, Inc.
    baseurl: "{{ datadog_agent7_yum_repo }}"
    enabled: yes
    repo_gpgcheck: "{{ do_yum_repo_gpgcheck }}"
    gpgcheck: "{{ datadog_yum_gpgcheck }}"
    gpgkey: [
      "{{ datadog_yum_gpgkey_current }}",
      "{{ datadog_yum_gpgkey_20200908 }}",
      "{{ datadog_yum_gpgkey_e09422b3 }}",
    ]
  register: repofile7
  when: (datadog_agent_major_version|int == 7) and (datadog_yum_repo | length == 0) and (not ansible_check_mode)

- name: Install Datadog Custom yum repo
  yum_repository:
    name: datadog
    description: Datadog, Inc.
    baseurl: "{{ datadog_yum_repo }}"
    enabled: yes
    repo_gpgcheck: "{{ do_yum_repo_gpgcheck }}"
    gpgcheck: "{{ datadog_yum_gpgcheck }}"
    gpgkey: [
      "{{ datadog_yum_gpgkey_current }}",
      "{{ datadog_yum_gpgkey_20200908 }}",
      "{{ datadog_yum_gpgkey_e09422b3 }}",
      "{{ datadog_yum_gpgkey }}",
    ]
  register: repofilecustom
  when: (datadog_yum_repo | length > 0) and (not ansible_check_mode)

- name: Clean repo metadata if repo changed # noqa 503
  command: yum clean metadata --disablerepo="*" --enablerepo=datadog
  failed_when: false # Cleaning the metadata is only needed when downgrading a major version of the Agent, don't fail because of this
  args:
    warn: no
  when: repofile5.changed or repofile6.changed or repofile7.changed or repofilecustom.changed

- name: Remove old yum repo files
  yum_repository:
    name: "ansible_datadog_{{ item }}"
    state: absent
  with_items: [ 5, 6, 7, "custom" ]

- include_tasks: pkg-redhat/install-pinned.yml
  when: datadog_agent_redhat_version is defined

- include_tasks: pkg-redhat/install-latest.yml
  when: datadog_agent_redhat_version is not defined

[bkabrda]

:wave: hi, so this is an interesting problem and I understand that you need to solve it. I've actually recently merged #475 which is technically going against the solution you propose here. I'll take a closer look at what we might be able to do and post a proposal here.

[bkabrda]

After taking a closer look, we can definitely still implement this even after merging #475 - we can just add a block (like the one that you showed in your example) that will fetch the keys with the condition when: not datadog_certs_remote_pull and overwrite datadog_yum_gpgkey_* with the local path, so the rpm_key will import directly from URL when running "online" and it will import from the uploaded files when running "offline".

Just a small question here to make sure I properly understand - IIUC you have intermediary repositories set up somewhere in your infrastructure that these "offline" servers can access for package downloads. Would it be possible for you to create a simple intermediary storage where you would also put the keys and override the datadog_yum_gpgkey_* variables to point there?

I'm happy to implement the functionality if the answer to the above question is "we can't do that", I'm just curious if you've considered it. Thanks!

[rockaut]

Heja @bkabrda,

sorry for late response. Well with #475 we could implement tasks to fetch and distribute the keys prior to including the role and then set the variable to those paths. I guess that would work.

[alopezz]

@rockaut Did you find a way to make it work for you? Can we close this?

EDIT: Renamed the issue to give it a possibly more representative name.

[rockaut]

Just a quick answer because I'm on vacation currently :D

Yes this can be closed. Sorry, I wasn't aware it's still open. In fact this was solved some releases of the role ago with another change.

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Alex Lopez @.> Sent: Monday, July 22, 2024 12:04:50 PM To: DataDog/ansible-datadog @.> Cc: Markus Fischbacher @.>; Mention @.> Subject: Re: [DataDog/ansible-datadog] Agent installation fails on pulling certs (Issue #461)

@rockauthttps://github.com/rockaut Did you find a way to make it work for you? Can we close this?

— Reply to this email directly, view it on GitHubhttps://github.com/DataDog/ansible-datadog/issues/461#issuecomment-2242577108, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AADOV6DDXIUOEMOOGNPDBLDZNTKMFAVCNFSM6AAAAABLH7LFECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGU3TOMJQHA. You are receiving this because you were mentioned.Message ID: @.***>

[alopezz]

Thanks for the update! I'll close it, then.