Prometheus custom rules not copying to rules folder

[cdestouni-fmp]

Setup

I upgraded to the latest release v0.23.0 and run the Prometheus role to configure Prometheus on a Ubuntu 22 virtual machine. My custom rules are located in prometheus/rules/*yml location and my files all end with the .yml extension as expected by the default variable prometheus_alert_rules_files.

Configuration

I 'm running the role using the below ansible configuration.

ansible [core 2.13.13]
  config file = /home/user/folder/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/.local/bin/ansible
  python version = 3.8.10 (default, Nov  7 2024, 13:10:47) [GCC 9.4.0]
  jinja version = 3.0.3
  libyaml = True

And the below OS configuration

Distributor ID:	Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal

Deploying to the below OS configuration

Distributor ID:	Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

Problem

The Copy custom alerting rule files did not copy over my custom rules to the rules directory on the Prometheus server. There was no feedback, even in verbose mode -vvvv so I assumed the glob working correctly and it thinks there's no files to copy over. After the role completed, the Prometheus server was running but no custom rules were displayed in the UI.

Testing done

• I tried to override the prometheus_alert_rules_files variable with a new location (In the expected list format). This resulted in an error of not locating the files instead of the original no feedback, which makes me think the location is not the problem.

• I tested at v0.19.0 before this task was last changed and it works fine and copied the custom rules over as expected.

• I tried the v0.23.0 task changes but using root permissions on the copy task which did not make a difference. This means it is not a permissions issue.

Conclusion

This leaves me to think the problem is situated in the change below.

Working version (v0.19.0)

with_fileglob: "{{ prometheus_alert_rules_files }}"

Not working version (v0.23.0)

loop: "{{ prometheus_alert_rules_files | map('ansible.builtin.fileglob') | flatten }}"

Please let me know if you can help resolve this.

[gardar]

Do you by any chance have prometheus_alert_rules_files configured as a string and not a list? See #454 for further explanation

[cdestouni-fmp]

I left the variable configured with the default list here. I also additionally tested an override where I had it configured as a list and it did not seem to make a difference in this case.

[gardar]

Hmm, what does your playbook look like? And can you show me the debug log of the task?

[cdestouni-fmp]

Playbook

Stored in playbooks -> prometheus -> prometheus.yml

- hosts: prometheus
  remote_user: root
  roles:
    - prometheus.prometheus.prometheus
  tags: [prometheus]
  vars:
    prometheus_version: "{{ prometheus.version }}"
    prometheus_storage_retention: "1w"
    prometheus_db_dir: /var/lib/prometheus
    prometheus_config_flags_extra:
      storage.tsdb.min-block-duration: 2h
      storage.tsdb.max-block-duration: 2h
      web.enable-admin-api:
      storage.tsdb.allow-overlapping-blocks:

    prometheus_alertmanager_config:
      - scheme: http
        static_configs:
          - targets: ["{{ alert_manager_url }}"]

    prometheus_alert_rules: []
    prometheus_alert_relabel_configs: []

    prometheus_external_labels:
      clusterName: "production"

    prometheus_targets:
      linux:
        - targets:
            - {{ exampleHostname }}:{{ port }}
            - {{ exampleHostname }}:{{ port }}

I have the prometheus rules setup like so... (In the new rule format) playbooks -> prometheus -> files -> rules -> custom-alert.yml

The debugging did not output a single thing unfortunately, I tried it with -vvvv and on that particular task there were no logs.

ansible-playbook  -i inventories/dev playbooks/prometheus/prometheus.yml -vvvv

TASK [Copy custom alerting rule files] *********************************************************************************************************************************************

TASK [Configure prometheus] *********************************************************************************************************************************************
changed: [prometheus.host]

[MajorP93]

Can confirm. Same issue for me after upgrading the collection to the latest version.

[MajorP93]

Found the cause: https://github.com/prometheus-community/ansible/pull/333/files

In this PR support for rule files with file ending .rules was removed.

This was not mentioned in changelog / release notes. The PR was mentioned under "minor changes". Breaking change was not mentioned.

[MajorP93]

Renaming my rules files from .rules to .yml alone did not fix my issue.

https://github.com/prometheus-community/ansible/commit/1e4e4c34156900d427a65430cd3eba805b441851#diff-a405b871a41e1b77e6d93c308fb136fd60b40efa6e85917adbde90a8fb9a0f16R93

This change is also problematic. After reverting this change (loop: "{{ prometheus_alert_rules_files | map('ansible.builtin.fileglob') | flatten }}" --> with_fileglob: "{{ prometheus_alert_rules_files }}") the prometheus role is working as expected.

In other words: I can confirm the conclusion of @cdestouni-fmp

[gardar]

@MajorP93 Any chance you could try to use  "{{ prometheus_alert_rules_files | map('ansible.builtin.fileglob') | flatten | list }}" and report back if that fixes the issue?

[lsimard]

I fixed the issue using an absolute path when configuring prometheus_alert_rules_files, like this:

prometheus_alert_rules_files:
  - "{{ playbook_dir }}/roles/prometheus/prometheus/rules/generic/*.yml"

The other thing I noticed is that doing {{ prometheus_alert_rules_files | map('ansible.builtin.fileglob') | flatten returns relative paths if you configure prometheus_alert_rules_files with relative paths, which is problematic with the use of ansible.builtin.copy. When using with_fileglob, the paths returned are always absolute.