remove_keys_from_values: new filter plugin, remove keys with certain values

[tanganellilore]

SUMMARY

Hi team, i will propose a special filter that i think is missing in this community, a filter that remove some keys on a dict (or list of dict) that contains certain values.

In origin I search a way to remove empty key in a dict and I can do it with select, match and so on, but i think that a filter that can remove key ina object absed on valuse can be very usefull.

This is why i'm poroposing a filter that can do it (as default behaviourremove keys with empty values), similar to remove_keysv filter.

ISSUE TYPE

• New Module/Plugin Pull Request

COMPONENT NAME

remove_keys_from_values.py

ADDITIONAL INFORMATION

[ansibullbot]

The test ansible-test sanity --test pep8 [explain] failed with 2 errors:

plugins/filter/remove_keys_from_values.py:172:1: W293: blank line contains whitespace
plugins/filter/remove_keys_from_values.py:208:1: E302: expected 2 blank lines, found 1

The test ansible-test sanity --test yamllint [explain] failed with 4 errors:

plugins/filter/remove_keys_from_values.py:44:100: error: DOCUMENTATION: syntax error: mapping values are not allowed here (syntax)
plugins/filter/remove_keys_from_values.py:44:100: unparsable-with-libyaml: None - mapping values are not allowed in this context
plugins/filter/remove_keys_from_values.py:60:3: error: EXAMPLES: syntax error: expected <block end>, but found '-' (syntax)
plugins/filter/remove_keys_from_values.py:60:3: unparsable-with-libyaml: while parsing a block mapping - did not find expected key

The test ansible-test sanity --test pep8 [explain] failed with 2 errors:

plugins/filter/remove_keys_from_values.py:172:1: W293: blank line contains whitespace
plugins/filter/remove_keys_from_values.py:208:1: E302: expected 2 blank lines, found 1

The test ansible-test sanity --test yamllint [explain] failed with 4 errors:

plugins/filter/remove_keys_from_values.py:44:100: error: DOCUMENTATION: syntax error: mapping values are not allowed here (syntax)
plugins/filter/remove_keys_from_values.py:44:100: unparsable-with-libyaml: DOCUMENTATION: None - mapping values are not allowed in this context
plugins/filter/remove_keys_from_values.py:60:3: error: EXAMPLES: syntax error: expected <block end>, but found '-' (syntax)
plugins/filter/remove_keys_from_values.py:60:3: unparsable-with-libyaml: EXAMPLES: while parsing a block mapping - did not find expected key

The test ansible-test sanity --test pep8 [explain] failed with 2 errors:

plugins/filter/remove_keys_from_values.py:172:1: W293: blank line contains whitespace
plugins/filter/remove_keys_from_values.py:208:1: E302: expected 2 blank lines, found 1

The test ansible-test sanity --test yamllint [explain] failed with 4 errors:

plugins/filter/remove_keys_from_values.py:44:100: error: DOCUMENTATION: syntax error: mapping values are not allowed here (syntax)
plugins/filter/remove_keys_from_values.py:44:100: unparsable-with-libyaml: DOCUMENTATION: None - mapping values are not allowed in this context
plugins/filter/remove_keys_from_values.py:60:3: error: EXAMPLES: syntax error: expected <block end>, but found '-' (syntax)
plugins/filter/remove_keys_from_values.py:60:3: unparsable-with-libyaml: EXAMPLES: while parsing a block mapping - did not find expected key

The test ansible-test sanity --test pylint [explain] failed with 1 error:

plugins/filter/remove_keys_from_values.py:172:0: trailing-whitespace: Trailing whitespace

The test ansible-test sanity --test pep8 [explain] failed with 2 errors:

plugins/filter/remove_keys_from_values.py:172:1: W293: blank line contains whitespace
plugins/filter/remove_keys_from_values.py:208:1: E302: expected 2 blank lines, found 1

The test ansible-test sanity --test yamllint [explain] failed with 4 errors:

plugins/filter/remove_keys_from_values.py:44:100: error: DOCUMENTATION: syntax error: mapping values are not allowed here (syntax)
plugins/filter/remove_keys_from_values.py:44:100: unparsable-with-libyaml: DOCUMENTATION: mapping values are not allowed in this context
plugins/filter/remove_keys_from_values.py:60:3: error: EXAMPLES: syntax error: expected <block end>, but found '-' (syntax)
plugins/filter/remove_keys_from_values.py:60:3: unparsable-with-libyaml: EXAMPLES: while parsing a block mapping did not find expected key

The test ansible-test sanity --test ansible-doc [explain] failed with the error:

Command "ansible-doc -t filter community.general.accumulate community.general.counter community.general.crc32 community.general.dict community.general.dict_kv community.general.from_csv community.general.from_ini community.general.groupby_as_dict community.general.hashids_decode community.general.hashids_encode community.general.jc community.general.json_diff community.general.json_patch community.general.json_patch_recipe community.general.json_query community.general.keep_keys community.general.lists_difference community.general.lists_intersect community.general.lists_mergeby community.general.lists_symmetric_difference community.general.lists_union community.general.random_mac community.general.remove_keys community.general.remove_keys_from_values community.general.replace_keys community.general.reveal_ansible_type community.general.to_days community.general.to_hours community.general.to_ini community.general.to_milliseconds community.general.to_minutes community.general.to_months community.general.to_prettytable community.general.to_seconds community.general.to_time_unit community.general.to_weeks community.general.to_years community.general.unicode_normalize community.general.version_sort" returned exit status 1.
>>> Standard Error
ERROR! filter community.general.remove_keys_from_values missing documentation (or could not parse documentation): community.general.remove_keys_from_values did not contain a DOCUMENTATION attribute (/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py). Unable to parse documentation in python file '/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py': mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100. mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100

The test ansible-test sanity --test ansible-doc [explain] failed with the error:

Command "ansible-doc -t filter community.general.accumulate community.general.counter community.general.crc32 community.general.dict community.general.dict_kv community.general.from_csv community.general.from_ini community.general.groupby_as_dict community.general.hashids_decode community.general.hashids_encode community.general.jc community.general.json_diff community.general.json_patch community.general.json_patch_recipe community.general.json_query community.general.keep_keys community.general.lists_difference community.general.lists_intersect community.general.lists_mergeby community.general.lists_symmetric_difference community.general.lists_union community.general.random_mac community.general.remove_keys community.general.remove_keys_from_values community.general.replace_keys community.general.reveal_ansible_type community.general.to_days community.general.to_hours community.general.to_ini community.general.to_milliseconds community.general.to_minutes community.general.to_months community.general.to_prettytable community.general.to_seconds community.general.to_time_unit community.general.to_weeks community.general.to_years community.general.unicode_normalize community.general.version_sort" returned exit status 1.
>>> Standard Error
ERROR! filter community.general.remove_keys_from_values missing documentation (or could not parse documentation): community.general.remove_keys_from_values did not contain a DOCUMENTATION attribute (/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py). Unable to parse documentation in python file '/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py': mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100. mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100

The test ansible-test sanity --test pylint [explain] failed with 1 error:

plugins/filter/remove_keys_from_values.py:172:0: trailing-whitespace: Trailing whitespace

The test ansible-test sanity --test pylint [explain] failed with 1 error:

plugins/filter/remove_keys_from_values.py:172:0: trailing-whitespace: Trailing whitespace

The test ansible-test sanity --test ansible-doc [explain] failed with the error:

Command "ansible-doc -t filter community.general.accumulate community.general.counter community.general.crc32 community.general.dict community.general.dict_kv community.general.from_csv community.general.from_ini community.general.groupby_as_dict community.general.hashids_decode community.general.hashids_encode community.general.jc community.general.json_diff community.general.json_patch community.general.json_patch_recipe community.general.json_query community.general.keep_keys community.general.lists_difference community.general.lists_intersect community.general.lists_mergeby community.general.lists_symmetric_difference community.general.lists_union community.general.random_mac community.general.remove_keys community.general.remove_keys_from_values community.general.replace_keys community.general.reveal_ansible_type community.general.to_days community.general.to_hours community.general.to_ini community.general.to_milliseconds community.general.to_minutes community.general.to_months community.general.to_prettytable community.general.to_seconds community.general.to_time_unit community.general.to_weeks community.general.to_years community.general.unicode_normalize community.general.version_sort" returned exit status 1.
>>> Standard Error
[ERROR]: filter community.general.remove_keys_from_values Missing documentation (or could not parse documentation): filter plugin 'community.general.remove_keys_from_values' did not contain a DOCUMENTATION attribute in '/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py': Unable to parse documentation in python file '/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py': mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100

The test ansible-test sanity --test pylint [explain] failed with 1 error:

plugins/filter/remove_keys_from_values.py:172:0: trailing-whitespace: Trailing whitespace

The test ansible-test sanity --test ansible-doc [explain] failed with the error:

Command "ansible-doc -t filter community.general.accumulate community.general.counter community.general.crc32 community.general.dict community.general.dict_kv community.general.from_csv community.general.from_ini community.general.groupby_as_dict community.general.hashids_decode community.general.hashids_encode community.general.jc community.general.json_diff community.general.json_patch community.general.json_patch_recipe community.general.json_query community.general.keep_keys community.general.lists_difference community.general.lists_intersect community.general.lists_mergeby community.general.lists_symmetric_difference community.general.lists_union community.general.random_mac community.general.remove_keys community.general.remove_keys_from_values community.general.replace_keys community.general.reveal_ansible_type community.general.to_days community.general.to_hours community.general.to_ini community.general.to_milliseconds community.general.to_minutes community.general.to_months community.general.to_prettytable community.general.to_seconds community.general.to_time_unit community.general.to_weeks community.general.to_years community.general.unicode_normalize community.general.version_sort" returned exit status 1.
>>> Standard Error
ERROR! filter community.general.remove_keys_from_values missing documentation (or could not parse documentation): community.general.remove_keys_from_values did not contain a DOCUMENTATION attribute (/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py). Unable to parse documentation in python file '/root/ansible_collections/community/general/plugins/filter/remove_keys_from_values.py': mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100. mapping values are not allowed in this context
  in "<unicode string>", line 36, column 100

click here for bot help

[felixfontein]

Thanks for your contribution!

If anyone is wondering how to do this with pure ansible-core + jinja:

- hosts: localhost
  gather_facts: false
  tasks:
    - ansible.builtin.debug:
        msg: >-
          {{
            input | dict2items | rejectattr("value", "none") | items2dict
          }}
      vars:
        input:
          foo: bar
          baz: bam
          bam: ~
    - ansible.builtin.debug:
        msg: >-
          {{
            input | map("dict2items") | map("rejectattr", "value", "none") | map("items2dict") | list
          }}
      vars:
        input:
          - foo: bar
            baz: bam
            bam: ~
          - foo:
            bar: baz
            baz:

I'm wondering whether it wouldn't be better to have a filter that's more flexible, by allowing the user to specify a test that's applied to values (similar to Jinja's reject/accept/rejectattr/acceptattr filters).

I'm not sure how simple that is to implement, though. You'd definitely need more Jinja2 internals.

[bcoca]

A 'drop' filter could work with map, does not need to iterate over objects itself slimdicts = {{listofdicts | map( 'drop', ['key1', 'key2', 'key3']) }}

[tanganellilore]

Hi @felixfontein, Yes this is how i resolve actually, trasform dict2items and rejectattr, multiple time based on attributes to be rejected.

For the key we have the filter and the map, but i don'se something more complex for value. I'm open to rework the solution.

Let me know if you want to track it with issue.

Thanks

[bcoca]

will add to my collection, but here is a quick and dirty 'drop' filter

# -*- coding: utf-8 -*-
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function
__metaclass__ = type


from collections.abc import Sequence

def do_drop(obj, keys):

    if isinstance(obj, dict):
        for k in keys:
            del obj[k]
    elif isinstance(obj, Sequence):
        for k in keys:
            obj.remove(k)
    else:
        raise TypeError("Drop only works on dictionaries or lists")

    return obj

class FilterModule(object):
    ''' Ansible core jinja2 filters '''

    def filters(self):
        return {
            'drop': do_drop,
        }            

[felixfontein]

@bcoca we already have a filter for removing keys from dictionaries: community.general.remove_keys. This PR is about removing specific values (in particular None).

[felixfontein]

Let me know if you want to track it with issue.

Using this PR for discussions is also fine IMO.

[bcoca]

TIL

[tanganellilore]

@bcoca we already have a filter for removing keys from dictionaries: community.general.remove_keys. This PR is about removing specific values (in particular None).

Yes, I also added the support for other value like "exact" or "regex", as the same as remove_keys filter.

In this implementation values are customizable, like list of match case or single match case, and with matching type exact we simple check if value of the key is in value, with regex we apply the match regex. Moreorve is present a recursive param that allo us to avoid iterate inside nested dict or list of dict (so stop to first level of dict or list of dict key/value)

Let me now What do you think, Thanks.

[ansibullbot]

@tanganellilore this PR contains the following merge commits:

• https://github.com/ansible-collections/community.general/commit/800e6d0d68b6d35d1dad122a134016f283ac051d

Please rebase your branch to remove these commits.

click here for bot help

[tanganellilore]

Hi @russoz , fix done as requested.

Thanks and sorry for delay.

[felixfontein]

Regarding my question about why not allow arbitrary Jinja2 tests, similar as for the reject filter (https://jinja.palletsprojects.com/en/stable/templates/#jinja-filters.reject).

For the key we have the filter and the map, but i don'se something more complex for value. I'm open to rework the solution.

Was this related to that question? If yes I'm afraid I don't understand what you mean.

[tanganellilore]

Regarding my question about why not allow arbitrary Jinja2 tests, similar as for the reject filter (https://jinja.palletsprojects.com/en/stable/templates/#jinja-filters.reject).

For the key we have the filter and the map, but i don'se something more complex for value. I'm open to rework the solution.

Was this related to that question? If yes I'm afraid I don't understand what you mean.

Sorry, not clear to me the request :-).

[felixfontein]

The reject filter (look at its docs!) allows to use arbitrary Jinja2 tests, like some_list | reject("eq", "foo") will remove all entries in some_list that are equal to the string "foo" (docs for eq test: https://jinja.palletsprojects.com/en/stable/templates/#jinja-tests.eq); some_list | reject("community.general.a_module") will remove all entries in some_list that name Ansible modules (https://docs.ansible.com/ansible/devel/collections/community/general/a_module_test.html); and some_list | reject("community.general.ansible_type", "dict[str, str]") will remove all entries in some_list which are dictionaries with string keys and string values (https://docs.ansible.com/ansible/devel/collections/community/general/ansible_type_test.html). This makes the reject filter extremely flexible, since you don't have to stick to one of the few provided tests, but can use an arbitrary Jinja2 test.

[tanganellilore]

@felixfontein thanks for clarification, now should be much clear. If I understood, correct if I'm wrong, your use case is to reject element in a list that match a test case, so yes, for this use case reject with a custom testcase is the correct way to do it.

I tried also usage of rejectattr but require a dict2items (or map of dict2items for list of dict) to apply reject in the value key and a rollback with items2dict. Moreover this not cover the "recursive" use case covered by my proposal and as far as i know regex use case.

Some example tested as per ref:

- name: Remove empty keys from list of dictionaries
  set_fact:
    cleaned_list: "{{ test_list_dict | map('dict2items')
      | map('rejectattr', 'value', 'in', [None, '', [], {}]) | map('items2dict')
      | list }}"
  vars:
    test_list_dict: 
      - a: []
        b: test
        h: false
        subkey:
          c: ''
          d: []
          e: 0
      - f: []
        g: test
        h: test_h
        subkey:
          i: test
          l: []
          m: 0
- name: Remove empty keys from dictionaries
  set_fact:
    cleaned_list: "{{ test_dict | dict2items
      | rejectattr('value', 'in', [None, '', [], {}])
      | items2dict }}"
  vars:
    test_dict: 
      a: []
      b: test
      h: false
      subkey:
        c: ''
        d: []
        e: 0
      

My proposal is to remove a key of dict based on it's value, on first or all level of dict and with support of multiple test case, that seems to me different from remove whole element in a list (as rejecet) or remove element with test case starting frok him key.

Yes, my proposal support list of dict, but also in this case I will remove the key of dict element that match value/options passed.

Let me know if it's clear.

Thanks

[felixfontein]

@felixfontein thanks for clarification, now should be much clear. If I understood, correct if I'm wrong, your use case is to reject element in a list that match a test case, so yes, for this use case reject with a custom testcase is the correct way to do it.

That's not my use case, that was an example of what "uses an arbitrary test plugin" means. My suggestion is to create a filter plugin remove_keys_from_values that uses the same technique to test values, instead of having hard-coded methods as in the current implementation.

With that you'd do something like my_list | community.general.remove_empty_keys("in", ['foo', 'bar']) insead of my_list | community.general.remove_empty_keys(values=['foo', 'bar']).