strings separated by comma become a list with jinja2_native enabled

[bluikko]

Summary

If two strings that are separated by a comma have templating applied to them then with jinja2_native=true the result incorrectly becomes a list consisting of 2 string elements, one for each of the strings.

With jinja2 native disabled, with the strings separated to different lines using a YAML block scalar, or with no templating applied to the strings then the result stays a string as is expected.

Issue Type

Documentation Report

Component Name

core

Ansible Version

$ ansible --version
ansible [core 2.15.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ansible/lib64/python3.9/site-packages/ansible
  ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ansible/bin/ansible
  python version = 3.9.14 (main, Jan  9 2023, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)] (/home/ansible/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CACHE_PLUGIN(/etc/ansible/ansible.cfg) = community.general.yaml
CACHE_PLUGIN_CONNECTION(/etc/ansible/ansible.cfg) = /tmp/.ansible-fact.$USER
CACHE_PLUGIN_TIMEOUT(/etc/ansible/ansible.cfg) = 5184000
CALLBACKS_ENABLED(/etc/ansible/ansible.cfg) = ['ansible.posix.timer']
CONFIG_FILE() = /etc/ansible/ansible.cfg
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 10
DEFAULT_JINJA2_NATIVE(/etc/ansible/ansible.cfg) = True
DEFAULT_LOCAL_TMP(/etc/ansible/ansible.cfg) = /tmp/.ansible.user/ansible-local-268537xzh471wi
DEFAULT_LOG_PATH(/etc/ansible/ansible.cfg) = /var/log/ansible.log
DEFAULT_STDOUT_CALLBACK(/etc/ansible/ansible.cfg) = community.general.yaml
DEFAULT_STRATEGY(/etc/ansible/ansible.cfg) = ansible.builtin.free
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 50
DEFAULT_VAULT_IDENTITY_LIST(env: ANSIBLE_VAULT_IDENTITY_LIST) = ['...', '...', '...']
DEFAULT_VAULT_ID_MATCH(/etc/ansible/ansible.cfg) = True
INJECT_FACTS_AS_VARS(/etc/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_silent
RETRY_FILES_SAVE_PATH(/etc/ansible/ansible.cfg) = /tmp/.ansible-retry.user

CACHE:
=====

jsonfile:
________
_timeout(/etc/ansible/ansible.cfg) = 5184000
_uri(/etc/ansible/ansible.cfg) = /tmp/.ansible-fact.user

CONNECTION:
==========

paramiko_ssh:
____________
ssh_args(/etc/ansible/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=300s -o ServerAliveInterval=10
timeout(/etc/ansible/ansible.cfg) = 50

ssh:
___
control_path_dir(/etc/ansible/ansible.cfg) = /tmp/.ansible-ssh.$USER
pipelining(/etc/ansible/ansible.cfg) = True
ssh_args(/etc/ansible/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=300s -o ServerAliveInterval=10
timeout(/etc/ansible/ansible.cfg) = 50
use_tty(/etc/ansible/ansible.cfg) = False

SHELL:
=====

sh:
__
remote_tmp(/etc/ansible/ansible.cfg) = /tmp/ansible.$USER

OS / Environment

EL9, pip install

Steps to Reproduce

---
- name: Test case for a string changing to a list with jinja2 native
  hosts:
    - all
  gather_facts: false
  tasks:
    - name: Demonstrate a string becoming a list
      ansible.builtin.debug:
        msg: |
          'string{{ "1" }}', 'string{{ "2" }}'

Note that it does not matter if the msg is a block scalar or not, the issue still happens. A block scalar is just used to easily demonstrate this workaround that makes it work as expected (output stays a string):

---
- name: Test case for a string changing to a list with jinja2 native (workaround)
  hosts:
    - all
  gather_facts: false
  tasks:
    - name: Workaround with separating the strings to 2 lines makes the output work right and stay a string
      ansible.builtin.debug:
        msg: |
          'string{{ "1" }}',
          'string{{ "2" }}'

Expected Results

The output should stay a string such as:

'string1', 'string2'

Actual Results

ok: [host] =>
  msg:
  - string1
  - string2

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description: None

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[sivel]

Duplicate of https://github.com/ansible/ansible/issues/73538

[bluikko]

In this case there is a single string passed as parameter as enclosed in the outmost double quotes that I used as my original test case (that I had changed to YAML block scalar):

    - name: Demonstrate string becoming a list
      ansible.builtin.debug:
        msg: "'string{{ 1 | string }}', 'string{{ 2 | string }}'"

I guess those double quotes make no difference then and it's not considered a single string with jinja2_native on.

[sivel]

Basically, you need 4 layers of quotes in your original example, which can be complicated. Using a block effectively counts as 1 quote layer:

    - name: Demonstrate a string becoming a list
      ansible.builtin.debug:
        msg: |
          '''string{{ "1" }}', 'string{{ "2" }}'''

So you have a block scalar | which tells YAML you are building a string, an outer set of ', then a YAML escaped internal ' represented by '', and then the innermost "

You are dealing with both the yaml parser, jinja2, and then python. With jinja2 native, you are templating out python data structures, and a comma separated string is a tuple in python, which we then serialize to a list.

>>> import ast
>>> type(ast.literal_eval("'foo', 'bar'"))
<class 'tuple'>
>>> type(ast.literal_eval('''"'foo', 'bar'"'''))
<class 'str'>

So you need to add an extra set of wrapping quotes, so that jinja/python know you are creating a string, and not a tuple.

While we attempt to make the switch from non-native jinja2 to native as easy as possible, there are just some caveats that require adjustments.

[bluikko]

Well thanks, I was not expecting this turn into a support case.

While we attempt to make the switch from non-native jinja2 to native as easy as possible, there are just some caveats that require adjustments.

I wonder if this could perhaps be documented somehow better. Currently I could not find any documentation beyond https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-jinja2-native , perhaps what you have just explained seems obvious to you but I'm not sure it is the same to many of us users.

[bluikko]

Using the range | join filters illustrates this "problem" well. Consider the task:

- name: Test string changing into a list
  ansible.builtin.copy:
    dest: /tmp/test
    content: '{{ range(1, 8, 2) | join(",") }}'

This results in a non-string being written to the file:

(1, 3, 5, 7)

I am not sure how adding more quotation marks is supposed to help here, as was suggested in the previous reply:

    content: '''{{ range(1, 8, 2) | join(",") }}'''

And similarly when using a block scalar:

    content: |
      '''{{ range(1, 8, 2) | join(",") }}'''

Does results in a string, but also additional quotation marks in the file content which is not useful and breaks the syntax:

'1,3,5,7'

Any even number of quotation marks results in a syntax error if not using a block scalar so I am not sure what "a block effectively counts as 1 quote layer" means - using 4 quotation marks without block scalar does not seem to work.

The suggested workaround of using additional quotes does not seem to help as the quotes are present in the output. Adding | string at the end results in the expected output 1,3,5,7. It is totally unintuitive that a string needs to be explicitly cast into a string. The output of | join is supposed to be a string, no?

[sivel]

In the case of this last example, all you want to do is add use of the |string filter:

content: '{{ range(1, 8, 2) | join(",") | string }}'

Using the string filter, will mark the result as something that should not be further evaluated with ast.literal_eval