The module community.general.read_csv was redirected to community.general.files.read_csv, which could not be loaded.

[clsmith70]

Summary

I just started working on a project and the first step is to read a CSV and process the data. I have an execution environment set up with community.general loaded in it and GitHub actions recompiles and republishes it weekly as well as when changes are introduced. When I run the role, which only contains this task right now, it returns "The module community.general.read_csv was redirected to community.general.files.read_csv, which could not be loaded."

Issue Type

Bug Report

Component Name

read_csv

Ansible Version

$ ansible --version
ansible [core 2.13.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/chris/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/chris/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 3.0.3
  libyaml = True

---

< Ansible Automation Platform Controller 4.1.1 >

      \
      \   ^__^
          (oo)\_______
          (__)      A )\
              ||----w |
              ||     ||

Community.general Version

$ ansible-galaxy collection list community.general
Collection        Version
----------------- -------
community.general 5.2.0

Configuration

$ ansible-config dump --only-changed
nothing returned

OS / Environment

Red Hat Enterprise Linux 8.6 (Ootpa)

Steps to Reproduce

- name: Read employee information
  community.general.read_csv:
    path: "{{ source_path }}"
  register: users

Expected Results

A list of the 2 records in the test file in the task detail info

Actual Results

{
  "msg": "The module community.general.read_csv was redirected to community.general.files.read_csv, which could not be loaded.",
  "_ansible_no_log": false
}

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/files/read_csv.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @dagwieers click here for bot help

[felixfontein]

I just installed ansible-core 2.13.0 and community.general 5.2.0 in a new docker container, and could not reproduce this.

How did you install community.general 5.2.0?

needs_info

[felixfontein]

Ah, I missed the execution environment part. I did try it again with an EE including community.geneal. quay.io/ansible/ansible-runner has no tags for 2.13 yet, though, so I build one with ansible-core 2.12. That one also worked fine for me. You probably want to check the directory /usr/share/ansible/collections/ansible_collections/community/general/plugins/modules/ (assuming it's the same directory in your environment, I looked at the ansible-galaxy collection list output in a container started with that image to obtain the base path /usr/share/ansible/collections/ansible_collections) inside your EE image. It should contain a list of subdirectories, including files, and files should contain a file read_csv.py. If it's not there, something went wrong during building the EE image.

[clsmith70]

I will check that and see if I can rebuild the EE, if necessary. I just tried it from the command line on my Ansible host and it worked like a charm. I'll update this issue when I find the proper sequence to get it working with the EE in case others run into this too.

[clsmith70]

When I pull a copy of the EE and inspect it, I find the path to read_csv.py as you outlined earlier. When I run a job using the EE in the controller web interface, it still reports that error. I've even tried setting a separate copy of the EE up so a new copy would be pulled, and the same error was still returned.

[clsmith70]

I tried this same job at work and received the same result there. The EE is built in a separate repo and stored in a different quay.io account. All attempts work using ansible-playbook on the command line, but they all fail via the Ansible controller web interface. It appears to be relative to the EE. I did check both community EE's and find read.csv in /usr/share/ansible/collections/ansible_collections/community/general/plugins/modules/. Everything I have tried to get this to work has failed.

I will use the win_shell module to import the CSV and make use of the data that way. Not the way I wanted to address this case, but I know it will work.

[felixfontein]

If it's possible to make the built EE available somehow (or the exact way to build it, if that can be done using non-proprietary tools/images/...), that would be great. I'd really like to debug this, but I cannot reproduce this using what's available to the community. Otherwise you'll have to take this to Red Hat support, since you apparently pay for Ansible Automation Platform, maybe they can figure out what goes wrong.

[clsmith70]

You can clone my ee-build repo. It currently builds an EE for Azure and Community. Nothing special or proprietary.

clsmith70 - ee-build

[felixfontein]

I build the community EE with ansible-builder build -v 3 -t test-ee:latest --container-runtime=podman (ansible-builder 1.0.1) and run a tiny playbook with ansible-navigator run --mode stdout --pull-policy never --execution-environment-image test-ee:latest -vvv ee.yml (ansible-navigator 2.1.0). It says ansible-playbook [core 2.12.5.post0] at startup though (that's the ansible-core version contained in the base image ansible-builder uses, quay.io/ansible/ansible-runner:latest). Anyway, I also cannot reproduce it this way...

[clsmith70]

I am reaching out to a resource to see if they can help reproduce and resolve this issue. For now, I have this functionality filled in using ansible.windows.win_powershell.

[felixfontein]

You can (probably) also work around this by limiting community.general to version < 5.0.0. Earlier versions are using symlinks instead of ansible-base 2.10+ redirects.

[clsmith70]

Thank you @felixfontein!!! That worked!

To help others that may run into this problem, here is an example of what your requirements.yml file might look like to build your EE container.

---
collections:
  - name: community.windows
  - name: community.general
    version: <5.0.0
  - name: community.dns
  - name: community.crypto

[felixfontein]

Well, but that's really not a good fix, since it prevents you from using the latest version. This seems to be a severe bug somewhere that needs to get fixed, so let's better keep this open :)

[felixfontein]

Another instance of this problem (with ansible-base 2.10.8 and the mail module) was reported in https://groups.google.com/d/msgid/ansible-project/ecbd618d-12d2-4e4e-b2c0-e07f6d311e98n%40googlegroups.com

[gotmax23]

I was not able to reproduce this. I am running the latest version of AAP (2.2) on RHEL 9, and I rebuilt your execution environment with the latest version of community.general using ansible-builder from the AAP yum repository. The ansible-builder version from this repository uses the base images published by Red Hat by default.

@clsmith70, are you able to try building your EE with the latest version of community.general and then substituting the module name in the playbook with community.general.files.read_csv and seeing if that works[^1]?

It would help if you could provide more information about your setup. Which EE image are you using? How did you build it? What version of AAP? What's the output of sudo dnf repolist on the system where Automation Controller is installed?

It would also help to have the ouput of ansible-navigator images -d --eei=[registry.io/execution-env-with-latest-community-general] (after replacing the placeholder value). You'll have to use a Gist or other pastebin for that, as it provides a LOT of information.

[^1]: You shouldn't do this normally. I'm just trying to narrow down the problem.

Also, where did you run ansible --version? It looks like that version is installed with pip and not the version from AAP. Either way, that's not really relevant, as ansible is run from within the EE, not the host.

[felixfontein]

Hmm, a comment by @gotmax23 on IRC ("Also, is the fact that they're running against a windows host a factor?") got me thinking. This could indeed be the important factor. Some wild guessing: maybe you are running against a Windows host, and use delegate_to: localhost to run the module locally. Now ansible-core for whatever reason tries to find a powershell module following the redirect, even though the local host is Linux. That would explain the error message. And maybe this problem only happens if a redirect is involved, and did not happen when there were symlinks inside the collection (which is why it works with c.g 4.x.y but not with 5.x.y)?

[felixfontein]

Actually I managed to reproduce this: with an inventory file like

[windows]
192.168.253.253 ansible_connection=winrm

(that machine doesn't actually exist) and playbook

- hosts: windows
  gather_facts: no
  tasks:
    - name: Read employee information
      community.general.read_csv:
        path: /foo
      register: users

Then I get

TASK [Read employee information] *************************************************************************************************************************************************************
fatal: [192.168.253.253]: FAILED! => {"msg": "The module community.general.read_csv was redirected to community.general.files.read_csv, which could not be loaded."}

When I add delegate_to: localhost it works fine (Unable to open file: [Errno 2] No such file or directory: '/foo'). With community.general 4.x.y (edited: and no delegate_to!) I get fatal: [192.168.253.253]: UNREACHABLE! => {"changed": false, "msg": "ssl: auth method ssl requires a username", "unreachable": true} (which is not surprising since that machine doesn't exist).

Since community.general does not support Windows (see for example https://github.com/ansible-collections/community.general#community-general-collection) it is not surprising that it doesn't work like this. But then @clsmith70's environment doesn't seem to be that simple, otherwise it should have never worked with community.general 4.x.y either.

[briantist]

When I add delegate_to: localhost it works fine (Unable to open file: [Errno 2] No such file or directory: '/foo'). With community.general 4.x.y I get fatal: [192.168.253.253]: UNREACHABLE! => {"changed": false, "msg": "ssl: auth method ssl requires a username", "unreachable": true} (which is not surprising since that machine doesn't exist).

That is surprising to me; if you delegated to localhost it should be using the local connection for execution and not trying to connect to the "remote" host at all, no? The target host using a different connection doesn't change that.

This whole issue seems like there is something wrong with the redirect code. Generally speaking delegation (including to the localhost) works fine when the inventory target is Windows or is using winrm/psrp connections.

[felixfontein]

I meant that with c.g 4.x.y and not delegating I got the 'unreachable' error (which isn't unsurprising). With c.g 4.x.y and delegate_to I also got "Unable to open file..." as expected.

[felixfontein]

@clsmith70 could you please tell us more about your environment?

[clsmith70]

@felixfontein, my apologies for the late response. I've been away from work recently.

I have a lab that I set up to mirror work as closely as possible. At work, we have an old domain running on an old FFL/DFL that we are migrating from. That domain has its oldest DCs running 2003 FFL/DFL on 2008 R2 Server. This has limited my use of some newer collections and modules due to interoperability issues. My lab is using a 2012 server running a 2008 R2 FFL/DFL and presents a few of the same issues around interoperability.

When I use the c.g collection <5.0.0, I can copy a file from a windows server to the local Ansible host, then process the file in the local host with no problems. If I use the latest c.g version, this doesn't work quite right. I am going to run some additional tests today to verify everything as I haven't worked on it in for a while.

[briantist]

I just noticed in the original post you're using ansible-core==2.13.0, have you tried >=2.13.1 yet? If I recall correctly it had a critical bug that prevented connecting to Windows systems: https://github.com/ansible/ansible/issues/77841

That manifests as a different error, and it doesn't seem to be related, but maybe when used with the redirect there's some interaction between those? Either way, it's a good idea to get an idea with the fix for working with Windows targets.

[clsmith70]

@briantist I have not tried that yet. My most recent review did not exhibit the problem in my test lab. I tested with both c.g <5.0.0 and using the latest version and both worked. I will put a test together in the production work environment to validate the issue there and then try some of the steps outlined here to see if it can be resolved or not.

@felixfontein I did use a different approach to reading the file. I copy the file from the Windows host to the Ansible controller node, then read and register the data there. The updated play information is listed below. The role is run with the Windows host and delegation is used with the AAP controller host.

- name: Copy source file {{ source_file }} at {{ source_path }}
  ansible.builtin.fetch:
    src: "{{ source_path }}{{ source_file }}"
    dest: /tmp/{{ tower_job_id }}-{{ source_file }}
    flat: true
  
- name: Read employee information
  community.general.read_csv:
    path: /tmp/{{ tower_job_id }}-{{ source_file }}
  register: users
  delegate_to: localhost

[felixfontein]

Is there any update on this? If not, since this was never reported again, I'm opting to simply close this.

needs_info

[clsmith70]

@felixfontein I have not experienced any further issues after modifying my job templates to copy the file before processing it. Thanks to everyone for their time and attention on this!