Receiving 'NoneType' error when using meraki_management_interface on some MR devices

[sstone173]

While updating DNS settings on a large group of MR devices some fail with the error "AttributeError: 'NoneType' object has no attribute 'read'\n". Haven't been able to discover any correlation when updating like devices and configurations, some devices just error out while others succeed.

Play

- name: Meraki Playbook - Update AP DNS
  collections:
    - cisco.meraki
  connection: local
  hosts: localhost

  vars:
    auth_key: XXXXXXXXXXXXXXXXXXXXXXXXXXX
    org_id: "XXXXXX"
    net_name: "XXXXXXXXXXXX"

  tasks:

    - name: Query Network Devices
      meraki_device:
        auth_key: "{{ auth_key }}"
        org_id: "{{ org_id }}"
        net_name: "{{ net_name }}"
        state: query
      delegate_to: localhost
      register: response

    - name: Update AP Network Settings
      include_tasks: loop_meraki_update_mr_ip_settings.yaml
      loop: "{{ response.data }}"

Loop Tasks (loop_meraki_update_mr_ip_settings.yaml)

- name: Gather Existing AP Network Configuration
  meraki_management_interface:
    auth_key: "{{ auth_key }}"
    state: query
    org_id: "{{ org_id }}"
    net_id: "{{ item.network_id }}"
    serial: "{{ item.serial }}"
  register: output

- name: Update Network Configuration on AP
  meraki_management_interface:
    auth_key: "{{ auth_key }}"
    state: present
    org_id: "{{ org_id }}"
    net_id: "{{ item.network_id }}"
    serial: "{{ item.serial }}"
    wan1:
      wan_enabled: enabled
      using_static_ip: yes
      static_ip: "{{ output.data.wan1.static_ip }}"
      static_gateway_ip: "{{ output.data.wan1.static_gateway_ip }}"
      static_subnet_mask: "{{ output.data.wan1.static_subnet_mask }}"
      static_dns:
        - 10.1.2.3
        - 10.4.5.6
      vlan: "{{ output.data.wan1.vlan }}"

Output

TASK [Gather Existing AP Network Configuration] ***************************************************************************************************************************************************
task path: /mnt/c/Users/user1/Automation/projects/ansible/production/loop_meraki_update_mr_ip_settings.yaml:7
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: user1
<127.0.0.1> EXEC /bin/sh -c 'echo ~user1 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user1/.ansible/tmp `"&& mkdir "` echo /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398 `" && echo ansible-tmp-1626361069.196938-18378-119557732644398="` echo /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398 `" ) && sleep 0'
Using module file /home/user1/.ansible/collections/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py
<127.0.0.1> PUT /home/user1/.ansible/tmp/ansible-local-18161xfzhqhe0/tmp9xgz21nq TO /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/AnsiballZ_meraki_management_interface.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/ /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1626361069.196938-18378-119557732644398/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "data": {
        "wan1": {
            "static_dns": [
                "8.8.4.4",
                "8.8.8.8"
            ],
            "static_gateway_ip": "192.168.1.1",
            "static_ip": "192.138.1.24",
            "static_subnet_mask": "255.255.255.0",
            "using_static_ip": true,
            "vlan": 100
        }
    },
    "invocation": {
        "module_args": {
            "auth_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "follow_redirects": "all",
            "host": "api.meraki.com",
            "internal_error_retry_time": 60,
            "net_id": "N_XXXXXXXXXXXXX",
            "net_name": null,
            "org_id": "XXXXXX",
            "org_name": null,
            "output_format": "snakecase",
            "output_level": "normal",
            "protocol": "https",
            "rate_limit_retry_time": 165,
            "serial": "XXXX-XXXX-XXXX",
            "state": "query",
            "timeout": 30,
            "use_https": true,
            "use_proxy": false,
            "validate_certs": true,
            "wan1": null,
            "wan2": null
        }
    },
    "response": "OK (unknown bytes)",
    "status": 200
}

TASK [Update Network Configuration on AP] *********************************************************************************************************************************************************
task path: /mnt/c/Users/user1/Automation/projects/ansible/production/loop_meraki_update_mr_ip_settings.yaml:17
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: user1
<127.0.0.1> EXEC /bin/sh -c 'echo ~user1 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user1/.ansible/tmp `"&& mkdir "` echo /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844 `" && echo ansible-tmp-1626361070.1113186-18405-131799258753844="` echo /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844 `" ) && sleep 0'
Using module file /home/user1/.ansible/collections/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py
<127.0.0.1> PUT /home/user1/.ansible/tmp/ansible-local-18161xfzhqhe0/tmp8mqcq264 TO /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/ /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py", line 100, in <module>
    _ansiballz_main()
  File "/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py", line 92, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', init_globals=dict(_module_fqn='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', _modlib_path=modlib_path),
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py", line 384, in <module>
  File "/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py", line 347, in main
  File "/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/module_utils/network/meraki/meraki.py", line 477, in request
AttributeError: 'NoneType' object has no attribute 'read'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py\", line 100, in <module>\n    _ansiballz_main()\n  File \"/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py\", line 92, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/user1/.ansible/tmp/ansible-tmp-1626361070.1113186-18405-131799258753844/AnsiballZ_meraki_management_interface.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', init_globals=dict(_module_fqn='ansible_collections.cisco.meraki.plugins.modules.meraki_management_interface', _modlib_path=modlib_path),\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py\", line 384, in <module>\n  File \"/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_management_interface.py\", line 347, in main\n  File \"/tmp/ansible_meraki_management_interface_payload_rxj5c4cp/ansible_meraki_management_interface_payload.zip/ansible_collections/cisco/meraki/plugins/module_utils/network/meraki/meraki.py\", line 477, in request\nAttributeError: 'NoneType' object has no attribute 'read'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

[kbreit]

Thank you for the report. I'll try to take a look at it this weekend.

Do you have any custom settings that may cause this? Have you tried this same function using a Python script natively? I'm trying to determine if it's a bug with my code (probably) or if it's in the API (much less likely) that's triggering a bug.

[sstone173]

I don't think I have anything custom that would cause this, my environment is pretty simple, but who knows. I haven't tried the same function using Python natively, I was happy to just get this working. The issue didn't occur very often, maybe only 5% of the time and sometimes repeating the play 3-4 times would finally get it to work, so it seems random to me.

[kbreit]

@sstone173 I tested this on my end and it seems to work fine. Admittedly, it's intermittent for you so I didn't expect it would reproduce easily here. A few thoughts:

• If I send you an updated file for improved debugging, are you able to run it and reproduce the bug with that?
• Do you find it's the same AP this happens on? Different APs? Same model? Anything like that we can use to hone in on the issue?
• I can work with Meraki to see if they have an idea where this could be a problem.

Either way, I should be handling this exception. That fixes the symptom but not the bug though.

[sstone173]

I can perform some additional testing if you send me the updated file, shouldn't be an issue. The issue didn't seem to be specific to model or any particular AP configuration, that's what made it a bit confusing to troubleshoot on my end as well.

[kbreit]

I'll need a few days to develop and test the code. My current thought is if it catches an exception in that area, it will output the payload from the request so we can see what is being submitted. I'll also have it output the HTTP code to see if we're getting back a bad code. It's probably not a bad thing to have the capability for anyways.

[kbreit]

I haven't tested this since I can't reproduce the bug. But please find the location where the Meraki collection is installed. Go into plugins/module_utils/network/meraki and you should see a meraki.py file. Replace that file with https://gist.github.com/kbreit/2f1d78e935f2dee7cac83b912d6a0275 . Reproduce the bug and it should give some more information if it fails with NoneType. If it doesn't, let me know and we can work on improvements.

Update: I had an error in the syntax and fixed it. Please redownload if you have it already.

[sstone173]

Updated the meraki.py file, but haven't been able to reproduce the issue so far, so I'm thinking it may have just been a temporary issue on my end, possibly network related.

[kbreit]

You bring up a good point. Before the custom code, it would fail as an unhandled exception. Now, catching the exception so it should still continue to execute. I'll modify the code to catch the exception and fail with better output.

I'm on PTO the next couple of days so I should be able to make the changes by the weekend.

[kbreit]

@sstone173 Sorry about the delay. I updated the gist so it should fail if NoneType happens. Give it a few more tries to see if you can reproduce the issue.

[sstone173]

@kbreit Updated the gist and ran the playbook against a bunch of our sites and was not able to replicate the issue. I'll be updating a large number of sites again in a month or so, so feel free to close this issue and I can reopen or create a new one if I run into it again during that project.

[kbreit]

Go ahead and keep this open for a while. No need to close it. I'll check back in later this year.

[kbreit]

@sstone173 Funny story. I sit down tonight to start some development on a new module. I'm putting together the framework for the integration tests and run one. I'll give you one guess on what error I received. I ran it again, worked fine.

[sstone173]

Yes, that does sound familiar :)

[mtb-xt]

still happens in other modules too, from time to time. usually, running again makes the error go away

 - An exception occurred during task execution. To see the full traceback, use -vvv. 
The error was: AttributeError: 'NoneType' object has no attribute 'read'

failed: [127.0.0.1] (item=MS350-1- port 5) => {"ansible_loop_var": "item", "changed": false, "item": [{"lat": 37.4180951010362, "lng": -122.098531723022, "model": "MS350-24", "name": "MS350-1-VA", "ports": [{"allowed_vlans": "all", "enabled": true, "isolation_enabled": false, "link_negotiation": "Auto negotiate", "poe_enabled": false, "port_id": "1", "rstp_enabled": true, 

.../ansible_meraki_switchport_payload_gqu5by09/ansible_meraki_switchport_payload.zip/ansible_collections/cisco/meraki/plugins/modules/meraki_switchport.py\", line 549, in main
File \"/tmp/ansible_meraki_switchport_payload_gqu5by09/ansible_meraki_switchport_payload.zip/ansible_collections/cisco/meraki/plugins/module_utils/network/meraki/meraki.py\", line 477, in request
AttributeError: 'NoneType' object has no attribute 'read'
", "module_stdout": "", "msg": "MODULE FAILURE
See stdout/stderr for the exact error", "rc": 1}

[kbreit]

That's what I've seen. My theory is it's a server side error but can't reproduce it enough to be helpful. If you'd like, I can see if I can get the Meraki team in contact with you to troubleshoot.

[mtb-xt]

That's what I've seen. My theory is it's a server side error but can't reproduce it enough to be helpful. If you'd like, I can see if I can get the Meraki team in contact with you to troubleshoot.

I'm communicating with them about some other issues, and this happens... sporadically. But, would it be possible to not error out on this - the modules should probably catch that error and not crash, and possible allow retrying the request?

[kbreit]

Makes sense. I'm going to at a minimum catch the exception and handle it so it doesn't crash. If I can get it to redo the request, I'll add that in too.

[kbreit]

@mtb-xt I'm doing a little troubleshooting and added debugging code. I've had this issue three times tonight and got this output.

fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "body": null, "changed": false, "msg": "Failure occurred", "response": "Request failed: <urlopen error [Errno 99] Cannot assign requested address>", "status": -1}

A quick search with that error and urllib, which I think Ansible uses, returns https://stackoverflow.com/questions/11981933/python-urllib2-cannot-assign-requested-address. I'm looking into this further but made progress.

To your point, yes, this should be caught somehow. But if I can solve the problem, that's even better!

[kbreit]

@mtb-xt When you receive these errors, are they local on your system or a dedicated server? I ask because I had a lot of errors while addressing #257 but after quitting Slack and Discord, the errors seem to have largely disappeared.

Update: This may not be the problem. Having lots of problems still.

[kbreit]

@sstone173 I created a pull request to handle and retry the request 5 times before failing. I think this is due to something with Docker because as soon as I started running my tests locally and not in Docker, the errors disappeared. Because this problem is intermittent I can't test it. But please try this patch in a test environment and let me know if this fixes the issues. Thank you.