OpenStack Watcher Modules fail to load

[uchi-mata]

trafficstars

Please make sure that you have checked the boxes:

• [X] Pease review the Troubleshooting doc for additional details regarding your issue.
  • Debug Logging is enabled
  • Reviewed other issues
• [X] Review the Quickstart guide
• [X] Search for both open and closed issues regarding the problem you are experiencing
  • https://github.com/Netflix/security_monkey/issues/1112 and https://github.com/Netflix/security_monkey/issues/1126 seem similar but result in different problems.
• [X] For permissions issues (Access Denied and credential related errors), please refer to the requisite docs before submitting an issue: AWS, GCP, OpenStack, GitHub

Description of issue:

I created an OpenStack account (which is active in the Dashboard). However, no data seems to get imported so I ran monkey find_changes manually to identify any issues. The only issue I can identify is that all openstack watcher modules fail to load. The full logfile is attached (monkey_find_changes.log), this is just an excerpt:

2019-02-05 09:19:47,363 DEBUG: Failed to load module openstack_watcher from /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/openstack_watcher.py [in /usr/local/lib/python2.7/dist-packages/security_monkey/common/utils.py:103]
2019-02-05 09:19:47,363 DEBUG: Loaded module __init__ from /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/__init__.py [in /usr/local/lib/python2.7/dist-packages/security_monkey/common/utils.py:105]
2019-02-05 09:19:47,364 DEBUG: Failed to load module openstack_port from /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/network/openstack_port.py [in /usr/local/lib/python2.7/dist-packages/security_monkey/common/utils.py:103]

I'm using the git master with docker-compose on docker-compose.yml. Is there any more data I can provide or do you already have any ideas?

Edit:

• The module files (e.g. /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/network/openstack_port.py) are available.
• I tested the monkey find_changes in the secmonkey-scheduler and secmonkey-worker container, if that makes a difference?

Thanks, Matthias

[mstair]

* The module files (e.g. /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/network/openstack_port.py) are available.

To confirm, those are available in the container? This error is typically due to the missing openstacksdk.

You also have your creds/yaml configured/mounted (https://github.com/Netflix/security_monkey/blob/master/docker-compose.yml#L69)?

[uchi-mata]

* The module files (e.g. /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/network/openstack_port.py) are available.

To confirm, those are available in the container? This error is typically due to the missing openstacksdk.

Does that work?

$ docker exec secmonkey-worker ls /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/network/
__init__.py
__init__.pyc
openstack_floating_ip.py
openstack_floating_ip.pyc
openstack_network.py
openstack_network.pyc
openstack_port.py
openstack_port.pyc
openstack_router.py
openstack_router.pyc
openstack_security_group.py
openstack_security_group.pyc
openstack_subnet.py
openstack_subnet.pyc
openstack@openstack-secmonkey:~/security_monkey$ docker exec secmonkey-scheduler ls /usr/local/lib/python2.7/dist-packages/security_monkey/watchers/openstack/network/
__init__.py
__init__.pyc
openstack_floating_ip.py
openstack_floating_ip.pyc
openstack_network.py
openstack_network.pyc
openstack_port.py
openstack_port.pyc
openstack_router.py
openstack_router.pyc
openstack_security_group.py
openstack_security_group.pyc
openstack_subnet.py
openstack_subnet.pyc

You also have your creds/yaml configured/mounted (https://github.com/Netflix/security_monkey/blob/master/docker-compose.yml#L69)?

No, but the clouds.yaml file is mounted which from my understanding is the correct one for the OpenStack connection?

  worker:
    [...]
    volumes:
      - ./docker/celeryconfig.py:/usr/local/src/security_monkey/security_monkey/celeryconfig.py
      - ./clouds.yaml:/clouds.yaml

/clouds.yaml is also configured as path in the account settings.

[uchi-mata]

Also there does not seem any request from the monkey instance to the openstack instance to take place (based on tcpdump while running monkey find_changes). I however verified that the identity API can be accessed from the monkey instance.

[mstair]

I actually wonder if this is an issue with the os-client-config library that recently came up. I have a PR to cloudaux (SM helper library) to address. https://github.com/Netflix-Skunkworks/cloudaux/pull/96

Testing a potential workaround pinning the os-client-config in Dockerfile pips

[mstair]

@mikegrima Just merged and pushed changes to pypi. Rebuild a clean image (shoud pull in cloudaux 1.6.1).