Issue in SSOT[ACI] sync job

[JockeW-DvL]

Environment

• Python version: 3.10.12
• Nautobot version: 2.0.3
• nautobot-ssot version: 2.0.0

Sync job from ACI should work

Observed Behavior

I get the following error when doing a sync job (both Dry-run and real job)

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot/jobs/base.py", line 322, in run
    self.sync_data(memory_profiling)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot/jobs/base.py", line 136, in sync_data
    self.load_source_adapter()
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot/integrations/aci/jobs.py", line 73, in load_source_adapter
    self.source_adapter = AciAdapter(job=self, sync=self.sync, client=aci_creds[self.apic])
KeyError: 'ash'

The env variables i've set are the following:

NAUTOBOT_APIC_BASE_URI_ASH=https://<apic URL>
NAUTOBOT_APIC_USERNAME_ASH=nautobot
NAUTOBOT_APIC_PASSWORD_ASH=<secret password>
NAUTOBOT_APIC_VERIFY_ASH=False
NAUTOBOT_APIC_SITE_ASH="ASH ACI"
NAUTOBOT_APIC_TENANT_PREFIX_ASH="ASH_ACI"

Steps to Reproduce

[Kircheneer]

Hi, thanks for submitting! Can you please let me know what your nautobot_configuration.py, specifically the section for PLUGINS_CONFIG looks like? What the environment variables do or don't do depends on that.

[JockeW-DvL]

Here are the config in the nautobot_configuratiion.py:

PLUGINS = ["nautobot_ssot"]
PLUGINS_CONFIG = {
    "nautobot_ssot": {
        "enable_aci": True,
        # URL and credentials should be configured as environment variables on the host system
        "aci_apics": {x: os.environ[x] for x in os.environ if "NAUTOBOT_APIC" in x},
        # Tag which will be created and applied to all synchronized objects.
        "aci_tag": "ACI",
        "aci_tag_color": "0047AB",
        # Tags indicating state applied to synchronized interfaces.
        "aci_tag_up": "UP",
        "aci_tag_up_color": "008000",
        "aci_tag_down": "DOWN",
        "aci_tag_down_color": "FF3333",
        # Manufacturer name. Specify existing, or a new one with this name will be created.
        "aci_manufacturer_name": "Cisco",
        # Exclude any tenants you would not like to bring over from ACI.
        "aci_ignore_tenants": ["common", "mgmt", "infra"],
        # The below value will appear in the Comments field on objects created in Nautobot
        "aci_comments": "Created by ACI SSoT Integration",
    }
}

[JockeW-DvL]

Any update on this issue?

[Kircheneer]

Sorry for the long wait, but can you share one last thing with me, which is a sanitized version of the following:

nautobot-server nbshell
$ from django.conf import settings
$ settings.PLUGINS_CONFIG["nautobot_ssot"]
...

Just check the output for whether there is any info you don't want publicly available. I have the feeling there is a config mismatch here somehow and this should help troubleshoot that.

[JockeW-DvL]

Hello,

here are the output from the command you reqested:

nautobot@eru456:~$ nautobot-server nbshell
# Shell Plus Model Imports
from constance.backends.database.models import Constance
from django.contrib.admin.models import LogEntry
from django.contrib.auth.models import Group, Permission
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from django_celery_beat.models import ClockedSchedule, CrontabSchedule, IntervalSchedule, PeriodicTask, PeriodicTasks, SolarSchedule
from django_celery_results.models import ChordCounter, GroupResult, TaskResult
from nautobot.circuits.models import Circuit, CircuitTermination, CircuitType, Provider, ProviderNetwork
from nautobot.dcim.models.cables import Cable, CablePath
from nautobot.dcim.models.device_component_templates import ConsolePortTemplate, ConsoleServerPortTemplate, DeviceBayTemplate, FrontPortTemplate, InterfaceTemplate, PowerOutletTemplate, PowerPortTemplate, RearPortTemplate
from nautobot.dcim.models.device_components import ConsolePort, ConsoleServerPort, DeviceBay, FrontPort, Interface, InterfaceRedundancyGroup, InterfaceRedundancyGroupAssociation, InventoryItem, PowerOutlet, PowerPort, RearPort
from nautobot.dcim.models.devices import Device, DeviceRedundancyGroup, DeviceType, Manufacturer, Platform, VirtualChassis
from nautobot.dcim.models.locations import Location, LocationType
from nautobot.dcim.models.power import PowerFeed, PowerPanel
from nautobot.dcim.models.racks import Rack, RackGroup, RackReservation
from nautobot.extras.models.change_logging import ObjectChange
from nautobot.extras.models.customfields import ComputedField, CustomField, CustomFieldChoice
from nautobot.extras.models.datasources import GitRepository
from nautobot.extras.models.groups import DynamicGroup, DynamicGroupMembership
from nautobot.extras.models.jobs import Job, JobButton, JobHook, JobLogEntry, JobResult, ScheduledJob, ScheduledJobs
from nautobot.extras.models.models import ConfigContext, ConfigContextSchema, CustomLink, ExportTemplate, FileAttachment, FileProxy, GraphQLQuery, HealthCheckTestModel, ImageAttachment, Note, Webhook
from nautobot.extras.models.relationships import Relationship, RelationshipAssociation
from nautobot.extras.models.roles import Role
from nautobot.extras.models.secrets import Secret, SecretsGroup, SecretsGroupAssociation
from nautobot.extras.models.statuses import Status
from nautobot.extras.models.tags import Tag, TaggedItem
from nautobot.ipam.models import IPAddress, IPAddressToInterface, Namespace, Prefix, RIR, RouteTarget, Service, VLAN, VLANGroup, VRF, VRFDeviceAssignment, VRFPrefixAssignment
from nautobot.tenancy.models import Tenant, TenantGroup
from nautobot.users.models import AdminGroup, ObjectPermission, Token, User
from nautobot.virtualization.models import Cluster, ClusterGroup, ClusterType, VMInterface, VirtualMachine
from nautobot_ssot.integrations.servicenow.models import SSOTServiceNowConfig
from nautobot_ssot.models import Sync, SyncLogEntry
from social_django.models import Association, Code, Nonce, Partial, UserSocialAuth
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When
from django.utils import timezone
from django.urls import reverse
from django.db.models import Exists, OuterRef, Subquery
# Django version 3.2.23
# Nautobot version 2.0.5
# Single Source of Truth version 2.0.1
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)

[Kircheneer]

Sorry, I am missing here the output of the second line, the settings - can you add that as well? This one is just the startup output of the shell.

[JockeW-DvL]

Sorry about that, here comes the missing output with password changed:

from django.conf import settings settings.PLUGINS_CONFIG["nautobot_ssot"] {'enable_aci': True, 'aci_apics': {'NAUTOBOT_APIC_BASE_URI_ASH': 'https://dc4apic01.ngninfra.net', 'NAUTOBOT_APIC_USERNAME_ASH': 'admin', 'NAUTOBOT_APIC_TENANT_PREFIX_ASH': 'ASH_ACI', 'NAUTOBOT_APIC_SITE_ASH': 'ASH ACI', 'NAUTOBOT_APIC_PASSWORD_ASH': 'XXXXXXXX', 'NAUTOBOT_APIC_VERIFY_ASH': 'False'}, 'aci_tag': 'ACI', 'aci_tag_color': '0047AB', 'aci_tag_up': 'UP', 'aci_tag_up_color': '008000', 'aci_tag_down': 'DOWN', 'aci_tag_down_color': 'FF3333', 'aci_manufacturer_name': 'Cisco', 'aci_ignore_tenants': ['common', 'mgmt', 'infra'], 'aci_comments': 'Created by ACI SSoT Integration', 'aci_site': '', 'aristacv_apply_import_tag': False, 'aristacv_controller_site': '', 'aristacv_create_controller': False, 'aristacv_cvaas_url': 'www.arista.io:443', 'aristacv_cvp_host': '', 'aristacv_cvp_password': '', 'aristacv_cvp_port': '443', 'aristacv_cvp_token': '', 'aristacv_cvp_user': '', 'aristacv_delete_devices_on_sync': False, 'aristacv_from_cloudvision_default_device_role': '', 'aristacv_from_cloudvision_default_device_role_color': '', 'aristacv_from_cloudvision_default_site': '', 'aristacv_hostname_patterns': [], 'aristacv_import_active': False, 'aristacv_role_mappings': {}, 'aristacv_site_mappings': {}, 'aristacv_verify': True, 'device42_host': '', 'device42_username': '', 'device42_password': '', 'device42_defaults': {}, 'device42_delete_on_sync': False, 'device42_use_dns': True, 'device42_customer_is_facility': True, 'device42_facility_prepend': '', 'device42_role_prepend': '', 'device42_ignore_tag': '', 'device42_hostname_mapping': [], 'enable_aristacv': False, 'enable_device42': False, 'enable_infoblox': False, 'enable_ipfabric': False, 'enable_servicenow': False, 'hide_example_jobs': True, 'infoblox_default_status': '', 'infoblox_enable_rfc1918_network_containers': False, 'infoblox_enable_sync_to_infoblox': False, 'infoblox_import_objects_ip_addresses': False, 'infoblox_import_objects_subnets': False, 'infoblox_import_objects_vlan_views': False, 'infoblox_import_objects_vlans': False, 'infoblox_import_subnets': [], 'infoblox_password': '', 'infoblox_url': '', 'infoblox_username': '', 'infoblox_verify_ssl': True, 'infoblox_wapi_version': '', 'ipfabric_api_token': '', 'ipfabric_host': '', 'ipfabric_ssl_verify': True, 'ipfabric_timeout': 15, 'ipfabric_nautobot_host': '', 'servicenow_instance': '', 'servicenow_password': '', 'servicenow_username': ''}

[Kircheneer]

Thanks for providing this, we will look into it!

[JockeW-DvL]

Any update on this issue? i've upgraded to the latest version of nautobot and ssot, still the issue persist

[chadell]

Taking it to reproduce it, and then solve it

[chadell]

I just run it from latest version and worked smoothly:

These were my creds using the devnet always on instance:

NAUTOBOT_APIC_BASE_URI_ASH=https://sandboxapicdc.cisco.com
NAUTOBOT_APIC_USERNAME_ASH=admin
NAUTOBOT_APIC_TENANT_PREFIX_ASH="ASH_ACI"
NAUTOBOT_APIC_SITE_ASH="ASH ACI"
NAUTOBOT_APIC_PASSWORD_ASH=<password>
NAUTOBOT_APIC_VERIFY_ASH=False

could you double check it?

[JockeW-DvL]

I saw that my include config needed updated, so i uppdated so it matched your. But i'm still getting the same error. So now my config is:

SSoT ACI for ACI fabric in Ashburn

NAUTOBOT_APIC_BASE_URI_ASH=https://dc4apic01.ngninfra.net NAUTOBOT_APIC_USERNAME_ASH=nautobot NAUTOBOT_APIC_PASSWORD_ASH="password removed" NAUTOBOT_APIC_VERIFY_ASH=False NAUTOBOT_APIC_SITE_ASH="ASH_ACI" NAUTOBOT_APIC_TENANT_PREFIX_ASH="ASH_ACI"

[JockeW-DvL]

I Just tried to uninstall nautobot_ssot[aci] and i got stuck on the nautobot migrate stage. So i reinstalled nautobot and i'm still getting the error.

It seems to me that something is missing in the database

[chadell]

Hi @JockeW-DvL , in the nbshell, I reproduced all the steps that are done to process the aci_credsdictionary (using your config), and it succeeds:

>>> config = {'aci_apics': {'NAUTOBOT_APIC_BASE_URI_ASH': 'https://dc4apic01.ngninfra.net', 'NAUTOBOT_APIC_USERNAME_ASH': 'admin', 'NAUTOBOT_APIC_TENANT_PREFIX_ASH': 'ASH_ACI', 'NAUTOBOT_APIC_SITE_ASH': 'ASH ACI', 'NAUTOBOT_APIC_PASSWORD_ASH': 'XXXXXXXX', 'NAUTOBOT_APIC_VERIFY_ASH': 'False'}}
>>> {key[4:]: value for key, value in config.items() if key.startswith("aci_")}
{'apics': {'NAUTOBOT_APIC_BASE_URI_ASH': 'https://dc4apic01.ngninfra.net', 'NAUTOBOT_APIC_USERNAME_ASH': 'admin', 'NAUTOBOT_APIC_TENANT_PREFIX_ASH': 'ASH_ACI', 'NAUTOBOT_APIC_SITE_ASH': 'ASH ACI', 'NAUTOBOT_APIC_PASSWORD_ASH': 'XXXXXXXX', 'NAUTOBOT_APIC_VERIFY_ASH': 'False'}}
>>> config_processed = {key[4:]: value for key, value in config.items() if key.startswith("aci_")}
>>> config_processed
{'apics': {'NAUTOBOT_APIC_BASE_URI_ASH': 'https://dc4apic01.ngninfra.net', 'NAUTOBOT_APIC_USERNAME_ASH': 'admin', 'NAUTOBOT_APIC_TENANT_PREFIX_ASH': 'ASH_ACI', 'NAUTOBOT_APIC_SITE_ASH': 'ASH ACI', 'NAUTOBOT_APIC_PASSWORD_ASH': 'XXXXXXXX', 'NAUTOBOT_APIC_VERIFY_ASH': 'False'}}
>>> aci_creds = {}
>>> for key in config_processed["apics"]:
...     subkey = key[key.rfind("_") + 1 :].lower()  # noqa: E203
...     aci_creds.setdefault(subkey, {})
...     if "USERNAME" in key:
...         aci_creds[subkey]["username"] = config_processed["apics"][key]
...     if "PASSWORD" in key:
...         aci_creds[subkey]["password"] = config_processed["apics"][key]
...     if "URI" in key:
...         aci_creds[subkey]["base_uri"] = config_processed["apics"][key]
...
{}
{'base_uri': 'https://dc4apic01.ngninfra.net'}
{'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin'}
{'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin'}
{'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin'}
{'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin', 'password': 'XXXXXXXX'}
>>> aci_creds
{'ash': {'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin', 'password': 'XXXXXXXX'}}

The aci_creds contains the "ash" key that your environment is complaining about.

Could you add some debug message in the code to check what do you have in aci_creds just before the error? (in nautobot_ssot/integrations/aci/jobs.py)

It's also weird that keyerror because the job renders the available options from the same dictionary:

class AciDataSource(DataSource, Job): 
    """ACI SSoT Data Source."""

    apic_choices = [(key, key) for key in aci_creds]

[JockeW-DvL]

Hello Christian,Thanks for the suggestion I will take a look at this next week as I’m at Cisco live in Amsterdam this week.RegardsJoacim W On 5 February 2024 at 06:45:52, Christian Adell @.***) wrote: Hi @JockeW-DvL , in the nbshell, I reproduced all the steps that are done to process the aci_credsdictionary (using your config), and it succeeds:

{key[4:]: value for key, value in config.items() if key.startswith("aci_")} {'apics': {'NAUTOBOT_APIC_BASE_URI_ASH': 'https://dc4apic01.ngninfra.net', 'NAUTOBOT_APIC_USERNAME_ASH': 'admin', 'NAUTOBOT_APIC_TENANT_PREFIX_ASH': 'ASH_ACI', 'NAUTOBOT_APIC_SITE_ASH': 'ASH ACI', 'NAUTOBOT_APIC_PASSWORD_ASH': 'XXXXXXXX', 'NAUTOBOT_APIC_VERIFY_ASH': 'False'}} config_processed = {key[4:]: value for key, value in config.items() if key.startswith("aci_")} config_processed {'apics': {'NAUTOBOT_APIC_BASE_URI_ASH': 'https://dc4apic01.ngninfra.net', 'NAUTOBOT_APIC_USERNAME_ASH': 'admin', 'NAUTOBOT_APIC_TENANT_PREFIX_ASH': 'ASH_ACI', 'NAUTOBOT_APIC_SITE_ASH': 'ASH ACI', 'NAUTOBOT_APIC_PASSWORD_ASH': 'XXXXXXXX', 'NAUTOBOT_APIC_VERIFY_ASH': 'False'}} aci_creds = {} for key in config_processed["apics"]: ... subkey = key[key.rfind("_") + 1 :].lower() # noqa: E203 ... aci_creds.setdefault(subkey, {}) ... if "USERNAME" in key: ... aci_creds[subkey]["username"] = config_processed["apics"][key] ... if "PASSWORD" in key: ... aci_creds[subkey]["password"] = config_processed["apics"][key] ... if "URI" in key: ... aci_creds[subkey]["base_uri"] = config_processed["apics"][key] ... {} {'base_uri': 'https://dc4apic01.ngninfra.net'} {'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin'} {'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin'} {'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin'} {'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin', 'password': 'XXXXXXXX'} aci_creds {'ash': {'base_uri': 'https://dc4apic01.ngninfra.net', 'username': 'admin', 'password': 'XXXXXXXX'}}

The aci_creds contains the "ash" key that your environment is complaining about. Could you add some debug message in the code to check what do you have in aci_creds just before the error? (in nautobot_ssot/integrations/aci/jobs.py) It's also weird that keyerror because the job renders the available options from the same dictionary: class AciDataSource(DataSource, Job):
"""ACI SSoT Data Source."""

apic_choices = [(key, key) for key in aci_creds]

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.> [ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/nautobot/nautobot-app-ssot/issues/262#issuecomment-1926273089", "url": "https://github.com/nautobot/nautobot-app-ssot/issues/262#issuecomment-1926273089", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[chadell]

hi @JockeW-DvL , if you are at Cisco Live, you can pass by the NTC booth and there are a few engineers who could help you it.

[jdrew82]

@JockeW-DvL Can you confirm whether the recent updates to the ACI integration have resolved your issue? I'll close this out in a week if we don't hear back from you.

[jdrew82]

Closing as we haven't heard back.