Unable to create IPAddress when using NautobotModel

[brendanbowden]

Environment

• Python version: 3.11.6
• Nautobot version: 2.0.3
• nautobot-ssot version: 2.0.1

Expected Behavior

Create an IPAddress object with a specified parent prefix or namespace.

Observed Behavior

Tried several ways of indicating what the parent prefix is with no luck:

• Using "parent__prefix" attribute gives me the error "Either a parent or a namespace must be provided";
• Not specifying a prefix at all gives the same error;
• Adding namespace__name so it has both parent and namespace returns error "IPAddress has no field named 'namespace'"
• The modeling doc suggests a series of "assigned_object" fields but that doesn't work at all, I think those are only for 1.x?

I'm successfully adding Device and Interface records; I suspect this might have something to do with the special handing of "either prefix or namespace is valid" in IPAddress.

Steps to Reproduce

Model:

class NautobotIPAddressModel(NautobotModel):
    _model = IPAddress
    _modelname = "ip_address"
    _identifiers = (
        "host",
        "mask_length",
    )
    _attributes = (
        "status__name",
        "description",
        # "namespace__name",
        "parent__prefix",
    )

    host: str
    mask_length: str
    status__name: str
    description: Optional[str]
    # namespace__name: str
    parent__prefix: str

Adapter (segment that adds IP Address objects:, apologies but had to sanitize the surrounding code a bit):

class RemoteSystemAdapter(NautobotAdapter):

ip_address = NautobotIPAddressModel

    def load(self):
        source_data = get_remote_device_info()
        ip_address_model = self.ip_address(
                        host=source_data['ipAddress'],
                        mask_length="32",
                        status__name="Active",
                        # namespace__name="Global",
                        parent__prefix="0.0.0.0/0"
        )
        self.add(ip_address_model)

[Kircheneer]

If you look into the IPAddress model - it only has a field for parent, not for namespace, which is causing this:

Adding namespace__name so it has both parent and namespace returns error "IPAddress has no field named 'namespace'"

---

Using "parent__prefix" attribute gives me the error "Either a parent or a namespace must be provided";

Can you share the entire stack trace for this? I suspect the problem is that in order to create a prefix you need to supply either a parent prefix or the namespace, so additionally parent__prefix__parent__prefix (for which, recursively the same rules would apply) or parent__prefix__namespace__name.

---

The modeling doc suggests a series of "assigned_object" fields but that doesn't work at all, I think those are only for 1.x?

This is only relevant if you want to assign IP addressed to interfaces. Is this something you want to do?

In any case, we still need to update the documentation here definitely.

[brendanbowden]

Not looking to create a new prefix at the same time (yet). At the moment these are individual IP addresses from network devices (don't even have mask in my source data, so I'm defaulting to /32) that I'm trying to import and attach to interfaces. We don't have integration to Infoblox setup yet so I was going to import them all into an overall "global" prefix for now, and then move them to the correct prefix hierarchy later. Stack trace of the error when using parent__prefix below (slightly sanitized for IPs/names).

Re: the second question: overall my flow is "import device; import interface and attach to device; import IP address and attach to interface." So far I've been using device__name in the Interface model and use model.add_child() to explicitly attach it to the device; and I hadn't gotten that far on the IP-to-interface relation yet. Ideally what I think I want is "device" as top-level because that's what I'm importing, and interface and IP address as "cascading" children because that's what the source data looks like. add_child() is the only way I'd found to "link in" records of types referenced in _children. Is there a different way I should be setting up the parent-child relationships?

Stack trace:

[2023-11-16 14:56:11,722: ERROR/ForkPoolWorker-7] nautobot_ssot_custom.jobs.sevone.SevOneDataSource[ba3499ca-25ee-4ea4-aa95-d1791fa47e95]: An exception occurred: `ValidationError: ["Parameters: {'host': '10.1.1.1', 'mask_length': '32', 'status__name': 'Active', 'parent__prefix': '0.0.0.0/0'}"]`
Traceback (most recent call last):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/contrib.py", line 357, in _update_obj_with_parameters
    obj.validated_save()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot/core/models/__init__.py", line 118, in validated_save
    self.full_clean()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/django/db/models/base.py", line 1251, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'parent': ['Either a parent or a namespace must be provided.']}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 322, in run
    self.sync_data(memory_profiling)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 166, in sync_data
    self.execute_sync()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 98, in execute_sync
    self.source_adapter.sync_to(self.target_adapter, flags=self.diffsync_flags)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/__init__.py", line 596, in sync_to
    return target.sync_from(self, diff_class=diff_class, flags=flags, callback=callback, diff=diff)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/__init__.py", line 568, in sync_from
    result = syncer.perform_sync()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 329, in perform_sync
    changed |= self.sync_diff_element(element)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 401, in sync_diff_element
    changed |= self.sync_diff_element(child, parent_model=dst_model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 401, in sync_diff_element
    changed |= self.sync_diff_element(child, parent_model=dst_model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 377, in sync_diff_element
    changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 426, in sync_model
    dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/contrib.py", line 297, in create
    cls._update_obj_with_parameters(obj, parameters)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/contrib.py", line 359, in _update_obj_with_parameters
    raise ValidationError(f"Parameters: {parameters}") from error
django.core.exceptions.ValidationError: ["Parameters: {'host': '10.1.1.1', 'mask_length': '32', 'status__name': 'Active', 'parent__prefix': '0.0.0.0/0'}"]

[bradh11]

@brendanbowden - I noticed you have namespace commented out in your steps to reproduce. Namespace is now a required field in 2.0 as referenced in ipam-namespaces-3337.

[bradh11]

@glennmatthews Is my understanding on namespaces in 2.0 correct?

[brendanbowden]

See I thought that too; but when I enable namespace__name I get this traceback that says "IPAddress has no field named namespace". Looking at the IPAddress model directly shows it's not an explicit attribute but gets added from args during IPAddress.init(), I'm guessing that's confusing some of the runtime introspection in NautobotModel?

[2023-11-16 19:16:13,371: ERROR/ForkPoolWorker-7] nautobot_ssot_custom.jobs.remote.RemoteDataSource[3aaf42e2-0e34-4347-8412-38ee4db36a12]: An exception occurred: `FieldDoesNotExist: IPAddress has no field named 'namespace'`
Traceback (most recent call last):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/django/db/models/options.py", line 608, in get_field
    return self.fields_map[field_name]
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'namespace'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 322, in run
    self.sync_data(memory_profiling)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 166, in sync_data
    self.execute_sync()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 98, in execute_sync
    self.source_adapter.sync_to(self.target_adapter, flags=self.diffsync_flags)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/__init__.py", line 596, in sync_to
    return target.sync_from(self, diff_class=diff_class, flags=flags, callback=callback, diff=diff)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/__init__.py", line 568, in sync_from
    result = syncer.perform_sync()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 329, in perform_sync
    changed |= self.sync_diff_element(element)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 401, in sync_diff_element
    changed |= self.sync_diff_element(child, parent_model=dst_model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 401, in sync_diff_element
    changed |= self.sync_diff_element(child, parent_model=dst_model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 377, in sync_diff_element
    changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/diffsync/helpers.py", line 426, in sync_model
    dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/contrib.py", line 297, in create
    cls._update_obj_with_parameters(obj, parameters)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/nautobot_ssot/contrib.py", line 322, in _update_obj_with_parameters
    django_field = cls._model._meta.get_field(related_model)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/nautobot-ssot-custom-bYjOkexK-py3.11/lib/python3.11/site-packages/django/db/models/options.py", line 610, in get_field
    raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.exceptions.FieldDoesNotExist: IPAddress has no field named 'namespace'

[glennmatthews]

At the database level, an IPAddress derives its namespace through its parent Prefix rather than directly referencing a namespace itself.

However, you can provide a namespace when creating an IPAddress as there's code in __init__ and clean() to specifically handle that, where it uses the namespace to find the appropriate parent Prefix. I imagine it isn't handling nested lookups like namespace__name as part of that logic. The expected code would be something like:

IPAddress.objects.create(address="192.0.2.1/24", status=status_instance, namespace=namespace_instance)

[glennmatthews]

If there's specifically a need to be able to do things like IPAddress(namespace__name="...") or IPAddress(parent__prefix="...") that would have to be some new code added in core.

[Kircheneer]

Yeah we use introspection to look at the actual model fields, if it is only used in __init__ then it is going to raise that error.

[Kircheneer]

@brendanbowden did the above comments help address this issue?