data_group.py removes type before updating but F5 complains about missing type

[oh-c]

In https://github.com/F5Networks/f5-common-python/blob/development/f5/bigip/tm/ltm/data_group.py

lines 57:69 there is an if statement to remove the "type" from the dictionary..

However, my BIG-IP 14.1.0.3 Build 0.0.6 Point Release 3 complains when calling "update" without this.

(note this is an internal data group)

In [42]: rpd.dg.update()
DEBUG:icontrol.session:put WITH uri: https://f5.foo.local:443/mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev/ AND suffix:  AND kwargs: {'json': {'fullPath': '/mediamanagement/mediamanagement_whitelist_dev', 'generation': 345485, 'kind': 'tm:ltm:data-group:internal:internalstate', 'name': 'mediamanagement_whitelist_dev', 'partition': 'mediamanagement', 'records': [{'data': '', 'name': '10.1.0.0/16'}], 'selfLink': 'https://localhost/mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev?ver=14.1.0.3'}}
DEBUG:urllib3.connectionpool:https://f5.foo.local:443 "PUT /mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev/ HTTP/1.1" 400 97
DEBUG:icontrol.session:RESPONSE::STATUS: 400 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
Text: '{"code":400,"message":"invalid property value \\"type\\":\\"\\"","errorStack":[],"apiError":26214401}'
DEBUG:icontrol.session:get WITH uri: https://f5.foo.local:443/mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev/ AND suffix:  AND kwargs: {}
DEBUG:urllib3.connectionpool:Resetting dropped connection: f5.foo.local
DEBUG:urllib3.connectionpool:https://f5.foo.local:443 "GET /mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev/ HTTP/1.1" 200 380
DEBUG:icontrol.session:RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
Text: '{"kind":"tm:ltm:data-group:internal:internalstate","name":"mediamanagement_whitelist_dev","partition":"mediamanagement","fullPath":"/mediamanagement/mediamanagement_whitelist_dev","generation":345485,"selfLink":"https://localhost/mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev?ver=14.1.0.3","type":"ip","records":[{"name":"10.1.0.0/16","data":""}]}'
---------------------------------------------------------------------------
iControlUnexpectedHTTPError               Traceback (most recent call last)
<ipython-input-42-8424931d1723> in <module>
----> 1 rpd.dg.update()

~/venv/py/lib64/python3.6/site-packages/f5/bigip/tm/ltm/data_group.py in update(self, **kwargs)
     67         if 'type' in self.__dict__:
     68             del self.__dict__['type']
---> 69         return self._update(**kwargs)
     70
     71

~/venv/py/lib64/python3.6/site-packages/f5/bigip/resource.py in _update(self, **kwargs)
    578         for _ in range(0, 30):
    579             try:
--> 580                 response = session.put(update_uri, json=data_dict, **requests_params)
    581                 self._meta_data = temp_meta
    582                 self._local_update(response.json())

~/venv/py/lib64/python3.6/site-packages/icontrol/session.py in wrapper(self, RIC_base_uri, **kwargs)
    293                              response.url,
    294                              response.text)
--> 295             raise iControlUnexpectedHTTPError(error_message, response=response)
    296         return response
    297     return wrapper

iControlUnexpectedHTTPError: 400 Unexpected Error: Bad Request for uri: https://f5.foo.local:443/mgmt/tm/ltm/data-group/internal/~mediamanagement~mediamanagement_whitelist_dev/
Text: '{"code":400,"message":"invalid property value \\"type\\":\\"\\"","errorStack":[],"apiError":26214401}'

If set my **kwargs to a dict containing the "type": "ip" it works fine..

Is the related to the version of BIG-IP I'm running?

[jasonrahm]

likely a change in expectation between versions. I know we added 14.x to the test suite but I'm not sure we've worked through all the changes introduced above 13.x. I'll run the test suite against latest 14.1 later this week and get a plan together to support it on the existing sdk functionality.

[codertao]

Experiencing the same, from f5-sdk 3.0.3 against BIG-IP v14.0.0.1 (Build 0.0.2)

[Timoses]

Is there an ETA when a fix is available?

[jasonrahm]

Sorry, I’ll try to make time this week.

[jasonrahm]

@codertao or @Timoses can you share your entire code segment that is problematic? I'm testing on 14.1.2 and am not having issues whether or not I supply the type as a kwarg. If I can run successfully with your code on 14.1.2, I can explore testing on your specific versions of 14.x tmos. My test code:

>>> dg = b.tm.ltm.data_group.internals.internal.load(name='dg_testaddr')
>>> dg.records
[{'name': '10.10.10.0/24', 'data': '45'}]
>>> dg.records.append({'name': '10.10.14.0/24', 'data': ''})
>>> dg.update()
>>> dg.records
[{'name': '10.10.10.0/24', 'data': '45'}, {'name': '10.10.14.0/24', 'data': ''}]
>>> dg.records.append({'name': '10.10.15.0/24', 'data': ''})
>>> dg.update(type='ip')
>>> dg.records
[{'name': '10.10.10.0/24', 'data': '45'}, {'name': '10.10.14.0/24', 'data': ''}, {'name': '10.10.15.0/24', 'data': ''}]

[Timoses]

@jasonrahm See the issue that codertao referenced. The code is probably located somewhere in the k8s-bigip-ctlr. It happens when configuring a Kubernetes ingress yaml to redirect http.

[jasonrahm]

looks to me like that's an implementation problem in utilizing the sdk, not a problem with the sdk itself in this case? Not intimately familiar with CIS inner workings....

[Timoses]

@codertao : Do you have more input? I'm not sure how you reached the conclusion that the k8s-bigip-ctlr issue is connected to this issue.

[codertao]

k8s-bigip-ctlr works by generating a config file picked up by f5-ctlr-agent, which is in turn using f5-common-python to talk to the F5. /the error message is from python logs.

here? is a stripped down and untested version of the config generated by k8s-bigip-ctlr / for consumption by f5-ctlr-agent; our past experience has been that changes in https_redirect_dg cause the failure. We're using v14.0.0.1 (an F5 provided AWS AMI), can't comment on if it's an issue in later versions.

Our local hack/fix was to just always set 'type' on datagroup to string (see patch below); and we haven't had an issue since. (with the caveat that we're only worried about one version of the F5, and one datagroup that is always string type)

Patch is:

d279ef0eafe7a71cfa5403bf23645d53e1c6b749 f5/bigip/tm/ltm/data_group.py | 5 +++++ 1 file changed, 5 insertions(+)

diff --git a/f5/bigip/tm/ltm/data_group.py b/f5/bigip/tm/ltm/data_group.py index 2be0f1d..1968b2a 100644 --- a/f5/bigip/tm/ltm/data_group.py +++ b/f5/bigip/tm/ltm/data_group.py @@ -66,6 +66,11 @@ class Internal(Resource): def update(self, **kwargs): if 'type' in self.dict: del self.dict['type']

•    # F5 complains about updates to datagroup that change types
  

•    # ... apparently, in v13 of F5, omitting the type was fine,
  

•    # and sending it was a problem (thus above)
  

•    # but, in v14, this is now detected as a problem :(
  

•    self.__dict__['type'] = 'string'
     return self._update(**kwargs)
  

[jasonrahm]

ah, ok. That's helpful. So it looks like in the v14 versions mentioned here as well as v14.1.2 at least, the string and integer types require the type to be submitted, whereas the address type does not. I checked v13 latest and this doesn't seem to be a problem with any of them. This is a rest interface issue that the SDK currently is not equipped to handle. The workaround you are using is sufficient but I'll look at version-specific dependencies for the string/integer data-group types.