netbox
netbox copied to clipboard
netbox-community
Change interface naturalize function
NetBox version
v3.2.2
Feature type
Change to existing functionality
Proposed functionality
The interface name is naturalized to make it sortable in a proper way. (1,2,10,11,20,21; and not 1,10,11,2,20,21) The current implementation is not covering all use cases to sort interfaces properly.
By making the function more generic, it could cover more use cases:
output = ''
parts = re.split(r'(\d+)',value.lower())
for part in parts:
if part.isdigit():
output += part.rjust(4, '0')
elif part not in ['.',':','/']:
output += part
It splits the interfacename into parts with and without numbers. the numbers are pre-padded with zero's and the chars [dot,colon,slash] are removed from between the numbers. There might be corner cases with mixed number schema's on one device, but it will keep them grouped.
Unless there was a reason why the slot/subslot/position/.. were prepended to the interfacetype, rather then post ?
Use case
On Arista we have interfaces that are split in lanes, without slot/pos/subpos identifiers.
Ethernet1
Ethernet2
Ethernet3
Ethernet48/1
Ethernet48/2
Ethernet48/3
Ethernet48/4
Ethernet49/1
Ethernet49/2
Ethernet49/3
Ethernet49/4
The current rules sorts them as
Ethernet48/1
Ethernet48/2
Ethernet48/3
Ethernet48/4
Ethernet49/1
Ethernet49/2
Ethernet49/3
Ethernet49/4
Ethernet1
Ethernet2
Ethernet3
because the /1 is prepended to the interfacetype, while with non-slashed interfaces, the prepend is 9999
https://github.com/netbox-community/netbox/blob/a2203da1c6fa674a55d3071a287094d70cb1c502/netbox/utilities/ordering.py#L46
Database changes
All _name fields of all interfaces need to be re-calculated.
External dependencies
No response
While someone is looking at this... on NetApp an "e" is prepended to Ethernet interfaces whereas FC interfaces have nothing prepended. The current sort looks like this:
10a
10b
10c
10d
e9a
e9e
e11a
e11e
but I would prefer:
e9a
e9e
10a
10b
10c
10d
e11a
e11e
I have the same issue in two places, but would also need to add '-' to the list of split characters.
On Cisco ASR 9K 1/10G linecards you can specify which interfaces are 1G and which are 10G and the router accordingly labels them as GigabitEthernet and TenGigE, but they currently end up sorted with all the 1G interfaces first, then the 10G ones, and not in proper port order, which is really confusing. With modules now, even putting them all on the same module doesn't fix the sort order.
The other example is on a Cisco ONS 15454/NCS 2K you can have a QSFP+ 4x10G breakout optic inserted that will create something like VCFAC-1-1-1 for the optic and VLINE-1-1-1-1-[1-4] for the individual 10G lanes, except it sorts it so that all the facility interfaces come first, then the line interfaces. In that case, they can even be modeled as child interfaces and that doesn't affect the sort order either.
While someone is looking at this... on NetApp an "e" is prepended to Ethernet interfaces whereas FC interfaces have nothing prepended. The current sort looks like this:
10a 10b 10c 10d e9a e9e e11a e11ebut I would prefer:
e9a e9e 10a 10b 10c 10d e11a e11e
That will be difficult. the order function only knows the interface names, not the underlaying device model.
I think there could be some room for improvement, perhaps we have to look at considering how the sort happens when a slot, module, or port are missing.
That said, I think we still need to maintain the sort with the interface type, but perhaps have interface type be sorted just before port (if it isn't already).
Maybe it would make sense for all of these to be individual fields and then have the name be a computed field that is the combination of all fields.
This sort of individual field setup would only work well with most network switches, regular interfaces for the most part, don't have a concept of slot/module.
Dan, why would we need to name the parts of an interface and try to fit ? I think that is we just make the numbers a zero pretended 5 digit number, that all sorting cases work. (Except the one mgoetze5 mentioned).
@PieterL75 you can find the existing tests for the natural ordering of interfaces here. As it has taken several years for us as a community to settle on the current scheme, we will only entertain modifications that do not alter or violate any of the existing tests. Is this something you want to pursue?
So, as long as the new sorting returns the same order for the test , we should be good? The 'calculated' values will change of course, the displayed order must remain the same.
So, as long as the new sorting returns the same order for the test , we should be good?
Provided the proposed change makes sense, yes. It'll also need a new test for the scenario you described above.
The 'calculated' values will change of course, the displayed order must remain the same.
We need to be careful with this. Any changes that would result in recalculating raw names will require that users run renaturalize against all interfaces when upgrading, which is best avoided.
I've been thinking that too, but it might be something that is inevitable and needs to be weighted... Could (if needed) a migration script take care of that?
Looking at the code, I think this is because it treats empty slot, subslot, position, subposition as "9999", would it make sense to treat these as "0000" instead? That would likely solve this specific use case.
I see now in that test script why the interface type is placed after the port's..
'GigabitEthernet0/1',
'GigabitEthernet0/2',
'GigabitEthernet0/10',
'TenGigabitEthernet0/20',
'TenGigabitEthernet0/21',
same for junipers where xe, ge and te interfaces can be mixed in one 'module'
We dont want those TenG before the Gig's Going to think some more, and check @DanSheps advise
@DanSheps @jeremystretch I have a version of the naturalize_interface ready in my fork. Shall I create a PR to have it reviewed ? or will you first check it out. https://github.com/PieterL75/netbox/tree/issue_9368
It does change the rawnames, as it uses a totally different logic.
There is an option to change the logic, based on the name of the interface, and the model properties..
The current implementation passes the ./manage.py test dcim.tests.test_natural_ordering
and it also sorts the ones in the comments (netapp, vmac, ....)
I have to fix the tests so that they match the new naturalization function.... But, what I see that the biggest 'exception' is that causes my biggest headache : The sorting requirement that 'FastEthernet0' comes after 'GigaEthernet 0/1'. I don't see a reason to do that, except that the FastEthernet0 is a mgmt only interface that you don't want 'sorted' with the rest. For that I added a condition that places the mgmt_only interface always at the end of the sorting. If that adheres the requirements of why Fa0 was after Gi0/1, then I would like to get rid of that hard requirement. (I dont have the history thread on how the current function was born)
I don't see a reason to do that, except that the FastEthernet0 is a mgmt only interface that you don't want 'sorted' with the rest. For that I added a condition that places the mgmt_only interface always at the end of the sorting.
I will take a look at the tests and see if I can figure out why this is. To me, FA should come before GI, especially because "0" is the port. I believe this is because in the naturalize we use 9999 for slot and module when it is non-existant, but to me that is backwards and it should be 0000 for a null value.
I suggested the same sort of general natural ordering algorithm at https://github.com/netbox-community/netbox/issues/6882#issuecomment-898508667 and https://github.com/netbox-community/netbox/issues/2872#issuecomment-491279206 - rejected both times.
Those links include some discussion of how this doesn't agree with the existing test cases.
To me, it seems that the decisions made in the past were not the best. We should start a new discussion to agree on a new sorting algorithm that will work for a lot of cases. The corner cases that are now taken into account, and mess up other natural sorting, should go away in favor of a 'logical' scheme without exceptions. (ex : placing FastEthernet1 at the end, because on some switch types that is the mgmt interface).
I'll work out a generic sorting algorithm and post a new PR to test. This will ofcourse change the current values of _name. But that can be handled with a migration schema file. Rollback is not possible, but that is simply not possible in an version of netbox. Forward is the only way :-)
@PieterL75 if you want to post up the new FR we can definitely discuss modifying the ordering.
@DanSheps I pushed a new PR https://github.com/netbox-community/netbox/pull/9463 I suspect it will not be approved yet, but I hope I was able to build a logic that is easily extendible.
I had a very interesting discussion.. @jeremystretch @DanSheps Why bother with the naturalize function ? If we could manually set/change/dragdrop the order on the device/module-type, then that order can be taken for the derived devices. That sort-order can be retained on a device. if a new interface is added to the device itself, then it could be placed at any place and stored.... the _name field can then simply become 0001, 0002, 0003, ...
@jeremystretch @candlerb did you had time to review the pr I submitted? However, I feel more for solution that interfaces are ordered manually in the template, or on the device... No more debates, each model sorts them by it's design
I don't think a manual sort is going to be in the cards, it is too difficult to manage. We had it, pulled it out for this.
That said, I think a better method could be devised.
Did you had it when templates existed? Because with those, it is just a one time manual sort. An the option to change it a bit when it became a device... Modules might make it a challenge though..
@jeremystretch On the PR you mention that you will not allow the current naturalization function to be changed. As this current one is really not doing what it should for multiple devicetypes, I have some suggestions.
- custom naturalization function, either by a plugin or by a config setting
- manual ordering by using templates.
I prefer the manual ordering, as it will cover all possible situations. The initial ordering can be fine by such a function, but manual reordering should be an option.
Still wondering on how this can be solved. 'One Solution Fits All' will not work, so we have to be able to provide a way to customize the naturalization.
Either by a plugin that can overrule the default, or by a configuration.py setting that refers to a py function. Or a manual reorder..
Would it be possible to get the GUI do a multi-column sort? I.e. sort by column A first, then sort by column B? I know this is something that exists in some operating systems and programs, but I don't know how feasible it is to implement within Django...
This could potentially be very helpful, as a user could sort by Interface Type first, and then by Name, which would effectively allow you to "force" (or at least influence) the grouping based on the port speed.
Plus, on many devices, the physical port labels are just a consecutive numeric string, so it would open the possibility to simply sort by "Label" and get everything in the "physical port order". (There currently seems to be a bug in the sorting for the "Label" column - 1, 10, and 11 are next to each other... I was searching to see if it had been reported when I turned up this discussion!)
