netmiko
netmiko copied to clipboard
ktbyers
Calling normalize_linefeeds on partial data results in additional empty lines
https://github.com/ktbyers/netmiko/blob/978ffbec0d5691d4b38ba6b0d455893d357591b7/netmiko/base_connection.py#L555-L559 calls normalize_linefeeds on partial data / "read chunks".
normalize_linefeeds converts remaining \r to \n: https://github.com/ktbyers/netmiko/blob/978ffbec0d5691d4b38ba6b0d455893d357591b7/netmiko/base_connection.py#L1828-L1841
If now one chunk ends in \r and the next chunk continues with \n, the concatenated result (send_command...) will contain \n\n, i.e. an additional empty line, instead of just \n.
The output in question contains legitimate empty lines, so I can't simply throw them away.
It would be nice to have a flag to disable this normalizing of incoming data completely.
(I also don't think I ever saw something using \n\r as (single) newline...)
@tik-stbuehler It looks like from the above that you could set self.RETURN = "\r\n" as part of the Netmiko ConnectHandler initialization and that would accomplish what you are looking for.
i.e. it would disable that last substitution so that span of two reads would not result in any substitutions happening.
Are you seeing examples of this occur in real data? I would be interested in the real example.
Some network devices do some odd things or at least they looked odd to me.
I wouldn't have added the \n\r in if I didn't come across some odd situation where the output was getting distorted (of course, I could have misinterpreted the situation or it could have been some strange side-effect of the normalization that I was doing). I seem to recall Cisco NX-OS having odd behavior here.
I am a bit reluctant to add a way to disable this as I am a bit worried about other Netmiko issues/side-effects that might spring up. Though I can definitely see the argument that doing this newline normalization was potentially a mistake in retrospect.
@tik-stbuehler It looks like from the above that you could set
self.RETURN = "\r\n"as part of the Netmiko ConnectHandler initialization and that would accomplish what you are looking for.
I guess you meant self.RESPONSE_RETURN, initialized from an optional response_return constructor argument? Hm, might work, although that is used in quite a few other places where it might break if the output doesn't contained the specified string (i.e. it is used to split lines, which probably should use something like re.compile(r'\r?\n') instead).
Are you seeing examples of this occur in real data? I would be interested in the real example.
Yes; running show running-config on IOS-XE + ASA and show file volatile:checkpoint_file on NXOS. The additional blank lines were a little bit random, but usually persistent on the same device for a few minutes when I repeated the command. I don't think I saw more than two additional lines in one output. I can't share the actual outputs though, sorry.
The blank lines I remember came either after an interface section ("just noise") or in the middle of an access list section. The latter breaks our (section-based) config diff, as an empty line terminates a section (I could probably ignore empty lines in section handling, but it feels wrong). (Also I really want a stable output, as those end up in git and so on.)
And we do have blank lines in the banner motd which we actually want.
Some network devices do some odd things or at least they looked odd to me.
I wouldn't have added the
\n\rin if I didn't come across some odd situation where the output was getting distorted (of course, I could have misinterpreted the situation or it could have been some strange side-effect of the normalization that I was doing). I seem to recall Cisco NX-OS having odd behavior here.
If I had to construct an input to trigger this I'd use the chunks '\r', '\n\r', '\n'. No idea what NXOS did though :)
I am a bit reluctant to add a way to disable this as I am a bit worried about other Netmiko issues/side-effects that might spring up. Though I can definitely see the argument that doing this newline normalization was potentially a mistake in retrospect.
I think the proper fix would be to run normalize_linefeeds only on the complete buffer; if you want to "partially" normalize (in case search patterns depend on it) I'd only substitute single \r if they are followed by something. (With some logic to handle this across chunks.. you probably don't want to run normalize on the full buffer each round. Something like "if new chunk starts with \r*\n and previous chunk ends in \r+, replace those with a single self.RESPONSE_RETURN")
Okay, let me look at it and thanks for the additional data.
Yeah, agreed I shouldn't be running this on the partial buffer (as it runs into exactly the type of probelms you describe). In other words, I should only be running it on the complete buffer.
I've ran into the same issue when using netmiko to backup Cisco IOS XR configs.
