genieparser icon indicating copy to clipboard operation
genieparser copied to clipboard

Published 20 hours ago • verified publisher icon CiscoTestAutomation

fix issue #724: iosxe - ShowCryptoSession SchemaMissingKeyError

Open awk5303 opened this issue 2 years ago • 7 comments

Description

modified Schema and parsing capabilities to resolve python execption failures in the genie parser capabilities to deal with possible duplicate interface names in the "show crypto session" CLI

Motivation and Context

not able to use genie parse on production devices where the static IPSEC tunnels were in DOWN state. Traceback (most recent call last): File "src/genie/cli/commands/parser.py", line 339, in genie.cli.commands.parser.ParserCommand.parse File "src/genie/conf/base/device.py", line 531, in genie.conf.base.device.Device.parse File "src/genie/conf/base/device.py", line 570, in genie.conf.base.device.Device._get_parser_output File "src/genie/conf/base/device.py", line 568, in genie.conf.base.device.Device._get_parser_output File "src/genie/metaparser/_metaparser.py", line 342, in genie.metaparser._metaparser.MetaParser.parse File "src/genie/metaparser/_metaparser.py", line 322, in genie.metaparser._metaparser.MetaParser.parse File "src/genie/metaparser/util/schemaengine.py", line 419, in genie.metaparser.util.schemaengine.Schema.validate genie.metaparser.util.exceptions.SchemaMissingKeyError: Missing keys: [['interface', '(unknown)', 'peer', '192.168.1.1', 'port', '500', 'ipsec_flow']]

Impact (If any)

Schema was adapted as current code used a python dictionary with key values that were not guaranteed to be unique in the output. SuperParsesSchema was used other python test units too - adjusted those files too.

Screenshots:

Adapted existing unit and included test with output from duplicate tunnel in output. (22.11) 17:06:14-apieters@as8368-netdevops-vm:~/pyats/22.11/genieparser (master)$ genie parse "show crypto session detail" --testbed-file /home/apieters/pyats/SRC/INPUT/TOPOLOGY/UAT/cml_testbed.yaml --device virl.aal1-dmvpn 0%| | 0/1 [00:00<?, ?it/s]{ "interface": { "1": { "interface": "Tunnel3111", "peer": { "192.168.1.1": { "port": { "500": { "desc": "none", "fvrf": "none", "ipsec_flow": { "permit ip 100.75.0.0/255.255.255.192 192.168.25.0/255.255.255.0": { "active_sas": 0, "inbound_life_kb": "0", "inbound_life_secs": "0", "inbound_pkts_decrypted": 0, "inbound_pkts_drop": 0, "origin": "crypto map", "outbound_life_kb": "0", "outbound_life_secs": "0", "outbound_pkts_drop": 0, "outbound_pkts_encrypted": 0 }, "permit ip 100.75.0.0/255.255.255.192 192.168.25.0/255.255.255.128": { "active_sas": 0, "inbound_life_kb": "0", "inbound_life_secs": "0", "inbound_pkts_decrypted": 0, "inbound_pkts_drop": 0, "origin": "crypto map", "outbound_life_kb": "0", "outbound_life_secs": "0", "outbound_pkts_drop": 0, "outbound_pkts_encrypted": 0 }, "permit ip 100.75.0.0/255.255.255.192 192.168.26.0/255.255.255.0": { "active_sas": 0, "inbound_life_kb": "0", "inbound_life_secs": "0", "inbound_pkts_decrypted": 0, "inbound_pkts_drop": 0, "origin": "crypto map", "outbound_life_kb": "0", "outbound_life_secs": "0", "outbound_pkts_drop": 0, "outbound_pkts_encrypted": 0 }, "permit ip host 100.74.10.1 192.168.25.0/255.255.255.0": { "active_sas": 0, "inbound_life_kb": "0", "inbound_life_secs": "0", "inbound_pkts_decrypted": 0, "inbound_pkts_drop": 0, "origin": "crypto map", "outbound_life_kb": "0", "outbound_life_secs": "0", "outbound_pkts_drop": 0, "outbound_pkts_encrypted": 0 }, "permit ip host 100.74.10.1 192.168.25.0/255.255.255.128": { "active_sas": 0, "inbound_life_kb": "0", "inbound_life_secs": "0", "inbound_pkts_decrypted": 0, "inbound_pkts_drop": 0, "origin": "crypto map", "outbound_life_kb": "0", "outbound_life_secs": "0", "outbound_pkts_drop": 0, "outbound_pkts_encrypted": 0 }, "permit ip host 100.74.10.1 192.168.26.0/255.255.255.0": { "active_sas": 0, "inbound_life_kb": "0", "inbound_life_secs": "0", "inbound_pkts_decrypted": 0, "inbound_pkts_drop": 0, "origin": "crypto map", "outbound_life_kb": "0", "outbound_life_secs": "0", "outbound_pkts_drop": 0, "outbound_pkts_encrypted": 0 } }, "ivrf": "inner", "phase1_id": "(none)" } } } }, "session_status": "DOWN" }, "2": { "interface": "(unknown)", "peer": { "192.168.1.1": { "port": { "500": { "desc": "none", "fvrf": "none", "ike_sa": { "1": { "capabilities": "none", "conn_id": "0", "lifetime": "0", "local": "94.140.184.80", "local_port": "500", "remote": "192.168.1.1", "remote_port": "500", "sa_status": "Inactive", "session_id": "0", "version": "IKEv1" } }, "ivrf": "none", "phase1_id": "(none)" } } } }, "profile": "ISAKMP-inner-LOC_A", "session_status": "DOWN-NEGOTIATING" } } }

due to impact on other python unit tests, adapted these expected output file also for these: ShowCryptoSessionDetail: ShowCryptoSessionInterfaceDetail: ShowCryptoSessionLocal: ShowCryptoSessionLocalDetail:

(22.11) 16:05:57-apieters@as8368-netdevops-vm:~/pyats/22.11/genieparser/tests (master)$ python folder_parsing_job.py -o iosxe -c ShowCryptoSession 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: | Starting testcase SuperFileBasedTesting | 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: | Starting section setup | 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: The result of section setup is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: The result of testcase SuperFileBasedTesting is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: | Starting testcase iosxe | 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: | Starting section setup | 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: The result of section setup is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: | Starting section ShowCryptoSession | 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1: iosxe -> ShowCryptoSession : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.1: Test Golden -> iosxe -> ShowCryptoSession : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.1.1: Gold -> iosxe -> ShowCryptoSession -> golden_outp : 2023-01-20T16:06:04: %AETEST-INFO: : ut1 : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.1.1: Gold -> iosxe -> ShowCryptoSession -> golden_output1 is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.1.2: Gold -> iosxe -> ShowCryptoSession -> golden_outp : 2023-01-20T16:06:04: %AETEST-INFO: : ut2 : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.1.2: Gold -> iosxe -> ShowCryptoSession -> golden_output2 is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.1.3: Gold -> iosxe -> ShowCryptoSession -> golden_outp : 2023-01-20T16:06:04: %AETEST-INFO: : ut3 : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.1.3: Gold -> iosxe -> ShowCryptoSession -> golden_output3 is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.1.4: Gold -> iosxe -> ShowCryptoSession -> golden_outp : 2023-01-20T16:06:04: %AETEST-INFO: : ut : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.1.4: Gold -> iosxe -> ShowCryptoSession -> golden_output is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.1: Test Golden -> iosxe -> ShowCryptoSession is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.2: Test Empty -> iosxe -> ShowCryptoSession : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : Starting STEP 1.2.1: Empty -> iosxe -> ShowCryptoSession -> empty_outp : 2023-01-20T16:06:04: %AETEST-INFO: : ut : 2023-01-20T16:06:04: %AETEST-INFO: +..............................................................................+ 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.2.1: Empty -> iosxe -> ShowCryptoSession -> empty_output is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1.2: Test Empty -> iosxe -> ShowCryptoSession is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: The result of STEP 1: iosxe -> ShowCryptoSession is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +..........................................................+ 2023-01-20T16:06:04: %AETEST-INFO: : STEPS Report : 2023-01-20T16:06:04: %AETEST-INFO: +..........................................................+ 2023-01-20T16:06:04: %AETEST-INFO: STEP 1 - iosxe -> ShowCryptoSession Passed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.1 - Test Golden -> iosxe -> ShowCryptoSessionPassed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.1.1 - Gold -> iosxe -> ShowCryptoSession -> golden_output1Passed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.1.2 - Gold -> iosxe -> ShowCryptoSession -> golden_output2Passed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.1.3 - Gold -> iosxe -> ShowCryptoSession -> golden_output3Passed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.1.4 - Gold -> iosxe -> ShowCryptoSession -> golden_outputPassed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.2 - Test Empty -> iosxe -> ShowCryptoSessionPassed
2023-01-20T16:06:04: %AETEST-INFO: STEP 1.2.1 - Empty -> iosxe -> ShowCryptoSession -> empty_outputPassed
2023-01-20T16:06:04: %AETEST-INFO: ............................................................ 2023-01-20T16:06:04: %AETEST-INFO: The result of section ShowCryptoSession is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: | Starting section cleanup | 2023-01-20T16:06:04: %AETEST-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %AETEST-INFO: The result of section cleanup is => PASSED 2023-01-20T16:06:04: %AETEST-INFO: The result of testcase iosxe is => PASSED 2023-01-20T16:06:04: %GENIE-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %GENIE-INFO: | Unittest results | 2023-01-20T16:06:04: %GENIE-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %GENIE-INFO: SECTIONS/TESTCASES RESULT
2023-01-20T16:06:04: %GENIE-INFO: -------------------------------------------------------------------------------- 2023-01-20T16:06:04: %GENIE-INFO: . 2023-01-20T16:06:04: %GENIE-INFO: |-- SuperFileBasedTesting PASSED 2023-01-20T16:06:04: %GENIE-INFO: | -- setup PASSED 2023-01-20T16:06:04: %GENIE-INFO: -- iosxe PASSED 2023-01-20T16:06:04: %GENIE-INFO: |-- setup PASSED 2023-01-20T16:06:04: %GENIE-INFO: |-- ShowCryptoSession PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1: iosxe -> ShowCryptoSession PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1.1: Test Golden -> iosxe -> ShowCryptoSession PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1.1.1: Gold -> iosxe -> ShowCryptoSession -> golde... PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1.1.2: Gold -> iosxe -> ShowCryptoSession -> golde... PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1.1.3: Gold -> iosxe -> ShowCryptoSession -> golde... PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1.1.4: Gold -> iosxe -> ShowCryptoSession -> golde... PASSED 2023-01-20T16:06:04: %GENIE-INFO: | |-- Step 1.2: Test Empty -> iosxe -> ShowCryptoSession PASSED 2023-01-20T16:06:04: %GENIE-INFO: | -- Step 1.2.1: Empty -> iosxe -> ShowCryptoSession -> empt... PASSED 2023-01-20T16:06:04: %GENIE-INFO: -- cleanup PASSED 2023-01-20T16:06:04: %GENIE-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %GENIE-INFO: | Summary | 2023-01-20T16:06:04: %GENIE-INFO: +------------------------------------------------------------------------------+ 2023-01-20T16:06:04: %GENIE-INFO: Number of ABORTED 0 2023-01-20T16:06:04: %GENIE-INFO: Number of BLOCKED 0 2023-01-20T16:06:04: %GENIE-INFO: Number of ERRORED 0 2023-01-20T16:06:04: %GENIE-INFO: Number of FAILED 0 2023-01-20T16:06:04: %GENIE-INFO: Number of PASSED 2 2023-01-20T16:06:04: %GENIE-INFO: Number of PASSX 0 2023-01-20T16:06:04: %GENIE-INFO: Number of SKIPPED 0 2023-01-20T16:06:04: %GENIE-INFO: Total Number 2 2023-01-20T16:06:04: %GENIE-INFO: Success Rate 100.0% 2023-01-20T16:06:04: %GENIE-INFO: -------------------------------------------------------------------------------- 2023-01-20T16:06:04: %GENIE-INFO: Total Passing Unittests 5 2023-01-20T16:06:04: %GENIE-INFO: Total Failed Unittests 0 2023-01-20T16:06:04: %GENIE-INFO: Total Errored Unittests 0 2023-01-20T16:06:04: %GENIE-INFO: Total Unittests 5 2023-01-20T16:06:04: %GENIE-INFO: --------------------------------------------------------------------------------

Checklist:

  • [ x ] I have updated the changelog.
  • [ ] I have updated the documentation (If applicable).
  • [ x ] I have added tests to cover my changes (If applicable).
  • [ x ] All new and existing tests passed.
  • [ x ] All new code passed compilation.

awk5303 avatar Jan 20 '23 16:01 awk5303

please fix the version mismtach so pipeline could pass

omehrabi avatar Feb 24 '23 17:02 omehrabi

please fix the version mismtach so pipeline could pass

@omehrabi Is this a request for me?

awk5303 avatar Feb 27 '23 14:02 awk5303

@awk5303, yes.

Taarini avatar Feb 27 '23 17:02 Taarini

@omehrabi Can you please provide your feedback on what you think about this improvement. The current parser is buggy as in it crashes when two interfaces with same name are present. But fixing it means breaking the current implementation.

sjpatel21 avatar Mar 01 '23 19:03 sjpatel21

@awk5303, yes.

I just have been testing version 23.02 and the python exception SchemaMissingKeyError is not showing anymore (without applying my fix).

awk5303 avatar Mar 22 '23 14:03 awk5303

My wrong - pushed my code based on v22.11 still.

awk5303 avatar Mar 23 '23 07:03 awk5303

@omehrabi any updates on this?

awk5303 avatar May 24 '23 13:05 awk5303