Feature request: YAML syntax verification problem

[dmontagner]

Hi Team,

is there a way we could detect the YAML problem seen on the test_no_diff_ERROR.yml ?

Below is the stack trace jsnapy presents. The problem with it is that it is not clear that there is a problem with the YAML syntax:

dmontagner@querencia ~/jnpr/jsnapy/testfiles> ../bin/jsnapy --check pre_snap1 post_snap1 -f config_check_ERROR.yml
*************************Performing test on Device: 172.27.45.132*************************
Tests Included: test_command_version
*************************Command is show interfaces terse lo**************************
----------------------Performing no-diff Test Operation----------------------
Test succeeded!! oper_status is same with value, before it is <['up']> now it is <['up']> with admin status <up>
Final result of no-diff: PASSED

Tests Included: test_command_bgp_neighbor
*************************Command is show bgp neighbor*************************
Traceback (most recent call last):
  File "../bin/jsnapy", line 9, in <module>
    load_entry_point('jsnapy==0.1', 'console_scripts', 'jsnapy')()
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/jsnapy.py", line 905, in main
    js.get_hosts()
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/jsnapy.py", line 296, in get_hosts
    self.login(output_file)
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/jsnapy.py", line 460, in login
    self.connect(hostname, username, password, output_file)
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/jsnapy.py", line 589, in connect
    action)
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/jsnapy.py", line 515, in get_test
    action)
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/jsnapy.py", line 351, in compare_tests
    post_snap_file)
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/check.py", line 469, in generate_test_files
    action)
  File "/Users/dmontagner/Documents/Diogo/JNPR/docs-jnpr/jsnapy/lib/python2.7/site-packages/jnpr/jsnapy/check.py", line 125, in compare_reply
    x_path = test.get('iterate').get('xpath', "no_xpath")
AttributeError: 'NoneType' object has no attribute 'get'
dmontagner@querencia ~/jnpr/jsnapy/testfiles>

Note: see the screenshots attached.

I wrote a simple python script to load an YAML file and print an error if there is any problem. Apparently, that syntax is perfectly fine with YAML but jsnapy is having problem with it.

Screenshots attached. YAML files attached.

jsnapy_yaml_issue.zip

[Jainpriyal]

@dmontagner: xpath, id, tests...all these fields in test file are values of key iterate/item and gets stored in form of list. All these fields require indentation and they should come after one tab of iteration/item.

I have modified the file and its working fine:

(venv)sh-3.2# cat test_no_diff_ERROR.yml 
tests_include:
  - test_command_version
  - test_command_bgp_neighbor

# use '/' in your test cases apart from xpath if u want to search all elements irrespective of hierarchy, ex: in id if u use /name instead of name
# then it will search in all the names in given xpath irrespective of their position
# for simple, one test using command

test_command_version:
  - command: show interfaces terse lo*
  - iterate:
      xpath: physical-interface
      id: './name'
      tests:
        - no-diff: oper-status       # element in which test is performed
          err: "Test Failed!! oper-status  got changed, before it was <{{pre['oper-status']}}>, now it is <{{post['oper-status']}}> with name <{{id_0}}> and admin status <{{post['admin-status']}}>"
          info: "Test succeeded!! oper-status is same with value, before it is <{{pre['oper-status']}}> now it is <{{post['oper-status']}}> with admin status <{{post['admin-status']}}> "

test_command_bgp_neighbor:
  - command: show bgp neighbor
  - iterate:
      xpath: bgp-peer
      id: './peer-address, ./peer-as'
      tests:
        - no-diff: peer-state
          err: "Test Failed!! The state of the BGP neighbor <{{id_0}}> (ASN: <{{id_1}}>) has changed. It was <{{pre['peer-state']}}>, now it is <{{post['peer-state']}}>"
          info: "Test succeeded!! The state of the BGP neighbor <{{id_0}}> (ASN: <{{id_1}}>) remains the same: <{{post['peer-state']}}>"
(venv)sh-3.2# 

Output:

jsnapy_file.zip

[dgarros]

@dmontagner, the issue is that your file is valid for YAML as YAML do not define how data are organized in your file If we had a schema we could detect the issue.

I created a library few months ago to solve this exact issue https://github.com/dgarros/defconf/ I'll try to see if it could help here to validate tests file structure.

[dmontagner]

@dgarros that is exactly what I was looking. While I knew where the problem was, I think we should have that kind of validation to alert the user.

We could add this as enhancement for the next version.

[vnitinv]

@dgarros @dmontagner Thanks for bringing this up. Will include the discussed library.

[astiphout]

I was also thinking about validation after I had a few cryptic errors after mal-formatted yaml; but you could also consider something the user could solve by running yamllint package as a preprocessor:

(jsnap)astiphou-mbp:~/jsnap> yamllint chassis.yml chassis.yml 1:1 warning missing document start "---" (document-start) 15:81 error line too long (117 > 80 characters) (line-length)

Users can provide their own configuration file with rules, for example if you don't care about line-length

[ydnath]

To have an option to validate the syntax of the test file. More than just checking the yaml.

Similar/close to ansible-playbook xxx.yml --syntax-check