apigee
Henchman sometimes fails with "unexpected end of JSON input" Error
02:06:39 echo '{"cmd":"yum install -y edge-router --enablerepo=epel --nogpgcheck","loglevel":"debug"}' | /bin/bash -c 'sudo -H -u root ${HOME}/.henchman/shell'
02:06:53 unexpected end of JSON input
Not stacktrace /error log found!
Actually still fixing the latest release, something is up with the copy module
00:04:17.471
00:04:29.129 time="2015-12-03T02:27:50Z" level=error msg="Error running task 'Install Apigee RPM'" error="While in exec_module :: While unmarshalling task results :: unexpected end of JSON input" host=10.17.6.32 plan="Install Router" task="Install Apigee RPM"
The issue at hand is some of the data is being dropped from ssh.Exec. So if the output should return {"status": "changed", "msg": "hello world"} sometimes the output will be {"status": "changed", "msg": "hello. The second output will result in the JSON input error because it does not follow proper JSON. So the temp fix to this is to slap on "} to the end. This will result in incomplete outputs for the time being. The user will be notified when this occurs though.
Notes:
- I have tried removing concurrency in plan.go
- Used ssh.go's methods in multiple ways including using stdoutPipe and what not
This could be an underlying issue with ssh.go
Any help on this? @madhurranjan / @sudharsh . This keeps happening and coulndt proceed with other tasks on playbook unless i have "ignore_errors: true" for that task.
@baskaran-md Does temp fix prevent henchman from throwing "End of JSON Input error?". It should. If it does happen let me know please. We're still trying to find the root cause of that issues
Have removed the explicit appending of '}' if this occurs as more log output is needed
Here's what we know so far to be able reproduce this (1 in 3-5 times)
1.) The target node has to be, * amazon-linux No problems so far in others including centos 7.x, centos 6.5 and rhel 7.2 2.) Python version is 2.7.10 3.) There should be atleast one call to the Popen() constructor. Doesn't matter if this invocation is being used or is in the critical path. The moment a call to Popen() is made, things become unreliable.