Fix issues with Python3

[av-gantimurov]

Fix issue described in https://github.com/fox-it/mkYARA/issues/6#issue-1019668718

Rewrote coding data bytes to hex by binascii.hexlify(). Intended to make python2-3 compatible code without python version checks.

Fix ida plugin getting md5 file string. In python3 with ida 7.2+ GetInputFileMD5 returns bytes instead of string. Add checks for rtype.

Must be working on all python version (2.7+, 3.5+).