vm: fix MODPOW operation

[AliceInHunterland]

Close #3612

[AnnaShaleva]

Also please, add one more compatibility test from https://github.com/neo-project/neo/pull/3557: (5 ^ -1) % 4 == 1

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.19%. Comparing base (b8a65d3) to head (b24144a). Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3649      +/-   ##
==========================================
- Coverage   83.26%   83.19%   -0.07%     
==========================================
  Files         334      334              
  Lines       46488    46494       +6     
==========================================
- Hits        38707    38681      -26     
- Misses       6205     6240      +35     
+ Partials     1576     1573       -3     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.