JIT Compatibility

[Y-Less]

I have always wanted to make YSI compatible with the JIT plugin (I found a post mentioning that from 2012) the other day. The level of compatibility has fluctuated, the current major blocker is y_inline, since that used run-time information to determine stack sizes at inline function call sites. They are initialised lazilly when first encountered and so could not be done in advance. The older version of y_inline was slightly better in that regard as that used basic initialisation AMX scanning to find all inline functions and rewrite them. That could be done in advance, but as a result didn't support closures. The first step to getting YSI JIT compatible was to solve this issue to support both features - ahead-of-time code generation and stack size analysis, which was done by creating codescan.inc.

Dependencies:

• [x] JIT Plugin. This needed a few tweaks itself for corner cases and dynamic function calling. Got a custom version running already, with some up-stream pushes made. Need to get Zeex to make a new release.
• [x] amx_assembly. Also needs to be made compatible. I've done most of this; no idea how to make the CallNative family of functions work, but I also don't use them, so it is good enough for YSI.
• [x] code-parse.inc. No updates needed, I just need to make it a full YSI dependency.
• [x] Better releasing. With an extra external dependency, "download these multiple things" is no longer an adequate distribution model (with that, I might start adding more external dependencies, like md-sort.

Libraries

• [ ] YSI_Coding
  • [x] y_hooks.inc
  • [x] y_inline.inc
  • [x] y_malloc.inc
  • [x] y_remote.inc
  • [x] y_stringhash.inc
  • [ ] y_timers.inc
  • [x] y_va.inc
• [ ] YSI_Core
  • [x] y_als.inc
  • [x] y_cell.inc
  • [x] y_debug.inc
  • [ ] y_functional.inc
  • [x] y_master.inc
  • [x] y_profiling.inc
  • [x] y_testing.inc
  • [ ] y_utils.inc
• [ ] YSI_Data
  • [x] y_bintree.inc
  • [x] y_bit.inc
  • [x] y_foreach.inc
  • [x] y_hashmap.inc
  • [x] y_iterate.inc
  • [x] y_jaggedarray.inc
  • [ ] y_playerarray.inc
  • [ ] y_playerset.inc
  • [ ] y_simpletree.inc
• [ ] YSI_Extra
  • [ ] y_extra.inc
  • [ ] y_extra_languages.inc
  • [ ] y_extra_other.inc
  • [ ] y_extra_users.inc
  • [ ] y_files.inc
• [ ] YSI_Game
  • [ ] y_vehicledata.inc
• [ ] YSI_Internal
  • [x] y_cgen.inc
  • [ ] y_classgroups.inc
  • [x] y_compilerpass.inc
  • [ ] y_distribute.inc
  • [x] y_funcinc.inc
  • [ ] y_gamerzps.inc
  • [x] y_globaltags.inc
  • [ ] y_incognitostreamer.inc
  • [ ] y_mock.inc
  • [x] y_natives.inc
  • [ ] y_plugins.inc
  • [ ] y_pp.inc
  • [x] y_renative.inc
  • [x] y_shortfunc.inc
  • [x] y_shortvar.inc
  • [ ] y_stripnumbers.inc
  • [x] y_unique.inc
  • [x] y_unqid.inc
  • [x] y_version.inc
  • [ ] y_writemem.inc
• [ ] YSI_Players
  • [ ] y_groups.inc
  • [ ] y_languages.inc
  • [ ] y_text.inc
  • [ ] y_users.inc
• [ ] YSI_Server
  • [ ] y_colors.inc
  • [ ] y_colours.inc
  • [ ] y_flooding.inc
  • [x] y_lock.inc
  • [ ] y_punycode.inc
  • [x] y_scriptinit.inc
  • [ ] y_td.inc
• [ ] YSI_Storage
  • [x] y_amx.inc
  • [ ] y_bini.inc
  • [ ] y_bitmap.inc
  • [ ] y_ini.inc
  • [x] y_php.inc
  • [ ] y_svar.inc
  • [ ] y_uvar.inc
  • [ ] y_xml.inc
• [ ] YSI_Visual
  • [ ] y_areas.inc
  • [ ] y_classes.inc
  • [x] y_commands.inc
  • [ ] y_dialog.inc
  • [ ] y_loader.inc
  • [ ] y_objects.inc
  • [ ] y_properties.inc
  • [ ] y_races.inc
  • [x] y_zonenames.inc
  • [x] y_zonepulse.inc

[Y-Less]

A large number of these should just work since they have no assembly in, but I just need to run through them all and check.

[Y-Less]

Just done y_hooks - first of the major chunks, though surprisingly easy compared to what I thought it would be.

Also just noticed that the P:Is in Test:y_testing_2 don't display - I don't know why that is, maybe an issue with states.

[Southclaws]

I've always wanted to try JIT in my SS project so I'm sure I can help with testing/logs. I'm no assembly/low-level expert but I'm up for testing even if it's just by getting YSI+JIT into a live project!

[Y-Less]

Nice, thanks.

[Y-Less]

OK, slight problem. I have determined that the server gets the index of the OnGameModeInit before calling OnJITCompile, but before calling either. Then y_hooks shuffles all the publics in the header, meaning that a different public is in that slot index already determined, and so that one gets called instead of OnGameModeInit. I'm not sure if the same happens for filterscripts yet. Not entirely sure how to fix this one...

Edit: Now solved.

[Y-Less]

y_malloc allocated data on the heap using a bug in the PAWN VM to overwrite internal values and essentially "hide" the data from the rest of the system (so it would think that the heap started higher up than it really did, with y_malloc using the lower part). I've not looked, but I doubt the same exploitable bug will exist in the JIT. Fortunately, I had already re-added the old version based on a massive global array behind a compiler flag, so now if you use the JIT you MUST use YSI_NO_HEAP_MALLOC. It is no slower (except on startup) - literally the only difference was were the data was located, and how big the .AMX file was (because global data is in there statically, while heap data is allocated at server startup).

[Y-Less]

I think I have now done all the parts except the big one - y_inline (plus some minor libraries, but they shouldn't take much work, if any). I even ported y_master to code-parse.inc, which made it simpler; and wrote some new small functions, which made it faster.

[Y-Less]

As of right now, I have completed the main port of y_inline, and got all existing tests passing! However, I've not written any new tests and I've not actually tried it on the JIT yet, which was the whole point of the excercise. It does only use JIT-compatible code though.

Notes:

How to convert from old y_inline to new y_inline:

1. Delete all [E_CALLBACK_DATA] everywhere. This can optionally be replaced by a specifier tag. So what used to be:

   new cb[E_CALLBACK_DATA];

   Becomes:

   new Func:cb<iiis>;

2. Replace Callback_Get and Callback_Release with Indirect_Claim and Indirect_Release. But only if you plan to store the callback for use in the future. If you plan to use it immediately, just delete them.

3. Use Callback_Find if you want to search for a callback by name. This means the old behviour of Callback_Call is now split between two functions, both of which are optional depending on what you want to do.

4. Replace callback: tags with specifier tags. So:

   MyFunc(callback:x)

   Becomes:

   MyFunc(Func:cb<iiis>)

5. Replace:

   Callback_Call(cb, your, params);

   With:

   @.cb(your, params);

Note that this is all optional. Not doing so will continue to work, but give warnings. These two are NOT optional - they were the only breaking changes, but were quite obscure features:

6. If you used plain strings as:

   MyFunc(callback_tag:"func");

   This has been entirely removed and must be:

   MyFunc(using func);

   Note the lack of inline or callback. This was the syntax for plain string searches, but plain strings were also possible. Now they are not.

7. If you used an inline that was not in scope as:

    {
        inline Func() {}
    }
    {
        Other(using Func);
    }

You now can't do that - it won't work. The old code was not fully aware of scopes and would attempt to find the nearest potential function. The new code is fully scope aware and will correct deal with it.

[Zelyutin]

Hello!

y_hooks is marked in your list, but YSI_core/y_utils isn't marked although it's used in y_hooks. So, can I use hooks with JIT or not?

[Y-Less]

The ticks are a little misleading. The ticked libraries have been rewritten with mostly JIT compatible code, but not fully tested and debugged. And y_utils is generic functions, so parts can be ported without it all working yet.

[Zelyutin]

Got it, thank you! Will be testing this thing and trying to join everything up, because hooks are brilliant. I want to use JIT, but if I choose between JIT and hooks, I will choose hooks cause I can't build the code without them already. It's very useful.