Xgboost binary format compatibility

In #1215, changes to the xgboost binary format break integration with the shap library and lead users to downgrade xgboost or create other hack solutions.

Xgboost makes no promises about the consistency of its binary format, except that it may save and load its own models, so this is likely to occur again in future xgboost releases.

I propose that shap migrates to the newer json based serialisation format that has better compatibility guarantees: https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html (cc the author of this feature @trivialfis)

After migration I propose adding shap tests to xgboost CI in order to prevent this from our end.

@RAMitchell I think this would be great. Currently, there are some new features that are not usable due to using the old binary format, for example categorical features (see #2662 ).

This seems even more important given that Xgboost plans to drop support from old binary format (see: here and here ).

It seems that SHAP used JSON before but it was reverted by @slundberg here due to conversion issues. Maybe the newer UBJSON format of Xgboost can help here?

Resolved via #3345