PokemonGoAPI-PHP
PokemonGoAPI-PHP copied to clipboard
NicklasWallgren
Signature aka Unknown6
Since the unknown6 field has been reverse engineered, did you start looking into implementing it in this API? My main concern is the encrypt.so/.dll that is used. I think we can't use it directly since it's not a PHP shared library. We would have to write a wrapper or port the function to PHP code. The latter would have copyright issues. Any thoughts?
Using a DLL file in PHP directly is a new one, that's for sure. Additionally, the python scripts are using xxHash, which is another dependency that we might have to deal with.
On a slightly related note, we should consider "re-merging" all the pogoprotos-php repositories to a common place, so there's not duplicate work done with so few people.
Regarding @Ni42's note here, merging them would be good, but that would (again) raise a discussion (jaspervdm/pogoprotos-php#8) whether bramp/protoc-gen-php or protobuf-php should be used. I think the main reason why everyone's creating his own fork is that they're fixing bugs with bramp's one :laughing: . The only problem that came up with protobuf-php so far was fixed within a day by the original dev after mentioning it (protobuf-php/protobuf-plugin#3). Of course by now, where most projects are quite advanced, switching to a new Protobuf-lib is some work to adjust (to be honest, though, the changes are not that big, mostly just other class names and passing streams in constructor instead of a read-method).
I don't have enough insight in the work required, but just from what I'm reading, would you rather keep fixing things in an inactive/abandoned library or do the work of switching to a maintained one? It's an easy decision I guess, especially if you have to maintain these files anyways to to game API updates (or the current unknown6 situation).
(We are getting a little off topic here)
This is what I had to change to get the RetrievePlayerProfile-Example working: https://github.com/NicklasWallgren/PokemonGoAPI-PHP/compare/master...DrDelay:patch5-protobuf-php
Disclaimer :laughing: : This is not properly tested. Probably adjustments to the AuthTicket saving need to be made (resetting streams). Every other example will probably fail as I didn't even run it once and only fixed until RetrievePlayerProfile worked. This is only a proof-of-concept.
But as you can see, it is not overwhelmingly much.
For sure protobuf-php seems the way to go, especially since that repo is still maintained. Ideally everyone that wants to use the protos for their API or whatever uses the same repo for the PHP classes to avoid doing double work. On Wednesday I am back at home. At that point I will test the test-protobuf branch some more (especially using same auth ticket multiple times as DrDelay already referenced) and merge it into master.
Regarding the actual topic of this thread: under Windows PHP extensions are DLLs. I was thinking if we could somehow use the encrypt.so/.dll as an extension. But like I said, we cannot use it directly since it's not written for PHP. I don't have experience writing extensions, but I am starting to look into it. I'm not sure if it's possible but the best option imo would be to write something that uses the encrypt.so/.dll as an external, so the extension itself wouldn't contain copyrighted material and we can put it on github.
I updated my branch with 67344129bd58ab92aa374b573e0ac0bf24a696cc, because I now encountered the "AuthTicket stream thing" (Exact error is PHP Fatal error: Uncaught RuntimeException: Failed to write stream with 64 bytes in .\vendor\protobuf-php\protobuf\src\Stream.php:238) mentioned. This fix works, however I'm not sure whether this should be handled in the pogoprotos dependency. Ref
I'm all for switching to protobuf-php, eventually. We should get a separate branch going.
So I tried around a little bit, and it is not too hard writing a PHP extension. I successfully compiled a PHP extension on my windows install that can find an external DLL and call a function from it... I can share my pathetic VC2015 file if you guys want it.
That basically means welcome to platform-specific PHP programming, because using a DLL using the Win32 API obviously doesn't work un Unix ^^
Anybody else got some progress?
I tried my luck with The COM class, no success however: Uncaught com_exception: Failed to create COM object 'encrypt.dll': Ungültige Dateierweiterung (Means "Invalid file extension")
Another thought I had: I saw a repo with the source (encrypt.c and encrypt.h) of this .dll/.so, I think it was this, but it says 404 now.
The thought was: Why do we need a .dll/.so? Wouldn't this do (sorry for mistakes, my C is a bit rusty):
/* pogoencrypt.c */
#include <stdio.h>
#include "unknown6/encrypt.h"
int main(int argc, char** argv)
{
printf("%s", encrypt(argv[1]));
return 0;
}
The repo would be included as a submodule unknown6. We could then just call pogoencrypt(.exe) with shell_exec.
Are the copyright issues with this (if we ever find that repo again).
You can only use the COM library for DLLs that are specially compiled, I may or may not have the C file, so we could try some things out.
The main problem was mentioned by @jaspervdm above, we can't really directly redistribute the encrypt DLL (or a recompiled version) without also going into copyright territory... That only really leaves us with the option of creating a way to "externally load" the DLL/SO, which has to be in the form of a PHP extension or external executable file.
Looks like there's a pure JS implementation of encrypt.c: https://www.reddit.com/r/pokemongodev/comments/4wo75h/pure_javascript_typescript_port_of_encryptc/
So I guess that would also be possible in PHP? We could include the "encrypt.php" from some external source then.
Might work too, would be easier than dealing with extensions/library files, two things come to mind: copyright (I have really no idea how this really works), and performance.
I've handled the xxhash dependency
https://github.com/MatthewKingDev/php-xxhash
tested vs the python implementation and the values are returning correctly so that's one less headache.
@MatthewKingDev Whats the difference between the original onces?
https://github.com/Megasaxon/php-xxhash https://github.com/dred86/php-xxhash
They both supports PHP 7
The difference between dred86 and Megasaxon is https://github.com/dred86/php-xxhash/commit/29c79570f44126b99d2d645263f732faad5f7f08 changing to a newer version of xxhash
I've had a go at converting the pure JS implementation to PHP tonight, feel I'm pretty close though I've run into a bit of a wall. My lack of understanding of both C / JS TypeArrays doesn't help. Can any of you guys point me in the right direction, I'm not understanding how the below is happening and how to recreate that in PHP?
The JS CODE
//output and slice arrays are below
var slice = new Uint32Array(output, offset); //offset is 32 in my example
seems to convert in the following way.
var input = Uint8Array {
'0': 0,
'1': 4,
'2': 101,
'3': 102,
'4': 103,
'5': 104,
'6': 105,
'7': 106,
'8': 107,
'9': 108,
'10': 109,
'11': 110,
'12': 111,
'13': 112,
'14': 113,
'15': 114,
'16': 115,
'17': 116,
'18': 117,
'19': 118,
'20': 119,
'21': 120,
'22': 121,
'23': 122,
'24': 97,
'25': 98,
'26': 99,
'27': 100,
'28': 101,
'29': 102,
'30': 0,
'31': 0,
'32': 0,
'33': 5,
'34': 17,
'35': 102,
'36': 2,
'37': 104,
'38': 26,
'39': 106,
'40': 31,
'41': 108,
'42': 109,
'43': 110,
'44': 111,
'45': 112,
'46': 113,
'47': 114,
'48': 115,
'49': 116,
'50': 117,
'51': 118,
'52': 119,
'53': 120,
'54': 121,
'55': 122,
'56': 97,
'57': 98,
'58': 99,
'59': 100,
'60': 101,
'61': 102,
'62': 0,
'63': 0,
'64': 0,
'65': 8,
'66': 202,
'67': 204,
'68': 206,
'69': 208,
'70': 210,
'71': 212,
'72': 214,
'73': 216,
'74': 218,
'75': 220,
'76': 222,
'77': 224,
'78': 226,
'79': 228,
'80': 230,
'81': 232,
'82': 234,
'83': 236,
'84': 238,
'85': 240,
'86': 242,
'87': 244,
'88': 194,
'89': 196,
'90': 198,
'91': 200,
'92': 202,
'93': 204,
'94': 0,
'95': 0,
'96': 0,
'97': 16,
'98': 149,
'99': 153,
'100': 157,
'101': 161,
'102': 165,
'103': 169,
'104': 173,
'105': 177,
'106': 181,
'107': 185,
'108': 189,
'109': 193,
'110': 197,
'111': 201,
'112': 205,
'113': 209,
'114': 213,
'115': 217,
'116': 221,
'117': 225,
'118': 229,
'119': 233,
'120': 133,
'121': 137,
'122': 141,
'123': 145,
'124': 149,
'125': 153,
'126': 0,
'127': 0,
'128': 0,
'129': 32,
'130': 43,
'131': 51,
'132': 59,
'133': 67,
'134': 75,
'135': 83,
'136': 91,
'137': 99,
'138': 107,
'139': 115,
'140': 123,
'141': 131,
'142': 139,
'143': 147,
'144': 155,
'145': 163,
'146': 171,
'147': 179,
'148': 187,
'149': 195,
'150': 203,
'151': 211,
'152': 11,
'153': 19,
'154': 27,
'155': 35,
'156': 43,
'157': 51,
'158': 0,
'159': 0,
'160': 0,
'161': 64,
'162': 86,
'163': 102,
'164': 118,
'165': 134,
'166': 150,
'167': 166,
'168': 182,
'169': 198,
'170': 214,
'171': 230,
'172': 246,
'173': 7,
'174': 23,
'175': 39,
'176': 55,
'177': 71,
'178': 87,
'179': 103,
'180': 119,
'181': 135,
'182': 151,
'183': 167,
'184': 22,
'185': 38,
'186': 54,
'187': 70,
'188': 86,
'189': 102,
'190': 0,
'191': 0,
'192': 0,
'193': 128,
'194': 172,
'195': 204,
'196': 236,
'197': 13,
'198': 45,
'199': 77,
'200': 109,
'201': 141,
'202': 173,
'203': 205,
'204': 237,
'205': 14,
'206': 46,
'207': 78,
'208': 110,
'209': 142,
'210': 174,
'211': 206,
'212': 238,
'213': 15,
'214': 47,
'215': 79,
'216': 44,
'217': 76,
'218': 108,
'219': 140,
'220': 172,
'221': 204,
'222': 0,
'223': 0,
'224': 0,
'225': 1,
'226': 89,
'227': 153,
'228': 217,
'229': 26,
'230': 90,
'231': 154,
'232': 218,
'233': 27,
'234': 91,
'235': 155,
'236': 219,
'237': 28,
'238': 92,
'239': 156,
'240': 220,
'241': 29,
'242': 93,
'243': 157,
'244': 221,
'245': 30,
'246': 94,
'247': 158,
'248': 88,
'249': 152,
'250': 216,
'251': 25,
'252': 89,
'253': 153,
'254': 0,
'255': 0,
'256': 0,
'257': 2,
'258': 178,
'259': 51,
'260': 179,
'261': 52,
'262': 180,
'263': 53,
'264': 181,
'265': 54,
'266': 182,
'267': 55,
'268': 183,
'269': 56,
'270': 184,
'271': 57,
'272': 185,
'273': 58,
'274': 186,
'275': 59,
'276': 187,
'277': 60,
'278': 188,
'279': 61,
'280': 176,
'281': 49,
'282': 177,
'283': 50,
'284': 178,
'285': 51,
'286': 0,
'287': 246 }
into
var slice = Uint32Array {
'0': 1712391424,
'1': 1780115458,
'2': 1852664863,
'3': 1920036975,
'4': 1987409011,
'5': 2054781047,
'6': 1684234849,
'7': 26213,
'8': 3435792384,
'9': 3570585806,
'10': 3705329878,
'11': 3840073950,
'12': 3974818022,
'13': 4109562094,
'14': 3368469698,
'15': 52426,
'16': 2576683008,
'17': 2846204317,
'18': 3115692461,
'19': 3385180605,
'20': 3654668749,
'21': 3924156893,
'22': 2441972101,
'23': 39317,
'24': 858464256,
'25': 1397441339,
'26': 1936417627,
'27': 2475393915,
'28': 3014370203,
'29': 3553346491,
'30': 588976907,
'31': 13099,
'32': 1716928512,
'33': 2794882678,
'34': 3872835254,
'35': 655820790,
'36': 1733773111,
'37': 2811725687,
'38': 1177953814,
'39': 26198,
'40': 3433857024,
'41': 1294798316,
'42': 3450703213,
'43': 1311641325,
'44': 3467546222,
'45': 1328484334,
'46': 2355907628,
'47': 52396,
'48': 2572747008,
'49': 2589596377,
'50': 2606439386,
'51': 2623282395,
'52': 2640125404,
'53': 2656968413,
'54': 433625176,
'55': 39257,
'56': 867303936,
'57': 901002419,
'58': 934688437,
'59': 968374455,
'60': 1002060473,
'61': 1035746491,
'62': 850473392,
'63': 4127208370 }
Hmm, what exactly is the problem? You don't understand the type conversion? That's a pretty huge list, can you maybe create a smaller example, or is the size relevant?
Check out:
https://github.com/Sjaakmoes/pokapi/blob/master/src/Pokapi/Rpc/Service.php
At the bottom it looks like they have this done already.
Maybe I'm missing it, but I don't see anything relating to the signature field. Which line are you referring to?
Can't really be it, since the changes for the signature are only 4 days old now, and there weren't many things updated on that project since then.
@Ni42 Do you have discord, so we could discuss the encrypt library? I am on the pokemongodev discord server, username kebabtent#5947
@KyleBoyer I didn't add the signature to that yet, planning to do that later still.
@Sjaakmoes I don't know if that will be practical, it may be better/faster to get up and running by simply including the encrypt files for windows/Linux/Mac (32/64bit) and have an ID statement decide which one to use(being, somewhat OS independent). Once that has been done, porting the encrypt library to PHP would be best.
Just my 2 cents: I would prefer if this library had nothing related to the Unknown 6. I mean there's tons of other libraries that got this available ( not PHP ) and using Map resources in this API would increase the chance of C&D by Niantic. I like it here, I don't want it gone :(
IMO any encrypt implementation/wrapper should be a separate repo anyway, since more than one API project could profit from it. In the same way, the proto classes should also be a separate repo.
There is now a pure PHP implementation available: https://github.com/POGO-PHP/POGOEncrypt-PHP
Now we still need the xxHash extension, do we? Also who wants to do the pull request? I might do it later today if nobody volunteers.
User @MatthewKingDev made one: https://github.com/MatthewKingDev/php-xxhash
@Ni42 : I could add a pull request to update composer.json, however that alone is not enough to generate valid requests
@POGO-PHP right, that wouldn't be the problem, it needs to be implemented on our RequestEnvelope... on that context, @NicklasWallgren can you upgrade the proto files? ;)
Thanks!
@Sjaakmoes I saw you already have the signature implemented in your version, are you planning to add a PR here? If not, would you mind if someone took your implementation and transferred it here?
@Ni42 By all means go ahead, I'm thinking about creating a PR for this. I wanted to ask, before I do that, @NicklasWallgren . My current implementation requires a PHP extension for xxhash, is that a problem? It's also PHP7+ only because of random_bytes but that can be fixed.
If you want to do it, that would be great, I can only do it tomorrow evening at the earliest.
Since there is no way around the xxhash extension for now (I don't think there is a PHP implementation), we should just add some error handling for the queries that currently need signing if the required functions aren't present, for those users that don't need the signed queries.
I think composer also has ability to do require for extensions. At least I have seen it for mbstring so maybe it also works for custom ones.
Yes, that should work, but can you make it optional if you only want to, for example, use the inventory functions?
I made a PR, couldn't test it on PHP 5.6 tho. Can anyone do that? I've included a polyfill for random_bytes in the composer file.
How about using this one? https://github.com/POGO-PHP/POGOEncrypt-PHP
No library DLL/.so needed.
@b6oy That repo was already referenced here and is already being included in the PR #92 (see the composer.json ). It is used to calculate the hash of the Signature protobuf. But some of the fields in this protobuf require a hash of specific variables (like latitude, longitude), calculated with xxhash. As far as we know, there is no pure PHP implementation of xxhash, which is why we are discussing using the extension (.dll/.so file).
I have already compiled it for windows, it can be found here. When I get around to it, I will also do it for linux and osx.
Why not using https://github.com/POGO-PHP/POGOEncrypt-PHP as @b6oy said ?
Because implementing it takes longer than writing a one-line issue text? :/ Pretty sure it will come soon enough.
@lambasoft Well I tried to explain that to b6oy in my message above, its not the only thing you need.
I understand we would need both POGOEncrypt-PHP and an xxHash implementation. But why are we so against an extension for xxhash? Something like:
https://github.com/Megasaxon/php-xxhash
Or others? I dont understand why some people were saying extensions would make the API Windows only?
I dont think people are fundamentally against using an extension. For example Droeftoeter/pokapi and jaspervdm/pogoapi-php both use this fork. However, at some point people were discussing whether a pure PHP option is desirable or not, but I dont think anyone actually wrote it.