JSON data missing

[qpautrat]

Hello again,

Why some valuable data from replay are missing into JSON plugin? Maybe i make a mistake but for example map attribute is not define, so we can't get map name.

Thanks!

[GraylinKim]

There is a map attribute:

https://github.com/GraylinKim/sc2reader/blob/master/sc2reader/plugins/replay.py#L64

But now that I look at it I don't think it will work like it used to. The toJSON code was gifted by someone else and I have done a poor job updating as changes are made to the core library. I'll try to fix this one up for you.

[qpautrat]

I saw this line, and i'm pretty sure map attribute doesn't work.

I stay tuned, your work is greatly appreciated, thank you ;)

[GraylinKim]

What is your use case for the JSON plugin @qpautrat?

Back when it was first written it encompassed almost all of the information available. Now-a-days it doesn't even come close to all the information available.

Are you wishing that it would include more detailed data, like events and selections, or just all of the general replay data, like map_name, expansion, game_length, real_length, etc?

[qpautrat]

@GraylinKim I think it will be great to have all data available. I don't have really specific needs here, so i'm trying to get all information.

[GraylinKim]

Okay @qpautrat, its going to be a busy couple weeks for me. Why don't you try adding the data you are looking for and pull request the solution that worked for you. I'd be happy to answer any questions that come up if you stop by the #sc2reader channel.

If you'd prefer to wait then I'll see what I can do about this come May.

[qpautrat]

I would like but like i said i'm not a Pyton dev, i'm not confortable with it, so i prefer let you handle this, sorry ;)

[VnC-]

I fixed the map issue by using map_name, and added the some other variables I found relevant in pull request #130

[GraylinKim]

The sc2json script should be extended to provide more data and cover all major resource types.

[delta1]

Hi @GraylinKim , I'm hoping to take a look at this when I have a chance, to get more familiar with the code. Is an updated sc2json script still required? Is it in use anywhere? Please elaborate on how it should be extended, if possible. Otherwise I will take a look and just try to add as much info as possible.

[GraylinKim]

Hi, thanks for volunteering!

I would still like to maintain the sc2json script. I would also like to deprecate the toJson plugin. Instead, I'd like to have sc2reader implement a JsonDecoder that knows how to handle all of sc2reader's objects. The sc2json script (and other users) can then serialize any sc2reader object with the following:

import sc2reader
replay = sc2reader.load_replay(args.path[0])
json.dumps(replay, cls=sc2reader.utils.JsonEncoder())

I think this is a good approach because it keeps all the json converting clutter out of the main code base and if people don't like our serialization they can either extend our encoder or just copy/paste and edit to their satisfaction.

I imagine this JsonEncoder looking something like this gist where I stole most of the code from the existing toDict function. Maybe the conversion for each class would be broken out into a function for readability or obj.dict could be used more heavily.

What do you think about this? I'm open to other approaches.

[delta1]

Sounds good, I'll give that a bash. Thanks!

[delta1]

Hey @GraylinKim please have a look at this pull request https://github.com/GraylinKim/sc2reader/pull/157

[GraylinKim]

Yes, I saw the request. Thanks.

I won't have time to look at this until this weekend.

[delta1]

Thanks for the feedback, will take a look this week hopefully. For testing do you just run the script over all the replays in the repo?

[GraylinKim]

Examine some results by hand first. Once you are happy with them it would be good to run over the whole set of replays; they cover a wide variety of edge cases you might run into. Once you get things in a good place it would also be good to add some regression tests (that verify the json output) to the test suite to protect against breaks as I continue working on the library.

[headwinds]

So this is an n00b question but this seems like the appropriate thread as everyone is working on a json solution here with a lot more success than I am.

I'm primarily a JS dev so only have very limited python experience. I'm really interested in visualizing this replay data over time using the svg library D3 similar to the awesome ggtracker.com but to do something different.

I've gone through the docs and am struggling a bit about where one should begin to get all the replay data dumped into a json file.

I was able to run the following file in the scripts folder which is encouraging...

python sc2json.py "myGame.SC2Replay"

...but this just dumps out a summary not the full replay data. So either my replay only contains a summary or am I'm not unlocking the rest of the data properly. How does one dump the full replay into json?

Perhaps we could turn this discussion into some documentation on the wiki ;-D I looked for a json wiki page without any luck - can you point to it or can we create one?

[GraylinKim]

This isn't a problem with your replay, it is a limitation of the sc2json script. There is not currently a mechanism in sc2reader for dumping all data to json. This is because the built-in python json library cannot serialize arbitrary objects. There is an open issue herehttps://github.com/GraylinKim/sc2reader/issues/117 tracking this issue.

With the current code base you will need to write your own script that builds a python dict with all the information you are looking for. The json library can then be used to serialize it for output. Check out the toDict functionhttps://github.com/GraylinKim/sc2reader/blob/master/sc2reader/factories/plugins/replay.py#L21for an idea of how this might work.

A partial solution https://github.com/GraylinKim/sc2reader/pull/157 has been implemented for better json support but is not merged because no one has completed it. There has been quite a bit of interest in dumping full replay objects to json recently so I may try to finish that work this weekend.

Does that answer your questions?

On Thu, Jan 30, 2014 at 12:31 PM, brandon flowers [email protected]:

So this is an n00b question but this seems like the appropriate thread as everyone is working on a json solution here with a lot more success than I am.

I'm primarily a JS dev so only have very limited python experience. I'm really interested in visualizing this replay data over time using the svg library D3 similar to the awesome ggtracker.com but to do something different.

I've gone through the docs and am struggling a bit about where one should begin to get all the replay data dumped into a json file.

I was able to run the following files in the scripts folder which is encouraging...

python sc2json.py "myGame.SC2Replay" python sc2printer.py "myGame.SC2Replay"

...but this just dumps out a summary not the full replay data. So either my replay only contains a summary or am I'm not unlocking the rest of the data properly. How does one dump the full replay into json?

Perhaps we could turn this discussion into some documentation on the wiki ;-D I looked for json wiki page without any luck - can you point to it or can we create one?

Reply to this email directly or view it on GitHubhttps://github.com/GraylinKim/sc2reader/issues/117#issuecomment-33711277 .

[headwinds]

Yes it does! thank you very much.

Since there is interest from other JS devs, let's not have them go digging through the weeds like I just did...

Let's improve the path to this gold

On the main readme.md file, can you simply add a heading like Exporting Data to JSON and link to a wiki page about it. For now, just dump that answer ;-D

I'd be happy to help edit and contribute feedback from a JS dev's view. In the meantime, I'll also roll up my sleeves and learn some more python to play around with your dictionary recommendation. thanks again!