gno icon indicating copy to clipboard operation
gno copied to clipboard

feat: byte slice JSON parser

Open notJoon opened this issue 2 years ago • 10 comments

Description

I implemented a JSON parser using byte slices and encoding/decoding features using struct instances.

Most of the parsing code was referenced from the jasonparser package. This package processes JSON using byte slices, which I thought would be suitable for this project. also discussed in #1038.

Additionally, I use the method of creating struct instances to handle marshaling/unmarshaling without reflect. I believe this method is most suitable in the current context as it allows runtime data creation and manipulation.

The Get function takes JSON data and a key path as input and returns value and type information. Therefore, unmarshaling was implemented by using this function to retrieve necessary information and assembling a struct, and marshaling was implemented by taking information from a struct and applying type-specific marshaling rules to create a string. I also created CRUD functions for instances to make data modification easier.

I couldn't think of a direct way to utilize the structures. So, if you have a better suggestion, please let me know.

Note: unmarshaling is not yet capable of handling all types, but this is still a work in progress.


Reference

Here's a list of items I'm referencing as I progress through this PR

[1] https://pkg.go.dev/encoding/json [2] https://github.com/buger/jsonparser [3] https://github.com/valyala/fastjson [4] Parsing Gigabytes of JSON per Second

notJoon avatar Dec 06 '23 10:12 notJoon

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 47.54%. Comparing base (9336c7f) to head (209a754).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1415   +/-   ##
=======================================
  Coverage   47.54%   47.54%           
=======================================
  Files         388      388           
  Lines       61279    61279           
=======================================
  Hits        29133    29133           
  Misses      29707    29707           
  Partials     2439     2439           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 06 '23 10:12 codecov[bot]

Nice :)

Have you considered porting over buger/jsonparser? It is an existing reflection-free JSON implementation.

thehowl avatar Dec 07 '23 09:12 thehowl

Have you considered porting over buger/jsonparser? It is an existing reflection-free JSON implementation.

Yes, I'm currently working with jsonparser as a reference.

notJoon avatar Dec 07 '23 10:12 notJoon

It's not finished yet, but I've updated the status for an interim check.

notJoon avatar Jan 08 '24 10:01 notJoon

Can we see a diff of jasonparser and your stuff? jasonparser is a well tested library and we mostly need to focus on the things you added.

petar-dambovaliev avatar Jan 18 '24 22:01 petar-dambovaliev

Maybe consider adding unicode/utf16 to the whitelist https://github.com/gnolang/gno/blob/6fb3317436a9d15eac460fd0cb68cfef091f4f1a/gnovm/pkg/gnolang/precompile.go#L31-L63

harry-hov avatar Feb 14 '24 20:02 harry-hov

Maybe consider adding unicode/utf16 to the whitelist

Oh... I didn't realize this existed, added it. thank you @harry-hov !

notJoon avatar Feb 15 '24 06:02 notJoon

Now that I think about it, unicode/utf16 should be PR'd separately. it's not being used in JSON so it seems unnecessary to do it together.

notJoon avatar Feb 15 '24 08:02 notJoon

Most of the parsing code was referenced from the jasonparser package.

Since most of the logic is taken from the buger/jasonparser. Please make sure the MIT License from original author is respected.

harry-hov avatar Feb 15 '24 11:02 harry-hov

Most of the parsing code was referenced from the jasonparser package.

Since most of the logic is taken from the buger/jasonparser. Please make sure the MIT License from original author is respected.

I have added the full author attribution to both the README and the license

notJoon avatar Feb 19 '24 05:02 notJoon