fastjson icon indicating copy to clipboard operation
fastjson copied to clipboard

Use make instead of append to reduce memory usage

Open vincent-163 opened this issue 6 years ago • 4 comments

Since the pointers to original values are always held, it doesn't make sense to use append() and keep the original values anyway. I expect this change to reduce memory consumption by about 30% for large JSON values and avoid copying.

vincent-163 avatar Jan 19 '19 07:01 vincent-163

Codecov Report

Merging #23 into master will increase coverage by 0.02%. The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #23      +/-   ##
=========================================
+ Coverage   92.78%   92.8%   +0.02%     
=========================================
  Files           8       8              
  Lines         970     973       +3     
=========================================
+ Hits          900     903       +3     
  Misses         48      48              
  Partials       22      22
Impacted Files Coverage Δ
parser.go 90.37% <100%> (+0.06%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f1ad9dd...74f997f. Read the comment docs.

codecov[bot] avatar Jan 19 '19 07:01 codecov[bot]

I expect this change to reduce memory consumption by about 30% for large JSON values and avoid copying.

Uh I didn't notice this line. I thought the patch is for performance optimization, not a memory optimization. Then it looks valid. Could you add a benchmark with ReportAllocs or reference an existing benchmark, which shows the claimed improvement in memory usage?

valyala avatar Jan 19 '19 13:01 valyala

In fact, this will also improve performance by avoiding the copy. The benchmarks are here: https://gist.github.com/hewenyang/91618e54342f65809f4cb3c9c3aa1240 The observation is based on the canada benchmark which has the largest memory consumption. Its memory consumption went down from 1019612 B/op to 707650 B/op, and all parse benchmarks had a small but visible improvement in performance.

vincent-163 avatar Jan 19 '19 14:01 vincent-163

@valyala can this be merged?

harshavardhana avatar Aug 25 '19 19:08 harshavardhana