itchyny
Performance Issue: High CPU Usage in GoJQ Operations
Problem Description
I'm experiencing significant CPU performance issues when using gojq. Profiling shows that several core functions are consuming excessive CPU time, making operations much slower than expected.
Profiler Analysis
Based on CPU profiling results, the following functions show concerning performance characteristics:
Top CPU Consumers:
- mapIternext: 15.2% of total CPU time
- mapassign_faststr: 14.1% of total CPU time
- normalizeNumbers: 9.5% of total CPU time
- aeshashbody: 9.1% of total CPU time
Performance Hotspots:
- Map operations appear to be the primary bottleneck
- String assignment operations are taking significant time
- Number normalization is consuming substantial CPU cycles
- Hash operations are also contributing to the performance degradation
Environment
- GoJQ Version:
v0.12.13 - Go Version:
1.23 - OS:
ubuntu - Architecture:
x86
Sample Query/Data
# Example:
cat large_file.json | gojq '.[] | select(.field == "value") | .other_field'
Possible Optimization Areas
Based on the profiler output, potential areas for optimization might include:
- Map iteration and assignment optimizations
- String handling improvements
- Number processing efficiency
- Hash function performance
Request
Could the maintainers please investigate these performance bottlenecks? The high CPU usage in map operations and string assignments suggests there might be opportunities for optimization in these core areas.
Profiler screenshot attached showing detailed breakdown of CPU usage by function.
How does other jq implementations perform on the same input file? also i think it will be hard to help without you sharing large_file.json or some similar file that behave the same way.
You can also try cat large_file.json | gojq empty to see if most of the time is spent parsing input JSON, the screenshot makes me suspect that.
