nexus-repository-composer icon indicating copy to clipboard operation
nexus-repository-composer copied to clipboard

Make ComposerJsonProcessor more memory-efficient

Open fjmilens3 opened this issue 7 years ago • 1 comments

The ComposerJsonProcessor class performs multiple operations on JSON:

  • Translation of the downloaded list.json from the Packagist API into the JSON file containing links to all the projects/packages.
  • Rewrite of the individual provider JSON files to point to the Nexus proxy repository instead of the actual upstream Git repository.

These are currently done by unmarshaling all the contents of the JSON files into memory as maps. For the time being this seems to work just fine, but a more parsimonious way to do this would be to parse the JSON using something like the JsonParser API rather than the ObjectMapper API that we're currently using.

(Sonatype employees can reference our internal NPM implementation for ideas here. In rare situations for NPM large JSON file could exhaust available memory when we used the ObjectMapper-based approach, which eventually convinced us to switch to the JsonParser for these kinds of operations.)

fjmilens3 avatar Feb 23 '18 15:02 fjmilens3

Some of this will change depending on the outcome of other work in progress, so it would be wise to check in before picking this one up. However, this may be something that someone with familiarity or interest in Jackson, Composer JSON formats, or JSON processing in general might be able to pick up without a larger understanding of the entire project.

fjmilens3 avatar Jun 18 '18 20:06 fjmilens3