intelmq icon indicating copy to clipboard operation
intelmq copied to clipboard

Message dumping is slow for big files

Open ghost opened this issue 8 years ago • 1 comments

Currently all the dump files are valid JSON files, which means that for every message to be dumped, this happens:

  • Read and parse the full dump-file
  • Append the new dump
  • Re-write the entire file

Obviously, this becomes incredibly slow for bigger files.

One possibility for fixing this is to work line-based. So, it's not a valid JSON file anymore (at least with more than one line), but each line is on itself. Consequently, only one line needs to be appended. That's faster and less error-prone.

ghost avatar Jan 30 '17 16:01 ghost

  • [x] We shouldn't dump the same line twice. (#994)

e3rd avatar Jun 08 '17 17:06 e3rd