intelmq
intelmq copied to clipboard
Message dumping is slow for big files
Currently all the dump files are valid JSON files, which means that for every message to be dumped, this happens:
- Read and parse the full dump-file
- Append the new dump
- Re-write the entire file
Obviously, this becomes incredibly slow for bigger files.
One possibility for fixing this is to work line-based. So, it's not a valid JSON file anymore (at least with more than one line), but each line is on itself. Consequently, only one line needs to be appended. That's faster and less error-prone.
- [x] We shouldn't dump the same line twice. (#994)