fprime icon indicating copy to clipboard operation
fprime copied to clipboard

BufferLogger CRC calculations are inefficient

Open kubiak-jpl opened this issue 2 years ago • 0 comments

F´ Version v3.1.1
Affected Component BufferLogger,Os/ValidateFileCommon.cpp


Feature Description

Update BufferLogger to calculate CRC32 as the file is being written to disk.

Rationale

Currently, when BufferLogger closes a file it goes on to create a validation file. On Linux, this re-opens the file and reads it 256 bytes at a time. For larger files, say 10 MB, this results in around 40,000 syscalls when closing a file. This can impact the performance of other threads running on the same core, regardless of the priority.

This 256 byte size is hidden in a header file and is not configurable without forking fprime. https://github.com/nasa/fprime/blob/91c578ee37c01ba92e556dbca0287d82f9f18b95/Os/ValidateFile.hpp#L16

Increasing this value would partially alleviate the issue but I want to propose another solution. Calculate the CRC at the same time as the buffers are being written to disk. This would not require any read syscalls or any additional processing when closing a file. However, it would mean that the ValidateFile abstraction could not be used within BufferLogger

kubiak-jpl avatar Jan 09 '23 19:01 kubiak-jpl