Logfile Management (DESIGN DISCUSSION)

[timbokopter]

Currently, all transactions log into a single logfile, which consequently grows over time and never gets deleted or truncated. Also, once a table is merged, its previous log entries must be removed/invalidated to avoid redo recovery in case of failure.

Solution 1: One logfile per table. Problems: increased logging overhead, commits must either be written into all logfiles or in a separate commit log.

Solution 2: One logfile for all tables, checkpoint entries when table is merged (i.e. "ignore previous log entries for table X"), new logfile after merge. When all tables have been merged once, the first logfile can be deleted, the second after all tables were merged twice and so on. Problems: Might take a while until all tables are merged.

Solution 3: One logfile for all tables, checkpoint entries when table is merged (i.e. "ignore previous log entries for table X"), new logfile after merge. Separate worker reads old logfiles on every table merge, removes entries and writes truncated log back. Problems: Potentially costly operation, IO overhead

[bastih]

Is this the same database that I'm looking at? Which log file are you talking about?

[timbokopter]

The issue is referring to an existing logger implementation that will be included as feature/logging into this repository soon (tested and working). However, it behaves as described above and the issue was opened to keep track of the problem and potentially come up with a good solution. Input welcome ;)

[bastih]

updated title to reflect that this is a design discussion