sepinf-inc
Extract SQLite time from record fields into lucene metadata fields.
Extract SQLite time from record fields into lucene metadata fields. Name the metadata field with tablename:column name.
Thanks. We already have some simple heuristic to detect different date formats/precision in sqlite rows here: https://github.com/sepinf-inc/IPED/blob/4.0.x/iped-parsers/iped-parsers-impl/src/main/java/iped/parsers/sqlite/SQLite3TableReader.java#L107-L180
Currently we just print the possible dates in the HTML table view and events are not extracted, this could be useful. Maybe it should be reused or unified with your on going work.
Just one tip, I think we should have some flag to enable/disable this, since it could result in false positives and it could generate dozens of thousands of events or more, the user may want to disable/enable depending on the case.
I think we should change the Generic SQLite Parser to break big tables in many itens. The number of items in each split can be the limit of values in a multivalored field, 32k. Or less, maybe a number that does not overload html viewer usage. This way, these extracted timestamps can be inserted in the correspondant table split item. Creating an item for each time stamp can exhaust docid space, which is a limit to the maximum number of items in a case.
I have pushed a branch named SQLSplit. It breaks the resultset in a parser configured param named tableRowsPerItem. I have implemented recognition of AppleORM tables based on table and column names. The values of FLOAT and INTEGER in fields that represents timestamps have a diferent EPOCH, so it should be parsed accordingly. Metadata of recognized date column values is added to a lucene field named DatabaseDate:tableName:columnName.