kraken icon indicating copy to clipboard operation
kraken copied to clipboard

Add --db-index, --db-file, --taxonomy, --db-pipe for piped db files

Open yesimon opened this issue 7 years ago • 8 comments

This allows loading the database from a pipe instead of a file. This is useful in cloud platforms like AWS/GCP to load the database from an object store download and decompress on the fly, which is both faster and cheaper than storing on networked disk.

Also allows processing multiple inputs in one run. This is more important for the loading from pipe scenario because the database files are not implicitly cached in the page cache between runs.

yesimon avatar Oct 24 '17 01:10 yesimon

👍 It would be great to see this merged in.

tomkinsc avatar Oct 24 '17 16:10 tomkinsc

I'll take a look at this ASAP (hopefully today, if not by the end of this week)

jenniferlu717 avatar Oct 24 '17 17:10 jenniferlu717

+1 Interested in this feature. Thanks!

clintval avatar Jan 05 '18 23:01 clintval

+1 also interested in this feature!

maxmaliska avatar Feb 07 '18 19:02 maxmaliska

@DerrickWood also interest on this end.

nh13 avatar Feb 07 '18 23:02 nh13

@jenniferlu717 Any chance of a review or merge?

clintval avatar Apr 20 '18 04:04 clintval

Apologies....we've been working on releasing Kraken 2.0 ASAP and haven't had a chance to review this. It has a number of conflicts with the current code so merging is a bit more difficult but I will try to look at it soon.....

jenniferlu717 avatar May 08 '18 04:05 jenniferlu717

In that case perhaps it's better for me to wait until 2.0 before resolving conflicts

yesimon avatar May 11 '18 06:05 yesimon