gchq
Optimise ref data lookups for entry type (all keys, all ranges, mixed)
If the ref data load recorded whether the data was made up of all key value entries, all range value entries or a mix and store this in the processing info table, we could then optimise the lookups. Assuming that we have to query the proc info table anyway then this will remove the pointless hit on the kv store for range lookups and remove the pointless lookup on the range store for kv entries that are not found.
Ideally we need some kind of stateful (for the life of the pipeline process) version of the RefDataStore that can establish and hold info about the loaded data on heap for faster checks on each lookup.
If we create a statefull (pipe scoped) RefDataStore that is held by one thread then we could hold a readtxn object on it and just reset/renew it all the time rather than ending txns all the time.