Investigate maintaining the rekor search index as a separate continuous job

[patflynn]

(this idea came from @vaikas during the weekly on-call) Description

Currently the rekor entry creation flow is responsible for calling Redis to include new entries in the index. Ideally the rekor entry creation flow could stay unaware of the Redis index and instead some other job would monitor the log for new entries and update the index appropriately.

Benefits:

• GA API doesn't depend on non-GA API
• Maintenance of the index and fixing gaps in the index are done by the exact same job. (gaps would only occur when it has failed for some reason and it should pick up where it left off if it's brought back up)
• Perhaps there's some watching capability built into the API already to enable monitors?

(@vaikas sorry if I butched this, feel free to edit/fix)

[dlorenc]

Note that this also has verifiability benefits too, especially if coupled with a max-allowable latency. @znewman01 can probably explain better, but we could make this API actually verifiable by using a verifiable map, but the downside is that someone would have to actually verify the entire map every time we update it.

If we could live with a delay of something like 6 or 24 hours between an entry appearing in the log and an entry showing up in the map, we could make this verifiable.

[vaikas]

Thanks @patflynn for jotting this down, and thanks @dlorenc for piping up. That was another point I was curious about if by decoupling this we may be able to improve verifiability as well. We currently run the 'watch' as a separate pod so was curious if that could be extended or if we may need to create a separate process for this.

[lumjjb]

+1 on the ability for a watch interface. I think GUAC would also want to be able to perform a watch as well so that it can collect the rekor entries.

[haydentherapper]

Related is https://github.com/sigstore/rekor/issues/191