Implement index update rework

[pietroalbini]

Implementation work needs to be done to add the things proposed in this docs.rs RFC.

[jyn514]

Some preliminary things can be done without switching completely:

• storing the current commit in the database is pretty easy
• we can implement the webhook endpoint in addition to the current index updates and run them both in parallel for a while to make sure they work
• synchronization needs to be implemented before fully switching over to webhooks, because there is no analogue of peek_changes for webhooks (there's no way to ensure durability in case of a crash)

[pietroalbini]

synchronization needs to be implemented before fully switching over to webhooks, because there is no analogue of peek_changes for webhooks (there's no way to ensure durability in case of a crash)

Yes there is, if we update the hash of the last visited commit in the same database transaction as adding crates to the queue we'd get the same level of consistency we have today.

[jyn514]

last visited commit

I don't think this makes sense in the context on web-hooks because we might not receive the hooks in the order the crates were released. If crates A and B release and we get the webhook for B before A, updating the commit to point to B would mean we could skip A.

[Nemo157]

The webhook is just a trigger "check what crates were released between last commit and now", it will just call the exact same update code as the time based trigger does.