[Lake] Add retries around filling the data lake.

[idiom-bytes]

Motivation

As an outcome of reviewing 514/#552, we emphasized that it would be valuable to add a retry around filling the data_lake.

Outline

Any inputs/triggers that cause gql_lake (and perhaps olhcv_lake too) to fetch/sync, should offer a clean "retry" around the outer-loop such that ["failures from subgraph", "failures from ccxt"] and should provide it in the fn interface.

This should improve the UX of cli_commands/agents, such that building the lake is more resillient to subgraph/api errors.

DoD:

• [ ] Add retries around data_factory loops such that they resume from last_checkpoint.
• [ ] Continue evolving data factory & data modelling (i.e. beyond pdr analytics) example: volume ticks, f(x) w/ 100 params, improved risk models for trading execution.

[trentmc]

Note: commit https://github.com/oceanprotocol/pdr-backend/commit/674ca138fdd6afd34905a536498eca26a02246b9 auto-linked above accidentally pointed to this issue, rather than the correct one #653

[idiom-bytes]

We have various retries added and improved this overall. I have closed this ticket and have updated the core issue.