subql icon indicating copy to clipboard operation
subql copied to clipboard

Improve datasource endblock feature with fetch

Open jiqiang90 opened this issue 2 years ago • 0 comments

Prerequisites

  • [ ] Are you running the latest version(s)?
  • [ ] Have you searched the documentation for your issue?
  • [ ] Did you search for an existing issue or pull request?

Description

dataSources:
  - kind: substrate/Runtime
    startBlock: 1
    endBlock: 1
    mapping:
      file: ./dist/index.js
      handlers:
        - kind: substrate/BlockHandler
          handler: handleGenesisBlock
  - kind: substrate/Runtime
    startBlock: 10
    mapping:
      file: ./dist/index.js
      handlers:
        - kind: substrate/BlockHandler
          handler: handleModuloBlock100
          filter:
            modulo: 100

telegram-cloud-photo-size-1-5071312910432709846-y

As we can see from above example, we have two datasources, we are expecting after genesisHnadler triggered, it should fetch only modulo blocks rather than all blocks, i think the modulo condition can be improved https://github.com/subquery/subql/blob/ed317c0cbf80dd28bc8dd05625236b1fde9d4afc/packages/node-core/src/utils/project.ts#L121C28-L121C44

Also we need to inspect endblock has any impact with method filters, it might require some change on exclud end datasource from fetch

Details

These details can help to reproduce the environment the issue is occurring

Local Environment: [You can get this information from executing subql version.] Query Version: [What is the version of the query service?] Indexer Version: [What is the version of the indexer service?] Network Details:

  • [Network]
  • [Block height, a block height where the issue is happening]
  • [Dictionary endpoint, if used]

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [and so on...]

Example project: [A link to a minimal example that can reproduce the issue]

Expected behavior: [What you expected to happen]

Actual behavior: [What actually happened]

Any other information

Is there any other information you would like to add?

jiqiang90 avatar Nov 09 '23 21:11 jiqiang90