framework icon indicating copy to clipboard operation
framework copied to clipboard

Invalid `prisma.block.findFirst()`, Invalid `prisma.block.aggregate() - invocation Timed out fetching a new connection from the connection pool.

Open anarkrypto opened this issue 1 year ago • 3 comments

I just reseted my chain data 4 times. Everytime it reaches more than 200K blocks, we have this kind of error when trying to fetch the Protokit module

It can be:

Invalid `prisma.block.findFirst()`  invocation Timed out fetching a new connection from the connection pool.
OR
Invalid `prisma.block.aggregate()` invocation Timed out fetching a new connection from the connection pool.
OR
nInvalid `prisma.state.findMany()`  invocation Timed out fetching a new connection from the connection pool.
OR
No Metadata has been set for block {\"height\":245780} yet

Query:

{"operationName":"StateRaw","query":"query StateRaw($path: String!) {\n  state(path: $path)\n}","variables":{"path":"16065717386198936724837320364046225795689660140832830837246559267598137910778"}}

Response example:

{
    "errors": [
        {
            "message": "\nInvalid `prisma.state.findMany()` invocation:\n\n\nTimed out fetching a new connection from the connection pool. More info: http://pris.ly/d/connection-pool (Current connection pool timeout: 10, connection limit: 9)",
            "path": [
                "state"
            ]
        }
    ],
    "data": {
        "state": null
    }
}

Server Log:

Timed out fetching a new connection from the connection pool. More info: http://pris.ly/d/connection-pool (Current connection pool timeout: 10, connection limit: 9)
    at si.handleRequestError (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:125:6817)
    at si.handleAndLogRequestError (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:125:6151)
    at si.request (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:125:5859)
    at async l (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@prisma/client/runtime/library.js:130:10025)
    at async PrismaTransactionStorage.getPendingUserTransactions (file:///app/node_modules/.pnpm/@[email protected]_tdl2jvgpdktg37cijcy3bniih4/node_modules/@proto-kit/persistance/dist/services/prisma/PrismaTransactionStorage.js:22:21)
    at async PrivateMempool.getTxs (file:///app/node_modules/.pnpm/@[email protected]_@[email protected][email protected]_tsyring_bktuhgt3hqwpzdd6xe2pr644um/node_modules/@proto-kit/sequencer/dist/mempool/private/PrivateMempool.js:41:16)
    at async TimedBlockTrigger.produceUnprovenBlock (file:///app/node_modules/.pnpm/@[email protected]_@[email protected][email protected]_tsyring_bktuhgt3hqwpzdd6xe2pr644um/node_modules/@proto-kit/sequencer/dist/protocol/production/trigger/TimedBlockTrigger.js:69:28)
    at async Timeout._onTimeout (file:///app/node_modules/.pnpm/@[email protected]_@[email protected][email protected]_tsyring_bktuhgt3hqwpzdd6xe2pr644um/node_modules/@proto-kit/sequencer/dist/protocol/production/trigger/TimedBlockTrigger.js:49:21) {
  code: 'P2024',
  clientVersion: '5.7.1',
  meta: { modelName: 'Transaction', connection_limit: 9, timeout: 10 }
}

Image

We can also see a lot of accumulated pending queries. I am ticking with 1 second

Image

System

Ubuntu 22.04.5 LTS Docker version 27.3.1, build ce12230 Protokit: 0.1.1-develop.1088

I am using the StarteKit

I am running in a VPS 6GB ram, 4 vCPU Cores, SSD but don't see a high resources usage.

I even reseted the OS entirely, it does not help.

anarkrypto avatar Jan 17 '25 10:01 anarkrypto

Hi 👋 ,

Would it be possible for you to upgrade to the latest version of Prisma and try to see if the issue persists?

ankur-arch avatar Jan 27 '25 09:01 ankur-arch

Hi 👋 ,

Would it be possible for you to upgrade to the latest version of Prisma and try to see if the issue persists?

thanks for the suggestion. I upgraded to 6.2.1 and I'am already testing it. Unfortunately it may take some weeks to see the results since I had to reset the chain AGAIN due to metadata missing, probably caused by the Prisma connections issue.

anarkrypto avatar Jan 28 '25 14:01 anarkrypto

Update: After upgrade the prisma to 6.2.1, protokit was able to work until block 394930

But again the same error. No Metadata has been set for block {\"height\":394931} yet

 at async TimedBlockTrigger.produceUnprovenBlock (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/protocol/production/trigger/TimedBlockTrigger.js:69:28)
    at async Timeout._onTimeout (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/protocol/production/trigger/TimedBlockTrigger.js:49:21)
Error: No Metadata has been set for block {"hash":"2878482696137401078971966177828847229262071160398026893966888705214911061702"} yet
    at PrismaBlockStorage.getBlockByQuery (file:///app/node_modules/.pnpm/@[email protected]_jibe6vn2ykm2jr5ngnp3vkzdv4/node_modules/@proto-kit/persistance/dist/services/prisma/PrismaBlockStorage.js:45:19)
    at async PrismaBlockStorage.getLatestBlock (file:///app/node_modules/.pnpm/@[email protected]_jibe6vn2ykm2jr5ngnp3vkzdv4/node_modules/@proto-kit/persistance/dist/services/prisma/PrismaBlockStorage.js:132:16)
    at async PrivateMempool.getStagedNetworkState (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/mempool/private/PrivateMempool.js:53:24)
    at async PrivateMempool.getTxs (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/mempool/private/PrivateMempool.js:60:31)
    at async TimedBlockTrigger.produceUnprovenBlock (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/protocol/production/trigger/TimedBlockTrigger.js:69:28)
    at async Timeout._onTimeout (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/protocol/production/trigger/TimedBlockTrigger.js:49:21)
Error: No Metadata has been set for block {"hash":"2878482696137401078971966177828847229262071160398026893966888705214911061702"} yet
    at PrismaBlockStorage.getBlockByQuery (file:///app/node_modules/.pnpm/@[email protected]_jibe6vn2ykm2jr5ngnp3vkzdv4/node_modules/@proto-kit/persistance/dist/services/prisma/PrismaBlockStorage.js:45:19)
    at async PrismaBlockStorage.getLatestBlock (file:///app/node_modules/.pnpm/@[email protected]_jibe6vn2ykm2jr5ngnp3vkzdv4/node_modules/@proto-kit/persistance/dist/services/prisma/PrismaBlockStorage.js:132:16)
    at async PrivateMempool.getStagedNetworkState (file:///app/node_modules/.pnpm/@[email protected]_@[email protected]_patch_hash=7f63g_gir5rtntmsb7dutribapk3gab4/node_modules/@proto-kit/sequencer/dist/mempool/private/PrivateMempool.js:53:24)

anarkrypto avatar Feb 25 '25 14:02 anarkrypto