graph-node Frequent query failures on Decentralized Network

Do you want to request a feature or report a bug? BUG

What is the current behavior? Subgraph ID: 6tGbL7WBx287EZwGUvvcQdL6m67JGMJrma3JSTtt5SV7 When querying a particular subgraph, we're encountering occasional server error messages. It does not happen all the time, only on occasional queries.

Error message:

returned error 502 Bad Gateway: \n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n<script src=\"/cdn-cgi/apps/head/a-7iaLT0SRdb_kvxGKIYLrjkGzU.js\"></script></head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n<script defer src=\"https://static.cloudflareinsights.com/beacon.min.js/v652eace1692a40cfa3763df669d7439c1639079717194\" integrity=\"sha512-Gi7xpJR8tSkrpF7aordPZQlW2DLtzUlZcumS8dMQjwDHEnw9I7ZLyiOj/6tZStRBGtGgN6ceN6cMH8z7etPGlw==\" data-cf-beacon='{\"rayId\":\"(censored)\",\"token\":\"(censored)\",\"version\":\"2022.6.0\",\"si\":100}' crossorigin=\"anonymous\"></script>\n</body></html>\n

Querying on Postman and Playgrounds encounter the same errors, albeit a slightly different error message.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem. Postman query: https://thegraph.com/explorer/subgraph?id=6tGbL7WBx287EZwGUvvcQdL6m67JGMJrma3JSTtt5SV7&view=Playground

https://thegraph.com/explorer/subgraph?id=6tGbL7WBx287EZwGUvvcQdL6m67JGMJrma3JSTtt5SV7&view=Playground I'm only seeing this error when querying this particular subgraph. I've yet to see this error on other subgraphs that we’ve deployed to the decentralized network. I’ve tried to make the same query on the playground and am getting a Failed to fetch error. Occasionally, I’m getting a “No indexer available” error too.

I've lowered the batch size (list of MarketIDs passed in as parameters), but the errors still occur occasionally. I can't conclusively say if lowering the batch size reduces the frequency of errors.

Example query body:

{
  markets(subgraphError: allow, orderBy:id, orderDirection: asc, where:{id_in:["0x158079ee67fce2f58472a96584a73c7ab9ac95c1", "0x39aa39c021dfbae8fac545936693ac917d5e7563", "0x80a2ae356fc9ef4305676f7a3e2ed04e12c33946", "0x95b4ef2869ebd94beb4eee400a99824bf5dc325b", "0xb3319f5d18bc0d84dd1b4825dcde5d5f7266d407", "0x4b0181102a0112a2ef11abee5563bb4a3176c9d7", "0x35a18000230da775cac24873d00ff85bccded550", "0x7713dd9ca933848f6819f38b8352d9a15ea73f67", "0x041171993284df560249b57358f931d9eb7b925d", "0x12392f67bdf24fae0af363c24ac620a2f67dad86", "0x4ddc2d193948926d02f9b1fe9e1daa0718270ed5", "0x5d3a536e4d6dbd6114cc1ead35777bab948e3643", "0x6c8c6b02e7b2be14d4fa6022dfd6d75921d90e4e", "0x70e36f6bf80a52b3b46b3af8e106cc0ed743e8e4", "0xc11b1268c1a384e55c48c2391d8d480264a3a7f4", "0xccf4429db6322d5c611ee964527d42e5d685dd6a", "0xe65cdb6479bac1e22340e4e755fae7e509ecd06c", "0xf5dce57282a584d2746faf1593d3121fcac444dc", "0xf650c3d88d12db855b8bf7d11be6c55a4e07dcc9", "0xface851a4921ce59e912d19329929ce6da6eb0c7"]}) {
    id
    dailySnapshots(first: 1, orderBy: timestamp, orderDirection: desc, where:{timestamp_lt: 1656720000}) {
        id
        market {
          id
           inputToken {
            id
            decimals
          }
          outputToken {
            id
            decimals
          }
        }
        timestamp
        rates {
          id
          rate
          type
          side
        }
        totalValueLockedUSD
        totalDepositBalanceUSD
        totalBorrowBalanceUSD
        dailyBorrowUSD
        dailyDepositUSD
        dailyLiquidateUSD
        dailyRepayUSD
        dailyWithdrawUSD
        cumulativeSupplySideRevenueUSD
        dailySupplySideRevenueUSD
        cumulativeProtocolSideRevenueUSD
        dailyProtocolSideRevenueUSD
        cumulativeTotalRevenueUSD
        dailyTotalRevenueUSD
        inputTokenBalance
        inputTokenPriceUSD
        outputTokenSupply
        outputTokenPriceUSD
      }
  }
}

What is the expected behavior? No errors and values returned.

Aug 15 '22 04:08 lowjiansheng

Hi! I think this is therefore a subgraph / indexer specific issue on the network - I expect that this error is a timeout from the Indexer or the Gateway. There are a reasonably large number of entities in the subgraph, but I think there are lots of prior values of the snapshots, which likely slow the query down. I think indexers could likely benefit from applying the account-like optimisation here, but interested in @lutter's thoughts

Aug 17 '22 13:08 azf20

I just ran this query against the hosted service (by using the URL https://api.thegraph.com/subgraphs/id/Qme5mFGEqDg1KJ8PeaWj7vPDccTpSeHGPz9Zr8T4zAqJ3F)

The first time, the query took 20s, subsequent executions, even with small changes in the timestamp, take < 1s.

I think it is likely that the failure is due to a timeout caused by the indexer taking a long time to respond to the query.

Aug 18 '22 22:08 lutter

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

Feb 15 '23 00:02 github-actions[bot]

Closing this issue 🙏

Mar 06 '23 18:03 azf20

graph-node graph-node copied to clipboard

Frequent query failures on Decentralized Network

graph-node
graph-node copied to clipboard