Cannot recover from long-running formula

[samkhal]

I have a formula which is taking a long time to run. It's running on a table of 1k rows and the formula is

col_ids = [col.col_id for col in table.table.all_columns.values() if not col.col_id.startswith("gristHelper_")]
return '\n'.join(col_ids)

Initially, I just saw the "still working" dialog. Now I seem to be in an unresponsive state for 20 minutes:

• Opening the document presents a blank page with just the top menu bar visible.
• I already has a snapshot open in another tab. Everything seems functional there (and I exported it just in case), except that the Snapshots sidebar is empty and displays "Snapshots are unavailable" after some time. Console:

ERROR: TypeError: NetworkError when attempting to fetch resource. [errors.ts:94:2](https://grist-static.com/v/263dea7ee/core/app/client/models/errors.ts)
    S errors.ts:94
    _buildSnapshots DocHistory.ts:77

• Console output when opening the document:

Comm.useDocConnection(redacted): useCount now 1 [Comm.ts:165:4](https://grist-static.com/v/redacted/core/app/client/components/Comm.ts)
Comm request #0 openDoc 
Array(3) [ "redacted", "default", undefined ]
[Comm.ts:262:4](https://grist-static.com/v/redacted/core/app/client/components/Comm.ts)
GristWSConnection.send[false] {"reqId":0,"method":"openDoc","args":["redacted","default",null]} [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection: onopen: Connected [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection: onmessage (311 bytes) [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection established: clientId fb9618fb85b35cd5 counter 1 needReload false [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection.send[true] {"reqId":0,"method":"openDoc","args":["redacted","default",null]} [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
Comm: Triggering event clientConnect [Comm.ts:375:8](https://grist-static.com/v/redacted/core/app/client/components/Comm.ts)
App clientConnect event: needReload false version redacted [App.ts:140:14](https://grist-static.com/v/redacted/core/app/client/ui/App.ts)
GristWSConnection: onmessage (322 bytes) [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection skipping duplicate 'clientConnect' message [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection.send[true] {"beat":"alive","url":"https://docs.getgrist.com/x9QjeLNLxoqc/Transactions","docId":"redacted"} [GristWSConnection.ts:99:12](https://grist-static.com/v/redacted/core/app/client/components/GristWSConnection.ts)
GristWSConnection.send[true] {"beat":"alive","url":"https://docs.getgrist.com/x9QjeLNLxoqc/Transactions","docId":"redacted"} [GristWSConnection.ts:99:12](https://grist-static.com/v/263dea7ee/core/app/client/components/GristWSConnection.ts)

[dsagal]

There is a somewhat hidden way to recover from a data engine that's in an infinite loop. Go to document "Settings" page (in the bottom right of the left panel), and find the Engine setting there:

Click the ☠️ icon. It kills the data engine, and causes the document to reload. You should see the version before your change that introduced the infinite loop.

[samkhal]

10 hours later, still broken. I cannot access document settings; the page looks like this:

I can access the sidebar from within an earlier snapshot. If I click the skull, the page reloads with a message: "Too many backlogged requests for document"

[dsagal]

OK, apologies. It looks like the document got into a bad state while getting closed. I think there is a genuine bug that allows an infinite loop to interfere with shutdown. We shut down the data engine forcibly on the server side, so if you re-open it, it should work again.

If you are still having trouble, could you please share access to the document with [email protected]? (You can find instructions for how to share from the docs homepage without having to open it at https://support.getgrist.com/sharing/.)

[samkhal]

The document works again for me.

[vinyll]

I'm having the same issue with document gJub8T9vt31d. I can't get out of of getting the message "Too many backlogged requests for document gJub8T9vt31d4LUraqYG3o - try again later?".

What to do?

[paulfitz]

Backlogged requests can happen for a few reasons, including if API requests are coming in faster than the document can handle. For that, reloading the data engine can help temporarily: https://support.getgrist.com/document-settings/#data-engine

Though the symptom will recur if requests keep coming in.

If the problem is because of the document being slow because of some long formula, and you can't access it to remove or disable the formula, you can trigger recovery mode from the api console in Document Settings. In the api console, search for "Set recovery mode for a document". All the defaults are fine, you just need to click "try it out" and then "execute". Then your doc should be in recovery mode, meaning formulas are not computed but you can still edit the document.