platform icon indicating copy to clipboard operation
platform copied to clipboard
verified publisher icon hcengineering

Unexpected content rollback in Tracker Issues during storage performance issues

Open copy1dollar opened this issue 7 months ago • 8 comments

I am internally using Huly as a self-hosted service deployed on Kubernetes (K8S). Our K8S environment mounts NAS disks for storage.

Recently, due to NAS performance and network issues, we've been experiencing intermittent significant drops in disk read and write speeds. While this affects the Huly service, we've been able to continue using it despite the occasional inconvenience.

However, I'm reporting an error that occurred recently while using the Huly service.

We've discovered that content in Tracker Issues that multiple users view and edit together is rolling back to previous versions unexpectedly. Users who have only read the content (not edited it) are showing up in the history as having modified the entire content back to an earlier version. The history clearly shows that a user modified the entire content, even though that user only viewed it without making any changes. This causes confusion for other users who then see the rolled-back content. While these changes are recorded in the History(Activity), it's very inconvenient to have to restore content from previous versions.

When we first noticed this issue, we thought it might be a browser cache problem. However, after continued observation, we believe this phenomenon occurs more frequently during periods of NAS performance issues.

While slow service is inconvenient but usable, having Issue content unexpectedly roll back creates critical communication errors that make the Huly service unusable.

Please help us resolve this issue.

copy1dollar avatar May 08 '25 01:05 copy1dollar

Connected to Huly®: UBERF-10538

huly-github-staging[bot] avatar May 08 '25 01:05 huly-github-staging[bot]

@copy1dollar It may help if you provide some additional information:

  • What Huly version do you use?
  • What browser and os do user use? Do you use browser on some mobile devices like mobile phones or tablets?
  • Do you think it depends on content of document? Is it reproducible for large or small documents? Do these documents have any specific content (tables, action items, etc.)
  • Do you see any errors in collaborator service logs?

I'm trying to catch this issue but so far without success. Even with slow storage for me it works just fine.

aonnikov avatar May 08 '25 16:05 aonnikov

@aonnikov Thank you for your help.

  • What Huly version do you use?
    • Huly version: v0.6.496
  • What browser and os do user use? Do you use browser on some mobile devices like mobile phones or tablets?
    • Mostly using Chrome browser on Windows 11, and some are using Chrome on macOS. Also, we don't use it on mobile phones, but some users are using Safari on iPads. This issue seems to be occurring consistently across different users.
  • Do you think it depends on content of document? Is it reproducible for large or small documents? Do these documents have any specific content (tables, action items, etc.)
    • The phenomenon appears to repeat consistently in documents where it has occurred once.
    • It seems to only happen with large documents. I'm not sure if this is because many people are using it or because there's a lot of content in the document. Also, not only is there a lot of content, but there's also a significant edit history (Activity) with over 300 modifications by 5 users.
    • The content includes various elements: Images, tables, action items, and code blocks.
  • Do you see any errors in collaborator service logs?
    • There are errors in the collaborator service logs. I'm attaching the collaborator service error logs below.
^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D981E20DE55A8","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"IncompleteBody","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"You did not provide the number of bytes specified by the Content-Length HTTP header.","name":"S3Error","requestid":"183D981E20DE55A8","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: You did not provide the number of bytes specified by the Content-Length HTTP header.\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"level":"info","message":"error","retries":4,"timestamp":"2025-05-08T15:51:55.665Z"}^[[39m

TypeError: fetch failed
    at node:internal/deps/undici/undici:13502:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  [cause]: Error: getaddrinfo ENOTFOUND stats
      at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'stats'
  }
}

^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982D2C04C25B","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982D2C04C25B","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onDisconnect (/usr/src/app/bundle.js:284540:9)\n    at async /usr/src/app/bundle.js:94591:11"},"level":"info","message":"error","retries":4,"timestamp":"2025-05-08T15:52:30.272Z"}^[[39m
^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982D2C19E823","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982D2C19E823","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"level":"info","message":"error","retries":4,"timestamp":"2025-05-08T15:52:30.275Z"}^[[39m

^[[31m{"error":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982D2E344B34","code":"UnknownError","message":"503","name":"S3Error","stack":"S3Error: 503\n    at Object.parseResponseError (/usr/src/app/bundle.js:305875:18)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.makeRequestAsyncOmit (/usr/src/app/bundle.js:306924:21)\n    at async Client.statObject (/usr/src/app/bundle.js:307328:21)\n    at async MinioService.stat (/usr/src/app/bundle.js:311313:26)\n    at async FallbackStorageAdapter.stat (/usr/src/app/bundle.js:285493:24)\n    at async loadCollabYdoc (/usr/src/app/bundle.js:163854:20)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.loadDocument (/usr/src/app/bundle.js:284892:24)\n    at async StorageExtension.loadDocument (/usr/src/app/bundle.js:284550:18)\n    at async StorageExtension.onLoadDocument (/usr/src/app/bundle.js:284500:16)\n    at async Hocuspocus.loadDocument (/usr/src/app/bundle.js:95066:11)\n    at async Hocuspocus.createDocument (/usr/src/app/bundle.js:95029:11)\n    at async ClientConnection.setUpNewConnection (/usr/src/app/bundle.js:94426:29)\n    at async ClientConnection.handleQueueingMessage (/usr/src/app/bundle.js:94483:15)"},"level":"error","message":"no object found","objectName":"6806db25dc19f95e4a149a8d%content","timestamp":"2025-05-08T15:52:30.316Z","workspaceId":"w-cswoo-innovationla-676429c0-03b42301be-11cc41"}^[[39m

^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F236679FD","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F236679FD","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"level":"info","message":"error","retries":1,"timestamp":"2025-05-08T15:52:38.721Z"}^[[39m
^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F23C6E26F","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F23C6E26F","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onDisconnect (/usr/src/app/bundle.js:284540:9)\n    at async /usr/src/app/bundle.js:94591:11"},"level":"info","message":"error","retries":1,"timestamp":"2025-05-08T15:52:38.723Z"}^[[39m
^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F2A1E84B6","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F2A1E84B6","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"level":"info","message":"error","retries":0,"timestamp":"2025-05-08T15:52:38.833Z"}^[[39m
^[[31m{"documentName":"w-cswoo-innovationla-676429c0-03b42301be-11cc41|681c0890503d816cbab392c8|681c0977503d816cbab39422|content","error":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F2A1E84B6","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F2A1E84B6","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"level":"error","message":"failed to save document ydoc content","timestamp":"2025-05-08T15:52:38.834Z"}^[[39m

^[[31m{"documentName":"w-cswoo-innovationla-676429c0-03b42301be-11cc41|681c0890503d816cbab392c8|681c0977503d816cbab39422|content","error":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F2A1E84B6","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F2A1E84B6","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"level":"error","message":"failed to save document","timestamp":"2025-05-08T15:52:38.835Z"}^[[39m
[onStoreDocument] Failed to save document
[onStoreDocument] Failed to save document
Caught error during storeDocumentHooks Error: Failed to save document
    at StorageExtension.storeDocument (/usr/src/app/bundle.js:284574:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)
^[[31m{"level":"error","message":"Unhandled Rejection at:","promise":{},"reason":{"message":"Failed to save document","stack":"Error: Failed to save document\n    at StorageExtension.storeDocument (/usr/src/app/bundle.js:284574:17)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async StorageExtension.onStoreDocument (/usr/src/app/bundle.js:284522:9)"},"timestamp":"2025-05-08T15:52:38.837Z"}^[[39m
^[[32m{"err":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F2A691F0C","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F2A691F0C","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onDisconnect (/usr/src/app/bundle.js:284540:9)\n    at async /usr/src/app/bundle.js:94591:11"},"level":"info","message":"error","retries":0,"timestamp":"2025-05-08T15:52:38.838Z"}^[[39m
^[[31m{"documentName":"w-cswoo-innovationla-676429c0-03b42301be-11cc41|681c0890503d816cbab392c8|681c0977503d816cbab39422|content","error":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F2A691F0C","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F2A691F0C","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onDisconnect (/usr/src/app/bundle.js:284540:9)\n    at async /usr/src/app/bundle.js:94591:11"},"level":"error","message":"failed to save document ydoc content","timestamp":"2025-05-08T15:52:38.839Z"}^[[39m
^[[31m{"documentName":"w-cswoo-innovationla-676429c0-03b42301be-11cc41|681c0890503d816cbab392c8|681c0977503d816cbab39422|content","error":{"amzId2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","amzRequestid":"183D982F2A691F0C","bucketname":"w-cswoo-innovationla-676429c0-03b42301be-11cc41","code":"SlowDownWrite","hostid":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","key":"681c0977503d816cbab39422%content","message":"Resource requested is unwritable, please reduce your request rate","name":"S3Error","requestid":"183D982F2A691F0C","resource":"/w-cswoo-innovationla-676429c0-03b42301be-11cc41/681c0977503d816cbab39422%content","stack":"S3Error: Resource requested is unwritable, please reduce your request rate\n    at parseError (/usr/src/app/bundle.js:305832:18)\n    at Object.parseResponseError (/usr/src/app/bundle.js:305873:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Client.makeRequestStreamAsync (/usr/src/app/bundle.js:306982:23)\n    at async Client.uploadBuffer (/usr/src/app/bundle.js:307778:21)\n    at async MinioService.put (/usr/src/app/bundle.js:311341:16)\n    at async saveCollabYdoc (/usr/src/app/bundle.js:163870:7)\n    at async withRetry (/usr/src/app/bundle.js:285027:18)\n    at async PlatformStorageAdapter.saveDocument (/usr/src/app/bundle.js:284931:11)\n    at async StorageExtension.storeDocument (/usr/src/app/bundle.js:284562:30)\n    at async StorageExtension.onDisconnect (/usr/src/app/bundle.js:284540:9)\n    at async /usr/src/app/bundle.js:94591:11"},"level":"error","message":"failed to save document","timestamp":"2025-05-08T15:52:38.840Z"}^[[39m
[onDisconnect] Failed to save document
[onDisconnect] Failed to save document
^[[31m{"level":"error","message":"Unhandled Rejection at:","promise":{},"reason":{"message":"Failed to save document","stack":"Error: Failed to save document\n    at StorageExtension.storeDocument (/usr/src/app/bundle.js:284574:17)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async StorageExtension.onDisconnect (/usr/src/app/bundle.js:284540:9)\n    at async /usr/src/app/bundle.js:94591:11"},"timestamp":"2025-05-08T15:52:38.840Z"}^[[39m

copy1dollar avatar May 09 '25 07:05 copy1dollar

@copy1dollar there is definitely a problem with your storage. For how long it is being unavailable? The collaborator services performs only several attempts to save the document. If it failed to save the document, then the changes will likely be lost ... I'm not completely sure that this is the root cause of the content rollback, but this can be a problem. We can increase number of attempts and add some delays between them. Additionally we can think about temporary saving the document in the collaborator local storage.

aonnikov avatar May 11 '25 16:05 aonnikov

@aonnikov Thank you for your help.

I'm aware that there are issues with saving. The problematic times are intermittent and occur no more than 5 times a day, lasting from as short as 1 second to as long as 10 seconds. First, I will try to move the Huly self-hosting server to a more stable server.

However, in the GitLab self-hosting that we use internally, these issues don't occur because there is no collaborative editing feature. If a save fails, it should simply not save. Content that has already been successfully saved by other users should not be rolled back.

User A saves "abcd" and the save is successful. Users B and C both read "abcd". User B changes it to "abefgh" and saves. User C makes no changes. A, B, and C all read "abefgh" as the latest content. After some time, A, B, and C all read "abcd", which is rolled-back content. When checking the activity log, there is a record showing that one of the users (A, B, or C - it's random) wrote "abcd". However, the user never actually changed the content from "abefgh". If there are saving issues, the content should remain at the latest version, which is "abefgh". But it's showing the rolled-back content "abcd", and there's a record in the activity log showing it was saved (even though it wasn't!). I think this behavior clearly indicates a bug.

copy1dollar avatar May 12 '25 00:05 copy1dollar

@copy1dollar I found the issue and implemented the fix, it will be available a bit later today in version v0.6.500 Will you be able to check and confirm that issue is not reproducible anymore? UPD: The version should already be available

aonnikov avatar May 13 '25 02:05 aonnikov

@aonnikov I will try it.

copy1dollar avatar May 14 '25 00:05 copy1dollar

@aonnikov Thank you for your help.

I updated to version 0.6.501 today and will be monitoring to see if the same issue occurs. Since the issue is intermittent, I expect that monitoring will take some time.

By the way, could you clarify what the root cause was and how it was fixed? Is the fix described in the issue below the one that addresses this problem? Knowing what was changed would help me determine whether my issue has been properly resolved, and it would also help reduce the time required for monitoring.

UPD: Are #8889 and #8896 the pull requests that contain the fix?

copy1dollar avatar May 16 '25 13:05 copy1dollar

@aonnikov Thank you for your help.

I updated to version 0.6.501 today and will be monitoring to see if the same issue occurs. Since the issue is intermittent, I expect that monitoring will take some time.

By the way, could you clarify what the root cause was and how it was fixed? Is the fix described in the issue below the one that addresses this problem? Knowing what was changed would help me determine whether my issue has been properly resolved, and it would also help reduce the time required for monitoring.

UPD: Are #8889 and #8896 the pull requests that contain the fix?

It is mostly https://github.com/hcengineering/platform/pull/8889 and https://github.com/hcengineering/platform/pull/8893

There were one issue with our storage adapters that swallowed error on storage requests and instead of retrying attempt considered that the document did not exist.

aonnikov avatar May 18 '25 13:05 aonnikov

According to our monitoring, the same issue no longer occurs. It seems that the problem has been resolved through the version update.

@aonnikov Thank you for the quick fix.

copy1dollar avatar May 22 '25 00:05 copy1dollar