App
App copied to clipboard
Fix replay effect
Details
The replay effect happens when a user takes multiple actions while offline, the requests are queued up in the SequentialQueue, and then they go back online and each request is processed on at a time. Since optimistic data has already been set while offline, each Pusher event that arrives updates Onyx with outdated data, causing the offline actions to "replay".
In order to fix the replay effect from the App side we are first sending the Pusher socket_id
with all requests. Then, while the SequentialQueue is processing, create a list of Onyx updates from each response. Once the SequentialQueue is empty, batch update Onyx. Together with the backend changes to send the socket_id
with Pusher events, preventing them from being delivered to the requesting client, and sending Onyx updates for the requesting client back in the response, the replay effect is solved.
Fixed Issues
$ https://github.com/Expensify/Expensify/issues/270219
PROPOSAL: GH_LINK_ISSUE(COMMENT)
Tests
Without backend changes
Response Onyx updates work
- Sign in with any account
- Open the Network tab
- Send a message in any chat
- Mark the message as unread
- Navigate to another chat
- Navigate back to the chat
- Verify that the message is now read
- Verify that the onyxData returned in the last OpenReport response contains an update for the
lastReadTime
. - Verify that the
queuedOnyxUpdates
key is empty in the Onyx DB: Application > IndexedDB > OnyxDB > keyvaluepairs > Search forqueuedOnyxUpdates
and verify that nothing comes up
Pusher Onyx updates work
- Sign in with account A
- Sign into account B on another device, making sure both devices remain active with the app open
- Send a message from A to B
- Verify that B receives the message from A
- [x] Verify that no errors appear in the JS console
Offline tests
Without backend changes
Send offline messages
- Log in with any account
- Go offline
- Send a message in an existing chat with account A
- Send a message in an existing chat with account B
- Go back online
- Verify that both messages sent successfully and aren't grayed out.
With backend changes from the Web-Expensify PR
Workspace rename
- Log in with any account
- Go offline
- Create a new worksapce
- Rename the workspace Workspace
- Go back to the settings
- Go back online
- Verify that the workspace shows the new name
New chat
- Log in with any account
- Go offline
- Create a new DM with someone you have never chatted with before
- Send a message
- Go back online
- Verify that the messages remain in order
Workspace member
- Open the app > Open any Workspace > Go to Manage members
- Go offline
- Invite and remove a member while offline
- Go online
- Verify that the member is removed when you go back online and remains removed
QA Steps
Most of the QA will be handled with the backend PR QA, so just run the Pusher Onyx updates work and Send offline messages tests.
- [ ] Verify that no errors appear in the JS console
PR Author Checklist
- [X] I linked the correct issue in the
### Fixed Issues
section above - [X] I wrote clear testing steps that cover the changes made in this PR
- [X] I added steps for local testing in the
Tests
section - [X] I added steps for the expected offline behavior in the
Offline steps
section - [X] I added steps for Staging and/or Production testing in the
QA steps
section - [X] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
- [X] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
- [X] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
- [X] I added steps for local testing in the
- [X] I included screenshots or videos for tests on all platforms
- [X] I ran the tests on all platforms & verified they passed on:
- [X] Android / native
- [X] Android / Chrome
- [X] iOS / native
- [X] iOS / Safari
- [X] MacOS / Chrome / Safari
- [X] MacOS / Desktop
- [X] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
- [X] I followed proper code patterns (see Reviewing the code)
- [X] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e.
toggleReport
and notonIconClick
) - [X] I verified that comments were added to code that is not self explanatory
- [X] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
- [X] I verified any copy / text shown in the product is localized by adding it to
src/languages/*
files and using the translation method- [X] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:
- [X] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
- [X] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the
Waiting for Copy
label for a copy review on the original GH to get the correct copy. - [X] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
- [X] I verified the JSDocs style guidelines (in
STYLE.md
) were followed
- [X] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e.
- [X] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
- [X] I followed the guidelines as stated in the Review Guidelines
- [X] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like
Avatar
, I verified the components usingAvatar
are working as expected) - [X] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
- [X] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
- [X] I verified that if a function's arguments changed that all usages have also been updated correctly
- [X] If a new component is created I verified that:
- [X] A similar component doesn't exist in the codebase
- [X] All props are defined accurately and each prop has a
/** comment above it */
- [X] The file is named correctly
- [X] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
- [X] The only data being stored in the state is data necessary for rendering and nothing else
- [X] For Class Components, any internal methods passed to components event handlers are bound to
this
properly so there are no scoping issues (i.e. foronClick={this.submit}
the methodthis.submit
should be bound tothis
in the constructor) - [X] Any internal methods bound to
this
are necessary to be bound (i.e. avoidthis.submit = this.submit.bind(this);
ifthis.submit
is never passed to a component event handler likeonClick
) - [X] All JSX used for rendering exists in the render method
- [X] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
- [X] If any new file was added I verified that:
- [X] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
- [X] If a new CSS style is added I verified that:
- [X] A similar style doesn't already exist
- [X] The style can't be created with an existing StyleUtils function (i.e.
StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG
)
- [X] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like
Avatar
is modified, I verified thatAvatar
is working as expected in all cases) - [X] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
- [X] If a new page is added, I verified it's using the
ScrollView
component to make it scrollable when more elements are added to the page. - [X] If the
main
branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTest
steps. - [X] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.
Screenshots/Videos
The changes are platform independent so I only tested on web.
Web
Without backend changes
Response Onyx updates work
https://user-images.githubusercontent.com/26260477/234414461-faa0b955-2191-47dc-a2be-c57cfcb4aff5.mov
Pusher Onyx updates work
https://user-images.githubusercontent.com/26260477/234414487-58b71787-b6d0-4234-be18-5a4915a140e3.mov
Send offline messages
https://user-images.githubusercontent.com/26260477/234414511-979c9fa5-887d-4b04-b0cc-0c3e328a8acd.mov
With backend changes from the Web-Expensify PR
Workspace rename
https://user-images.githubusercontent.com/26260477/231017362-c0de2f5c-d2bf-4f0a-b82f-4043495f5661.mov
New chat
https://user-images.githubusercontent.com/26260477/231245296-78be67a4-e487-4d89-b364-568a9d7f469a.mov
Workspace member
https://user-images.githubusercontent.com/26260477/231246769-9ce6c706-d342-4897-9864-9cee80e51ebc.mov
Mobile Web - Chrome
Mobile Web - Safari
Desktop
iOS
Android
Adding myself as a reviewer since I'm interested in this change 👀
@aimane-chnaif @cristipaval One of you needs to copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]
Woohoo this is ready for review @jasperhuangg!
Bump @aimane-chnaif, @cristipaval
started reviewing
Updated, thanks for the reviewing Jasper! Can we get this merged today? It's been up for review for almost a week. @aimane-chnaif will you be able to finish your review today? @cristipaval are you planning to review?
will you be able to finish your review today?
yes, will do

This still happens to me (around 50s in video)
https://user-images.githubusercontent.com/96077027/235774325-978b4129-89ba-4181-ab10-e2cc3fa90db3.mov
And what is Web-Expensify PR status? Deployed to staging or production or not deployed yet?
@aimane-chnaif the Web-E PR is held on this PR. Please only run the tests without the backend changes, or message one of the internal employees on this PR and we can set up a ngrok url for you to hit our dev environment with the backend changes. Please review again today.
@neil-marcellini ok, does https://github.com/Expensify/App/pull/16545#issuecomment-1532087608 also require backend deploy?
Yes. The replay effect isn't fixed for any case without the backend portion as well
Looks good so far "without backend changes".
So Response/Pusher Onyx updates and offline tests work correctly.
queuedOnyxUpdates
onyx data is synced well.
One thing I noticed that this code is run once more unnecessarily at the end after bulk requests (so 2 times total)
@neil-marcellini please pull main as branch is very behind
message one of the internal employees on this PR and we can set up a ngrok url for you to hit our dev environment with the backend changes
Can anyone please help me on this to test with backend changes from the Web-Expensify PR? cc: @cristipaval @jasperhuangg @neil-marcellini
@jasperhuangg says he will help you test. I'm going to merge main now. I'll also look into why the finally block runs twice.
One thing I noticed that this code is run once more unnecessarily at the end after bulk requests (so 2 times total)
@aimane-chnaif I added some console logs and did some debugging. I think you probably encountered this in the case of the Response Onyx updates work test right? Regardless I tested by opening a single report once.
What I found is that we make two identical API.write calls to OpenReport when the report screen updates and when the ReportActionsView mounts. However, looking at the Network tab I only saw one request get made, which means we are actually de-duplicating identical requests somewhere. When we finishing processing the first request we remove all persisted requests here that are equal to the request we want to remove. Since we have two identical OpenReport requests they both get removed as you can see here.
Full OpenReport logs
Neil API.write OpenReport
SequentialQueue.js:102 Neil push request = {command: 'OpenReport', data: {…}, successData: Array(1), failureData: Array(1)}
SequentialQueue.js:54 Neil flush
Log.js:61 [info] Making API request - {"command":"OpenReport"}
Log.js:61 [info] Navigating to route - {"path":"/r/4548882972288752"}
index.js:865 [Violation] Added non-passive event listener to a scroll-blocking 'wheel' event. Consider marking event handler as 'passive' to make the page more responsive. See https://www.chromestatus.com/feature/5745543795965952
setupWebWheelHandler @ index.js:865
componentDidMount @ index.js:834
commitLayoutEffectOnFiber @ react-dom.development.js:23204
commitLayoutMountEffects_complete @ react-dom.development.js:24578
commitLayoutEffects_begin @ react-dom.development.js:24564
commitLayoutEffects @ react-dom.development.js:24502
commitRootImpl @ react-dom.development.js:26779
commitRoot @ react-dom.development.js:26638
performSyncWorkOnRoot @ react-dom.development.js:26073
flushSyncCallbacks @ react-dom.development.js:12009
flushSyncCallbacksOnlyInLegacyMode @ react-dom.development.js:11988
scheduleUpdateOnFiber @ react-dom.development.js:25438
enqueueSetState @ react-dom.development.js:13264
Component.setState @ react.development.js:354
keyChanged @ web.development.js:624
eval @ web.development.js:863
Report.js:382 Neil API.write OpenReport
SequentialQueue.js:102 Neil push request = {command: 'OpenReport', data: {…}, successData: Array(1), failureData: Array(1)}
Log.js:61 [info] [Pusher] Attempting to subscribe to channel - {"channelName":"private-report-reportID-4548882972288752-fedb07a3eb09464e90c6d179634afdc5","eventName":"client-userIsTyping"}
Log.js:61 [info] [PusherAuthorizer] Attempting to authorize Pusher - {"channelName":"private-report-reportID-4548882972288752-fedb07a3eb09464e90c6d179634afdc5"}
Log.js:61 [info] Making API request - {"command":"AuthenticatePusher"}
Timing.js:46 Timing:development.new.expensify.switch_report.warm 366
Log.js:61 [info] Finished API request - {"command":"AuthenticatePusher","jsonCode":200,"requestID":"4cVwdP"}
Log.js:61 [info] [PusherAuthorizer] Pusher authenticated successfully - {"channelName":"private-report-reportID-4548882972288752-fedb07a3eb09464e90c6d179634afdc5"}
index.js:149 [Violation] Added non-passive event listener to a scroll-blocking 'wheel' event. Consider marking event handler as 'passive' to make the page more responsive. See https://www.chromestatus.com/feature/5745543795965952
componentDidMount @ index.js:149
commitLayoutEffectOnFiber @ react-dom.development.js:23204
commitLayoutMountEffects_complete @ react-dom.development.js:24578
commitLayoutEffects_begin @ react-dom.development.js:24564
commitLayoutEffects @ react-dom.development.js:24502
commitRootImpl @ react-dom.development.js:26779
commitRoot @ react-dom.development.js:26638
performSyncWorkOnRoot @ react-dom.development.js:26073
flushSyncCallbacks @ react-dom.development.js:12009
flushSyncCallbacksOnlyInLegacyMode @ react-dom.development.js:11988
scheduleUpdateOnFiber @ react-dom.development.js:25438
enqueueSetState @ react-dom.development.js:13264
Component.setState @ react.development.js:354
setWithOnyxState @ web.development.js:2342
sendDataToConnection @ web.development.js:677
eval @ web.development.js:814
Log.js:61 [info] [Report] Handled onyxApiUpdate event sent by Pusher - [{"onyxMethod":"merge","key":"report_4548882972288752","value":{"lastReadTime":"2023-05-04 18:50:02.239"}}]
Log.js:61 [info] Finished API request - {"command":"OpenReport","jsonCode":200,"requestID":"JpqdCU"}
PersistedRequests.js:32 Neil persisted requests before remove = (2) [{…}, {…}]0: {command: 'OpenReport', data: {…}, successData: Array(1), failureData: Array(1)}command: "OpenReport"data: {reportID: '4548882972288752', emailList: '', appversion: '1.3.10-2', apiRequestType: 'write', pusherSocketID: '464095.159175', …}apiRequestType: "write"appversion: "1.3.10-2"canCancel: trueemailList: ""pusherSocketID: "464095.159175"reportID: "4548882972288752"shouldRetry: true[[Prototype]]: ObjectfailureData: [{…}]successData: [{…}][[Prototype]]: Object1: {command: 'OpenReport', data: {…}, successData: Array(1), failureData: Array(1)}command: "OpenReport"data: apiRequestType: "write"appversion: "1.3.10-2"canCancel: trueemailList: ""pusherSocketID: "464095.159175"reportID: "4548882972288752"shouldRetry: true[[Prototype]]: ObjectfailureData: [{…}]successData: [{…}][[Prototype]]: Objectlength: 2[[Prototype]]: Array(0)
PersistedRequests.js:36 Neil persisted requests after remove = []
SequentialQueue.js:77 Neil process finished
QueuedOnyxUpdates.js:30 Neil queuedOnyxUpdates = (4) [{…}, {…}, {…}, {…}]
SequentialQueue.js:54 Neil flush
SequentialQueue.js:77 Neil process finished
QueuedOnyxUpdates.js:30 Neil queuedOnyxUpdates = (4) [{…}, {…}, {…}, {…}]
2QueuedOnyxUpdates.js:22 Neil QueuedOnyxUpdates clear
So that's super interesting, thank you for pointing that out! The end result is that the SequentialQueue only processes one request but it still calls flush and process twice, which calls this finally block twice. Flush is called twice because every time we push a request we eventually call flush, and for the second request we call flush once the queue is ready here. I'll think about how to fix this, but it's an existing problem and probably not a blocker for this PR.
@aimane-chnaif @jasperhuangg Ok that's resolved now, we only call process and update Onyx when there are actually requests remaining to be processed. I tested and verified that the duplicate calls to OpenReport only result in Onyx.update in the finally block getting called once.
I've performed the back-end changes, they all work as expected! Ignore the Onyx error with creating a workspace, that's a different, unrelated error:
https://user-images.githubusercontent.com/31285285/236373813-d8b34fed-cbb2-4409-ad1d-8847a79ddc2d.mov
Reviewer Checklist
- [x] I have verified the author checklist is complete (all boxes are checked off).
- [x] I verified the correct issue is linked in the
### Fixed Issues
section above - [x] I verified testing steps are clear and they cover the changes made in this PR
- [x] I verified the steps for local testing are in the
Tests
section - [x] I verified the steps for Staging and/or Production testing are in the
QA steps
section - [x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
- [x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
- [x] I verified the steps for local testing are in the
- [x] I checked that screenshots or videos are included for tests on all platforms
- [x] I included screenshots or videos for tests on all platforms
- [x] I verified tests pass on all platforms & I tested again on:
- [x] Android / native
- [x] Android / Chrome
- [x] iOS / native
- [x] iOS / Safari
- [x] MacOS / Chrome / Safari
- [x] MacOS / Desktop
- [x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
- [x] I verified proper code patterns were followed (see Reviewing the code)
- [x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e.
toggleReport
and notonIconClick
). - [x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g.
myBool && <MyComponent />
. - [x] I verified that comments were added to code that is not self explanatory
- [x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
- [x] I verified any copy / text shown in the product is localized by adding it to
src/languages/*
files and using the translation method - [x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
- [x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the
Waiting for Copy
label for a copy review on the original GH to get the correct copy. - [x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
- [x] I verified the JSDocs style guidelines (in
STYLE.md
) were followed
- [x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e.
- [x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
- [x] I verified that this PR follows the guidelines as stated in the Review Guidelines
- [x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like
Avatar
, I verified the components usingAvatar
have been tested & I retested again) - [x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
- [x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
- [x] If a new component is created I verified that:
- [x] A similar component doesn't exist in the codebase
- [x] All props are defined accurately and each prop has a
/** comment above it */
- [x] The file is named correctly
- [x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
- [x] The only data being stored in the state is data necessary for rendering and nothing else
- [x] For Class Components, any internal methods passed to components event handlers are bound to
this
properly so there are no scoping issues (i.e. foronClick={this.submit}
the methodthis.submit
should be bound tothis
in the constructor) - [x] Any internal methods bound to
this
are necessary to be bound (i.e. avoidthis.submit = this.submit.bind(this);
ifthis.submit
is never passed to a component event handler likeonClick
) - [x] All JSX used for rendering exists in the render method
- [x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
- [x] If any new file was added I verified that:
- [x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
- [x] If a new CSS style is added I verified that:
- [x] A similar style doesn't already exist
- [x] The style can't be created with an existing StyleUtils function (i.e.
StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG
)
- [x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
- [x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like
Avatar
is modified, I verified thatAvatar
is working as expected in all cases) - [x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
- [x] If a new page is added, I verified it's using the
ScrollView
component to make it scrollable when more elements are added to the page. - [x] If the
main
branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTest
steps. - [x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.
Screenshots/Videos
Web
https://user-images.githubusercontent.com/31285285/236373813-d8b34fed-cbb2-4409-ad1d-8847a79ddc2d.mov
Mobile Web - Chrome
Mobile Web - Safari
Desktop
iOS
Android
I've performed the back-end changes, they all work as expected!
oh great! I will also test in a few hrs against ngrok url you shared.
I've performed the back-end changes, they all work as expected!
oh great! I will also test in a few hrs against ngrok url you shared.
@aimane-chnaif cool, I was only able to test against Web, so feel free to update my comment above with tests on the other platforms, thanks!
@jasperhuangg I tried with your ngrok url but stuck on login. I never received email with magic code.
Thanks so much @jasperhuangg!
:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.
🚀 Deployed to staging by https://github.com/neil-marcellini in version: 1.3.11-0 🚀
platform | result |
---|---|
🤖 android 🤖 | success ✅ |
🖥 desktop 🖥 | success ✅ |
🍎 iOS 🍎 | success ✅ |
🕸 web 🕸 | success ✅ |
Sorry which App issue?
Sorry which App issue?
Nvm, you already answered to https://github.com/Expensify/App/issues/12775#issuecomment-1538964248
🚀 Deployed to production by https://github.com/roryabraham in version: 1.3.12-0 🚀
platform | result |
---|---|
🤖 android 🤖 | success ✅ |
🖥 desktop 🖥 | success ✅ |
🍎 iOS 🍎 | success ✅ |
🕸 web 🕸 | success ✅ |
@neil-marcellini is the backend PR for this issue merged and deployed to staging? I think it might resolve another issue caused by the replay effect but cannot test without backend changes as you have mentioned above.
@Nikhil-Vats the backend PR is not on staging yet. We'll keep you updated in the tracking issue #12775
Thanks @neil-marcellini. I asked because I found out that the root cause of this issue #18492 is also same. The emojis appear correctly first but then disappear because of the pusher events received. Although what I didn't understand was why the pusher events are always out of order, the response from the first API call is always returned last due to which the emojis on the UI are wrong (only 1 emoji is added since the first AddEmojiReaction pusher event was received at the end and it has only 1 reaction).