App Start chat - User's name containing accent not shown in results if no accent in search query

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

Version Number: 9.0.72.0 Reproducible in staging?: Yes Reproducible in production?: Yes If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: Yes, reproducible on both If this was caught during regression testing, add the test name, ID and link from TestRail: N/A Email or phone of affected tester (no customers): N/A Issue reported by: Applause Internal Team

Action Performed:

precondition: there is a user with a name containing an accent mark e.g. Álex, Timón, D'artgnan

Go to https://staging.new.expensify.com/ and log in
Click on FAB > Start a chat
Enter a user's name without an accent mark, e.g. Álex, Timón

Expected Result:

User is displayed in the search results

Actual Result:

User is not displayed in the the search results unless the accent mark is entered. However, when setting an address in Profile, countries that have diacritics are shown when searching without them.

Workaround:

Unknown

Platforms:

[x] Android: Standalone
[x] Android: HybridApp
[x] Android: mWeb Chrome
[x] iOS: Standalone
[x] iOS: HybridApp
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

https://github.com/user-attachments/assets/0fff9c9a-b77c-45df-8f50-376af6aecd15

View all open jobs on GitHub

Dec 05 '24 23:12 vincdargento

Triggered auto assignment to @strepanier03 (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

Dec 05 '24 23:12 melvin-bot[bot]

Edited by proposal-police: This proposal was edited at 2024-12-06 05:19:16 UTC.

Proposal

Please re-state the problem that we are trying to solve in this issue.

Start chat - User's name containing accent not shown in results if no accent in search query

What is the root cause of that problem?

We don't sanitize the search term to removes diacritical marks like we did in the country selector and we don't sanitize the data during the filter https://github.com/Expensify/App/blob/840c227be7fd907ab096cda7d6f28e1eef4a1d30/src/pages/settings/Profile/PersonalDetails/CountrySelectionPage.tsx#L35

https://github.com/Expensify/App/blob/840c227be7fd907ab096cda7d6f28e1eef4a1d30/src/libs/OptionsListUtils.ts#L1630-L1665

What changes do you think we should make in order to solve the problem?

Sanitize the search term and the data while filtering

Change this code to the following

See updated code

    const matchResults = searchTerms.reduceRight((items, term) => {
        const sanitizedTerm = StringUtils.sanitizeString(term);
    
        const recentReports = filterArrayByMatch(items.recentReports, sanitizedTerm, (item) => {
            const values: string[] = [];
            if (item.text) {
                values.push(StringUtils.sanitizeString(item.text));
            }
    
            if (item.login) {
                values.push(StringUtils.sanitizeString(item.login));
                values.push(StringUtils.sanitizeString(item.login.replace(CONST.EMAIL_SEARCH_REGEX, '')));
            }
    
            if (item.isThread) {
                if (item.alternateText) {
                    values.push(StringUtils.sanitizeString(item.alternateText));
                }
            } else if (!!item.isChatRoom || !!item.isPolicyExpenseChat) {
                if (item.subtitle) {
                    values.push(StringUtils.sanitizeString(item.subtitle));
                }
            }
    
            return uniqFast(values);
        });
    
        const personalDetails = filterArrayByMatch(items.personalDetails, sanitizedTerm, (item) =>
            uniqFast(getPersonalDetailSearchTerms(item).map(StringUtils.sanitizeString))
        );
    
        const currentUserOptionSearchText = items.currentUserOption
            ? uniqFast(getCurrentUserSearchTerms(items.currentUserOption).map(StringUtils.sanitizeString)).join(' ')
            : '';
    
        const currentUserOption = isSearchStringMatch(sanitizedTerm, currentUserOptionSearchText)
            ? items.currentUserOption
            : null;
    
        return {
            recentReports: recentReports ?? [],
            personalDetails: personalDetails ?? [],
            userToInvite: null,
            currentUserOption,
        };
    }, options);

Also we need to update the BE accordingly as well, and sanitize the searchInput param if needed

POC

https://github.com/user-attachments/assets/932de70f-17ef-43c2-a83d-1491c903a1f0

What alternative solutions did you explore? (Optional)

Same solution as the main solution, but use StringUtils.normalizeAccents instead of StringUtils.sanitizeString

Dec 06 '24 04:12 nyomanjyotisa

@strepanier03 Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

Dec 09 '24 09:12 melvin-bot[bot]

I am having a hard time confirming the expected behavior as well on Mac/Chrome.

I created an account with the name Álex Timón D'artgnan.

When I search any portion of Álex Timón D'artgnan, or the whole name, I get no results.
It's also true I get no results when searching Alex Timon Dartgnan either.

Dec 10 '24 22:12 strepanier03

@strepanier03 Eep! 4 days overdue now. Issues have feelings too...

Dec 16 '24 09:12 melvin-bot[bot]

@strepanier03 Still overdue 6 days?! Let's take care of this!

Dec 18 '24 09:12 melvin-bot[bot]

Job added to Upwork: https://www.upwork.com/jobs/~021869454978810524666

Dec 18 '24 18:12 melvin-bot[bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @c3024 (External)

Dec 18 '24 18:12 melvin-bot[bot]

@strepanier03 @c3024 this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

Dec 19 '24 09:12 melvin-bot[bot]

There seems to be a problem with logging in now. I will check the proposals later and update.

Dec 19 '24 15:12 c3024

We need to sanitize the text in the returned results to ensure they appear correctly in the search results for the search text.

@nyomanjyotisa 's RCA and proposal LGTM for the frontend!

Backend needs to normalize accents and send relevant results too!

👀 🎀 👀 C+ reviewed.

Dec 23 '24 06:12 c3024

🎀 👀 🎀

Dec 23 '24 06:12 c3024

Triggered auto assignment to @MonilBhavsar, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

Dec 23 '24 06:12 melvin-bot[bot]

@strepanier03, @MonilBhavsar, @c3024 Whoops! This issue is 2 days overdue. Let's get this updated quick!

Dec 26 '24 09:12 melvin-bot[bot]

Looking here...

Dec 26 '24 20:12 MonilBhavsar

Isn't the root cause that backend is not returning expected data and just making the backend change will fix this issue, or I am missing something?

Dec 27 '24 04:12 MonilBhavsar

Searching with the term ‘Alex’ for already locally existing reports containing ‘Álex’ on the frontend fails because we are not normalizing the search term, and the filtering logic on the frontend excludes the accented results.

https://github.com/Expensify/App/blob/dae99d19e5886ddbe12d9fec04ce43104fe1449e/src/libs/OptionsListUtils.ts#L1652-L1658 2. When there are many reports, such as in a high-traffic account, not all reports are loaded on login. In this case, the backend sends reports matching the search term via SearchForReports. Currently, the backend does not send results for ‘Álex’ when searching with ‘Alex’. The backend results are merged into the existing reports, and the filtering for results matching the search term happens in the frontend code specified above.

So, we need a frontend fix for (1) and a backend fix for (2).

Dec 30 '24 07:12 c3024

Thanks for clarifying! I'll take a look and get back here

Dec 30 '24 15:12 MonilBhavsar

@strepanier03 @MonilBhavsar @c3024 this issue is now 4 weeks old, please consider:

Finding a contributor to fix the bug
Closing the issue if BZ has been unable to add the issue to a VIP or Wave project
If you have any questions, don't hesitate to start a discussion in #expensify-open-source

Thanks!

Jan 02 '25 09:01 melvin-bot[bot]

@strepanier03, @MonilBhavsar, @c3024 Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

Jan 02 '25 09:01 melvin-bot[bot]

Missed posting the video in the last comment. This shows that the new results are merged into the existing reports and the filtering logic based on the search term is on frontend.

https://github.com/user-attachments/assets/fc96e781-5035-4c82-a693-cf028748204a

Jan 03 '25 09:01 c3024

Okay thanks! Let's move ahead and make the frontend fix to prevent filtering already fetched reports. I'll meanwhile figure out the backend fix.

Jan 03 '25 17:01 MonilBhavsar

📣 @nyomanjyotisa 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link Upwork job Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻 Keep in mind: Code of Conduct | Contributing 📖

Jan 03 '25 17:01 melvin-bot[bot]

@nyomanjyotisa it would be great to add automated tests for this use case

Jan 03 '25 17:01 MonilBhavsar

This went out in the deploy but I do want to highlight the fact that it wasn't working for our QA team: https://github.com/Expensify/App/pull/55698#issuecomment-2619542377

So there might be some revisiting necessary in this issue.

Jan 29 '25 05:01 yuwenmemon

Reviewing label has been removed, please complete the "BugZero Checklist".

Jan 29 '25 06:01 melvin-bot[bot]

The solution for this issue has been :rocket: deployed to production :rocket: in version 9.0.90-6 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:

https://github.com/Expensify/App/pull/55698

If no regressions arise, payment will be issued on 2025-02-05. :confetti_ball:

For reference, here are some details about the assignees on this issue:

@nyomanjyotisa requires payment automatic offer (Contributor)
@c3024 requires payment through NewDot Manual Requests

Jan 29 '25 06:01 melvin-bot[bot]

@c3024 @strepanier03 @c3024 The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]

Jan 29 '25 06:01 melvin-bot[bot]

@nyomanjyotisa and @c3024 - Can one of you check on Yuwen's comment here and confirm if there's anything more that needs to happen here?

If not, feel free to complete the checklist and I'll handle the reg test if you suggest one. Then I'll handle payment.

Feb 05 '25 23:02 strepanier03

@yuwenmemon

When the contact is not available on the client, it is searched for on the server. A backend fix is required for that, as specified here.

When the contact is available on the client, the filtering works fine, as seen in the initial part of the video shared. So, I think there is nothing left to be handled on the frontend here.

cc: @strepanier03

Feb 06 '25 03:02 c3024