Synced History entries are not displayed when enrolled in the Treatment branch of the `Firefox iOS Search Highlights` experiment

[SoftVision-CiprianMuresan]

Preconditions

• Have the Firefox Release 104 (14776) installed through TestFlight.
• Have a Firefox Account with History, Bookmarks, and Remote Tabs.

Steps to reproduce

1. Install Firefox Release from TestFlight.
2. Connect to the Firefox Account from prerequisites.
3. Set Firefox to read experiment from the Preview collection using the Secret Menu.
4. Enroll in the Treatment branch of the Firefox iOS Search Highlights experiment.
5. Fully close and reopen the app.
6. Search for a keyword that would give you results from the History that was synced using the Firefox Account.
7. Observe the behavior.

Expected behavior

• At most 3 History search results are displayed in the awesome bar suggestions.

Actual behavior

• No History search results are displayed.

Device & build information

• Device: iPhone 11 - iOS 15.5
• Device: iPhone XR - iOS 15.3.1
• Build version: Firefox Release 104 (14776), Firefox Release 103.0 (13910), Firefox Release 102.0 (13909)

Notes

• The issue is not reproducible on the control branch, in this case all History related results are displayed.
• The issue is not reproducible with History entries that were created using the device before or after enrolling in the Treatment branch.
• Link to the experiment page: Firefox iOS Search Highlights

Attachments:

https://user-images.githubusercontent.com/18304511/182369789-2345b868-294d-4371-8949-499880a89795.mp4

┆Issue is synchronized with this Jira Task

[nishant2718]

Hey @SoftVision-CiprianMuresan ! I suspect that this is occurs outside of experimentation as well. I've checked on the latest beta and main (but we can check on the now released v104 in prod as well), and I don't get an expected result.

Can you also double check if this is reproducible on the latest main?

I think what's happening is that the sync isn't completed in time. We don't sync all account info often so the fix could be to sync more, but that burdens the system and user's device of course.

I can also check using Proxyman to see when these network calls occur (or don't in our case)

[data-sync-user]

➤ Nishant Patel commented:

Ciprian Muresan Wondering if you had a chance to look into this

[SoftVision-CiprianMuresan]

Hey @nishant2718, sorry it took so long, we've had a lot of things come at the team at once. I've rechecked the issue on the latest Firefox Release v104 (15741) downloaded from the AppStore.

• After enrolling in the experiment I've logged into a Firefox Account that has some History.
• I've left the device alone for a couple of minutes to ensure that the Sync between device was done naturally.
• After this, I've searched for a keyword that should have given me history results and I've had no results displayed.
• I've confirmed that all Links have been synchronized by navigating to the History section in the App.

I've also tried to manually force a synchronization both from the Desktop I'm synced with and on the App I'm trying to see the behavior on, but this

I've also tried the same on the latest main (59568c9) and it's reproducible there as well, it's also reproducible on the control branch in this case, but I think that's because the default feature configuration on the developer channel is search-highlights: true and the experiment doesn't specifically turn the feature off in this case.

As far as I can tell, the issue is reproducible whenever search-highlights is set to true.

[nishant2718]

Thanks @SoftVision-CiprianMuresan ! These details are really helpful - I'll dig deeper into this one.

[nishant2718]

Had a look and you're right - syncing isn't the issue. We specifically ignore history data if the experiment is on.

I think we might have removed history results intentionally, based on https://mozilla-hub.atlassian.net/browse/FXIOS-2903 with https://github.com/mozilla-mobile/firefox-ios/pull/10863.

We show highlights only, and items that qualify as "HistoryHighlights" are shown. @dnarcese what should we do in this case? Do you want synced history to show up again, along with history highlights? If so, do we continue limiting to three results?

[dnarcese]

Synced history should be part of the history highlights. What do you mean by we might have removed history results intentionally?

[dnarcese]

Learnings after discussion: Currently, synced history metadata is not being stored, so the algorithm for highlights will likely not return any synced history items. This is also a bug in the android implementation. Because of this, I am going to put a pause on launching this experiment until we can improve the user experience.

[github-actions[bot]]

This issue has been automatically marked as stale. Has the issue been fixed, or does it still require the community's attention? Please leave any comment to keep this issue opened. It will be closed automatically if no further update occurs in the next 30 days. Thank you for your contributions!

[github-actions[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue linking to this one, with up-to-date information. Thank you!