Allow for phone number validation but also allow for duplicate phone numbers

What feature do you want to improve? Currently if an XForm uses our Phone Widget, this widget both validates the phone number's format and ensures that no two contacts have the same phone number.

MSF-Goma relies on an SMS-workflow of "shared phones". They wish to validate the input but allow for duplicate phone numbers.

Describe the improvement you'd like Can we allow for validation but not uniqueness?

Describe alternatives you've considered MSF-Goma team is currently working around this by using a text input without validation

One more thing, even if you didn't want to allow shared phones, it would be good to show a clear error message. Currently, if you try to input an existing number it just shows a generic error message: "Please enter a valid local number, or use the standard international format ..." Ideally it would say something along the lines of: "This number is already tied to another user"

Thank you!

adding to v3.10

Deprioritised on request of @kennsippell

To clarify - I didn't ask for this to be deprioritised. Rather, I proposed bumping this from v3.10 to allow for a faster release timing since affected projects have a workaround. This seemed to be the lowest priority of the issues in the v3.10 release at the time, but please indicate otherwise if you feel this should be prioritised in our v3.10 covid release.

I proposed bumping this from v3.10 to allow for a faster release timing since affected projects have a workaround.

That's what deprioritising means.

Moved to 3.12.0 to free up engineers to work on 3.11.0.

https://forum.communityhealthtoolkit.org/t/single-phone-number-cant-be-used-for-multiple-primary-contacts/1346/2

Not sure how much engineering effort this will require, but I am tempted to think that now that we are working on enketo, it would be desirable to get this done. Your thought @jkuester ? I also left a comment on the enketo ticket wrt to phone validation.

I definitely agree that this is an important issue to get fixed (and if the validation message is written like we have it in the default forms, it can be very misleading). Unfortunately I do not think this is going to be trivial to fix. As long as we are not changing how this behavior currently works, I think it is best to not try to add this to the current Enketo work (since we have enough happening there already...).

It would be great to know how soon this can be done since with the integrations coming in the field will be one of the client identification details. @abbyad

I don't think this is being worked on, and if it is needed should be discussed in more detail with @craig-landry.

In the meantime, just a couple clarifications:

• If phone number is an identifying detail for a client, would it be expected to be unique?
• In what direction is the integration that you mentioned? Since the validation is happening within the form in CHT, an external program creating a contact or updating the phone number for a client would not be subject to the validation of any phone number. That may help in your scenario, but depends on your workflows.

Hi @abbyad @craig-landry more user stories from ongoing eCHIS UATs and CHV Neo design sessions last week.

1. For the CHV Neo research project, which will require this. SMS messages will be sent to pregnant and postpartum women. This feature is expected to benefit those who share telephone numbers.

2. In the context of the MoH Ke eCHIS, they wanted to make this a required field due to integration with the client registry, where it is one of the identifiers used in addition to the National ID for unique identification. The issue here is that when adding new HH members under the age of 18, the expectation is that the HHH phone number will be reused. And of course for other adults in the HH who can be reached through a common telephone number.

FYI @PhilipNgari @EvelynWaweru @katanu @JenniferMuli

More requests for this today from eCHIS team. https://github.com/moh-kenya/config-echis-2.0/issues/1799

Thanks for the extra references @kennsippell, I (and probably others here) don't have access to that repo. I'll follow up via other avenues though.

There's a solution being investigated for https://github.com/medic/cht-core/issues/8681 which includes the ability to pass parameters to widgets which I think would be helpful here too.

fyi that we are disabling the tel widget for eCHIS Kenya and reverting to regex because of this issue

A project I am supporting needs to register multiple people with the same number as well, and considering switching to the regex method until it is directly supported in the CHT.

@kennsippell, could you share the regex used, and any tips for others considering going this route? Thanks!

hey @abbyad ! @eljhkrr You wrote the regex, would you kindly share?

@abbyad :wave: Definitely @kennsippell, regex validation expression is regex(., '^(\+254|0)([7][0-9]|[1][0-1]){1}[0-9]{1}[0-9]{6}$') remember to use string type if you need +, otherwise integer will work fine.

For the record, @garethbowen had previously mentioned that functionality from https://github.com/medic/cht-core/issues/8681 might prove useful for actually addressing this ticket. I just wanted to note here that after a cursory review of the code in question, I think that now it should be trivial to add support for toggling off the checking of duplicate phone number on a per-field basis (based on configuration provided in the xform).

In https://github.com/medic/cht-core/issues/8681, we added support to the CHT api server for dynamically loading custom data values from xform instance xml elements in the cht: namespace. This is currently being used by the new implementation of the countdown-timer to load the default duration for the timer from the custom instance::cht:duration column in the xlsxform. However, the server code was intentionally written to be generic such that it will load any value from a instance::cht:### column into a corresponding HTML attribute (e.g. data-cht-###) on the generated element for the form field in question.

This means that, if I am correct, we could just have a custom xlsxform column that could be set to toggle off the dup-checking. Then, in the phone widget code, we would just check for the attribute value in the widget constructor and set a flag that would toggle of f the dup checking in the tel.validate function.

---

Unfortunately, I do not have the same confidence that this custom attribute functionality will provide a simple way to improve the validation message that is shown when the field fails validation. (So, there would still be no good way to provide different messages for invalid vs duplicate numbers if you have both checks enabled.)

Thank you @eljhkrr for the example regex!

And thanks @jkuester for the detailed update! I'm glad to hear that there is progress on this issue through https://github.com/medic/cht-core/issues/8681, now available in v4.7. When should we reasonably expect an update to the phone widget to allow for toggling off the validation? Do you need any project testers for that? ✋🏽😄

Hey @abbyad ! So fun to see you here ;)

Allies focused working group is committed to taking this issue on ASAP. This likely means by the end of the month (Jun 2024) we hope to have it completed. Watch this space!

@garethbowen I have finally completed a deep dive into this functionality and would appreciate your feedback on this proposed tech design:

Allow validating number without checking for duplicates

We can add a column to the xlsxform (e.g. instance::cht:tel_allow_dupe) that, when populated with true in a tel row, would trigger the phone widget to not run the duplicate checking. (I am not super happy with the column name, so interested in suggestions!)

Currently in the phone widget we are extending the enketo types object to include a tel type with our own custom validation. My proposal is that we add a second tel_allow_dupe object to the types with a validate function that does not run the dupe check. Then during the construction of the phone widget we can check the data-cht-tel_allow_dup attribute value (coming from the xlsxform) and if it is set, we update the input's data-type-xml to be tel_allow_dup. That will trigger the Enketo logic to use our new type to validate the input. This will allow for configuring the dupe checking on a field-by-field basis within each form.

Allow displaying separate constraint messages when checking both validity and duplication

The main challenge when it comes to displaying separate constraint messages for invalid numbers vs duplicate numbers is actually figuring out how to get the separate internationalized messages to display. Trying to cram multiple translations into the same xlsxform cell is going to be messy. Having custom columns for each of the localized messages could be possible (e.g. instance::cht:tel_dupe_msg_en vs instance::cht:tel_dupe_msg_es) but would definitely result in a lot of columns even with just a couple of languages to support (need to have localized versions of both the dupe message and the invalid number message).

Instead, I propose that we take advantage of the fact that we are operating with custom widget code and just load the translated strings from the CHT translation files via the window.CHTCore.Translate service. My assumption is that generic messages for both duplicate and invalid phone numbers would be sufficient for all forms and do not really need to be customizable at the field level. This would give a better out-of-the-box experience since we can ship default translations for the messages and partners would not need to mess with constraint_message values for tel fields in their forms. (But could still customize the generic translated messages if they wanted.)

Functionally, the way I think we should dynamically choose which of the messages to display is by using a MutationObserver to watch the question element for the invalid-constraint class to be added (which Enketo does when it detects a constraint violation). Adding that class to the question is all that is needed to display the constraint message (which already exists as a hidden span on the DOM). So, when we see the invalid-constraint, we can determine which message should be shown and set that onto the span.

One challenge here is that the Enekto validate call (where we do the initial validity/dupe checking) is completely decoupled and isolated from the widget state (or any other useful context). I have not been able to come up with any feasible way to indicate the result of the validate back to the widget. This means that when we see invalid-constraint, we know the validate failed, but we don't know why. However, this is an exception case (not on the happy path), so a bit of extra processing seems acceptable. Re-running phoneNumber.validate is fine, but running the dupe check query again seems excessive. Instead, I think we could keep a global (for the form) Set of duplicate phone number values that have been found (during the validate calls). Then when deciding what constraint message to show, we can just check this set instead of re-running the dup query.

As a part of our custom widget code, we could also fix things so that these error messages show in the form even if there is no value in the constraint column. Currently, if you are using a tel field with no constraint value and the validate fails, the question will turn red (and the value will be rejected), but no message will be shown to the user (even if there is a message in the constraint_message column). This has led to the "standard practice" of setting true() in the constraint column on tel fields. It would be nice to make this unnecessary!

Deprecate the tel type and add support for tel appearance

This is not strictly necessary to get the above functionality to work, but it feels like the appropriate time to do this and should be trivial to achieve! Currently the phone widget is triggered by using the type tel in your xlsxform. This tel type is out of spec (custom to the CHT code). It is partially to blame for our custom XSL code and it might also require custom Pyxform handling...

The ODK "recommended" method of accepting phone numbers in a form is via string type fields with the numbers appearance. Ultimately, string fields with the numbers appearance get transformed in the form HTML to be input elements with type="tel" (which is the same result as our custom tel fields). These inputs with type="tel" will pop the numeric keyboard when highlighted in a mobile browser.

I think we should update the widget to also get selected for string type fields that have both the numbers and the telappearances. This way we can deprecate the tel xlsxform type and eventually remove it. For maximum passivity, we could also use this as the determining factor for whether to trigger the new constraint message logic. For forms still using the old tel type, we would only use the constraint_message set in the form (and not do any dynamic loading).

---

I have confirmed via testing in the code that all of the most fancy functionality I have proposed here is workable in the phone widget

Really appreciate the very deep dive, validating reality of running this in actual code and being transparent on a public GH ticket - thanks @jkuester !

Overall the approach looks solid to me. Some thoughts...

1. You asked for naming suggestions. I think the naming is proving hard in part because you're trying to label a negative (turning validation off). It becomes a lot easier if we default to not checking for uniqueness, and use something like instance::cht:unique_tel. Changing the default is obviously a breaking change but if we do this at the same time as switching to using appearance then we can document the change.
2. Using out-of-the-box translations for the two error cases is going to be necessary IMO. I think that's ok. I imagine if the constraint message is supplied in the form it would override both messages from CHT translations. Is that what you were thinking?
3. The mutation observer feels like a hack but without enketo changes I don't see that we have an option. Let's make sure we e2e test it so we detect regressions...

Overall the original request was just to be able to have phone number validation WITHOUT the uniqueness check, but all the complexity in this solution is regarding trying to give better validation messages when validating uniqueness as well. These are two separate issues. Are we overcomplicating this? Is there a benefit to doing both at the same time? Or can we just work on the validation w/o uniqueness solution and raise a new issue to support multiple validation messages?

Ideally, the issue was to allow for HHs members to re-use the HH head contact. In this, you just need to validate the telno without the uniqueness check. The second part "validation messages"may be an improvement along the way. cc @christinawere @JenniferMuli as you support this convo closely as well.

I have spoken to @garethbowen more about this design and his suggestions and am following up here with conclusions and some additional findings.

A couple of factors understood from additional investigations should greatly reduce the hackiness of supporting dynamic constraint messages:

• Enketo will actually emit an invalidated event from the input element when the validation fails. So, instead of needing a MutationObserver, we can simply listen to $input for invalidated events.
• Also, we should be able to actually cache the validation results from the validate function calls (instead of just keeping a set of dupe phone numbers). This will let us avoid doing any weird re-validation when we detect an invalidated event. Can just check the cache.
• And regarding clearing the cache and memory leaks, etc, I have confirmed that the phone-widget's static globalReset function will get called by the enketo service unload function. This unload function is called at the end of every form workflow. But even if something goes weird and the unload is not called for a form, the form service will preemptively call unload again the next time a form gets rendered. So, the possibility of leaking a cache between forms seems minimal.

---

Here is a brief summary of the proposed functionality (as I understand it):

Old type: tel fields:

• Checks for valid formatting (no change)
• Prevents dups - no way to disable dup check (no change)

New appearance: tel fields:

• Checks for valid formatting
• Allows dups by default - can trigger dupe check by adding instance::cht:unique_tel: true

Both:

• The constraint message will default to the message provided in constraint_message
  • If no constraint_message is provided, and the phone number has invalid formatting or is a dup, will use a message from the server translations that is specific to the issue
    • This is a slight behavior change for the old type: tel fields. Previously, a constraint message would only be shown if a constraint value was set for the field. If there was no constraint, and the validation/dupe-check failed, the field would turn red and be invalidated, but no message would be shown (even if there was a value for constraint_message ). Now, we will show a constraint message when appropriate for the invalid tel regardless of the value in the constraint column.

---

They dynamic constraint message logic can be implemented independent of the dup-checking. https://github.com/medic/cht-core/issues/7514 already exists and seems to be addressed by the proposed behavior.

So, my plan is to make 2 separate (but sequential) changesets. One that addresses https://github.com/medic/cht-core/issues/6390 by supporting appearance: tel and the unique_tel column. Then another to address https://github.com/medic/cht-core/issues/7514 by making the constraint message dynamic.

@jkuester Should this be added to the release milestone?

good point @garethbowen ! I've added it to 4.11