Describe best practices for configuration-based ID assigned by building out example FHIR resources

[pld]

Describe the issue to be researched

There are currently 3 approaches to assigning IDs

1. Generate random non unique IDs (UUIDs count as this but the space is large enough that they are effectively unique)
   • An example of this is calling something like rand-string(length: 6, chars:a-Z0-9)
2. Generate random algorithmically unique IDs, like where the first 2 characters are based on the practitioner
3. Use pre-generated IDs (with whatever properties they have, probably unique)

A detailed discussion on (3.) is here.

Describe the goal of the research

The goal of the research is to prove out a method for approach (3.) and add the related configuration files to the opensrp.io documentation.

Describe the methodology

The steps to achieve this goal are:

1. Create an example resource to hold a list of IDs
   • Looking through nothing is a perfect fit but a Group resource seems sufficient with no members, e.g.
   • here we store the IDs as text in a valueCodeableConcept in the characteristic field
   • the length of characteristic is the number of IDs we have
   • we abuse the quantity field to store an offset to the next ID to use, e.g. whenever we assign an ID in location quantity in the characteristic array we increment the unsigned integer stored in quantity so it points to the next available ID
   • we use managing entity to assign this set of IDs to a Practioner

Group
    .characteristic: [
        .valueCodeableConcept.text = "00001",
        .valueCodeableConcept.text = "00002",
        ...
    ]
    .quantity: 0
    .type: device
    .code.text: "identifier-list"
    .managingEntity: Practioner/AA

2. Modify the sync configuration so that this Group resources is synced when the Practioner it has as its managing entity logs in
   • [@allan-on please add implementation steps, if any, to get this working]
3. Write a Questionnaire that completes an uneditable (but visible) field by entering the ID that's at location quantity in the array of characteristic
4. Write a StructureMap that, upon submission of the above Questionnaire, loads the Group and increments the value in quantity

---

A limitation of the above approach is once you exhaust the IDs in the group, you are out of IDs, there are 2 ways to resolve this, 1) add more IDs to the .characteristic field in the group, or 2) add another group for the Practitioner. (2) seems clearly better because you avoid any conflicts between the .quantity offset in an on-device file and in an on-server file that has new entries in .characteristic. But it means we add complexity to the Questionnaire and StructureMap.

Approach for this is to

• use Group.active to designate that all IDs have been used
• use the IDs in a group with Group.quantity>0 before those with Group.quantity=0

1. on creation in FHIR API set Group.active=true and Group.quantity=0
2. when fetch IDs for the Questionnaire, get the first Group where .code.text = "identifier-list" and .active=true ordered by .quantity and .id
   • this gives us an ordering by groups that are in use (i.e. quantity>0), then by .id if both are unused
3. in the Structure Map use the same fetch logic to figure out which group to increment the .quantity value of

[dubdabasoduba]

@allan-on I took some time on this. I think the Group resource works well though I feel like we are semantically abusing it.

I would have suggested we use the ValueSet and CodeSystem. but they do not have ideal ways to link them to a Practitioner

[allan-on]

A. Representing a batch of IDs

Characteristic based Group resource

The Characteristic-based Group resource will be used to hold a batch of IDs. The group will make use of the following main attributes:

• type = Type distinction for this Group (Allows us to distinguish this Group from Practitioner and household groups).
• active = To indicate whether this batch is in use. Once the IDs are exhausted, this value will be set to false.
• managing-entity = Practitioner whom the batch is assigned to
• count = Quantity (count) of IDs in the batch.
• characteristic = Included list (array) of IDs.
• Group.characteristic.exclude = Included list (array) of IDs.

The search parameters that apply to our use case are: - Group.code - Group.identifier - Group.managingEntity

B. Syncing of IDs

This will be defined in Sync configs so that the IDs are synced along with other resources.

[
  {
    "resource": {
      "resourceType": "SearchParameter",
      "name": "managing-entity",
      "code": "managing-entity",
      "base": [
        "Group"
      ],
      "type": "token",
      "expression": "#managing-entity"
    }
  },
  {
    "resource": {
      "resourceType": "SearchParameter",
      "name": "type",
      "code": "type",
      "base": [
        "Group"
      ],
      "type": "token",
      "expression": "type"
    }
  }
]

• This will allow syncing by managing-entity who is the Practitioner to whom the IDs are assigned.

FHIR Core sync updates

• The SyncListenerManager will be updated to populate the practitioner ID in the SearchParameter expression.
• Override TimestampBasedDownloadWorkManagerImpl so that we can provide a list of syncParams: ResourceSearchParams to allow syncing Group resources by different params e.g. member. organization & count, OR managing-entity & type.

Server-side sync updates

• The FHIR Gateway will be updated to allow syncing of these Group resources by managing-entity by adding a sync filter ignored query entry.

ID assignment via FHIRPath and Code

• We’ll use FHIRPath to query for an available ID and repopulate the value in the Questionnaire. For instance:

"Group.active = true and Group.type = 'device' and Group.name = 'Unique IDs'\"
Group.characteristic.where(exclude=false and code.text='phn').first().value.text

• On submission of the Questionnaire, update the exclude value for the identifier to true to mark the ID as used
  1. Pass the required values as Questionnaire ActionParameters
     1. The Group ID.
     2. The unique ID.
  2. On submitting a Questionnaire, after saving extracted resources:
     1. If we have the unique ID and Group ID, load the Group resources using the id in the ActionParams and update the Characteristic exclude value as false.
     2. If all identifiers are exhausted, update the Group to inactive (active = false)
     3. Save the updated Group resource

Challenges with this updating consumed using StructureMaps

• The major challenge is with marking and ID as used and updating a count/index
  • Retrieving the specific Group resources for updating in a StructureMap isn’t trivial. Updating the resources will require recreating the Group before doing the required update.
  • We’ll need the implementation replicated in multiple StructureMaps for the different projects.

---

Sample

{
	"resourceType": "Group",
	"id": "37312ad4-538e-4535-82d2-ea14f40deeb9",
	"meta": {
		"versionId": "6",
		"lastUpdated": "2023-12-11T09:14:37.220+00:00",
		"source": "#3dd77b3a4b918f39",
		"tag": [
			{
				"system": "https://smartregister.org/location-tag-id",
				"code": "3816",
				"display": "Practitioner Location"
			},
			{
				"system": "https://smartregister.org/care-team-tag-id",
				"code": "3e005baf-854b-40a7-bdd5-9b73f63aa9a3",
				"display": "Practitioner CareTeam"
			},
			{
				"system": "https://smartregister.org/app-version",
				"code": "Not defined",
				"display": "Application Version"
			},
			{
				"system": "https://smartregister.org/practitioner-tag-id",
				"code": "49b72a3d-44cd-4a74-9459-4dc9f6b543fa",
				"display": "Practitioner"
			},
			{
				"system": "https://smartregister.org/organisation-tag-id",
				"code": "41eae946-bdc4-4179-b404-6503ff12f59c",
				"display": "Practitioner Organization"
			}
		]
	},
	"identifier": [
		{
			"system": "http://smartregister.org",
			"value": "37312ad4-538e-4535-82d2-ea14f40deeb9"
		}
	],
	"active": true,
	"type": "device",
	"actual": true,
	"name": "Unique IDs",
	"quantity": 5,
	"managingEntity": {
		"reference": "Practitioner/49b72a3d-44cd-4a74-9459-4dc9f6b543fa"
	},
	"characteristic": [
		{
			"code": {
				"text": "phn"
			},
			"valueCodeableConcept": {
				"text": "1000010001"
			},
			"exclude": false
		},
		{
			"code": {
				"text": "phn"
			},
			"valueCodeableConcept": {
				"text": "1000020002"
			},
			"exclude": false
		} // ...
	]
}

[allan-on]

@pld @dubdabasoduba Here's an update on the results of the R&D that yielded the example that was demoed in the WDF Diabetes Compass app earlier this week ☝🏾

[pld]

In the FHIR Core sync updates section, don't we only need to sync by managing entity, what are the use cases for member.org & count syncing?

[allan-on]

In the FHIR Core sync updates section, don't we only need to sync by managing entity, what are the use cases for member.org & count syncing?

@pld That's how we currently sync households. But in the future, this won't be necessary since we'll be relying on the Location sync strategy enforced by the Gateway

[pld]

Got it, so we can current sync using member.org & count, the extension here is to allow it to sync with managing entity and type, is that correct?

Currently do we create households do we set the managing entity to a practitioner? If not we don't need to also filter on type, but if we do, then I see why we need to distinguish those

[pld]

FHIR Core sync updates

    The SyncListenerManager will be updated to populate the practitioner ID in the SearchParameter expression.

how will it know when to populate this / can it always populate it? it seems useful

Server-side sync updates

    The FHIR Gateway will be updated to allow syncing of these Group resources by managing-entity by adding a sync filter ignored query entry.

What's the filter we're ignoring?

ID assignment via FHIRPath and Code

in ii.a. why are we setting exclude to false? would that make a used ID usable again? I'd think IDs should only be going from unused (false) to used (true).

Challenges with this updating consumed using StructureMaps

    The major challenge is with marking and ID as used and updating a count/index
        Retrieving the specific Group resources for updating in a StructureMap isn’t trivial. Updating the resources will require recreating the Group before doing the required update.
        We’ll need the implementation replicated in multiple StructureMaps for the different projects.

Aren't we updating and saving the group, is that what you mean by "recreating"?

But yea, complex, the simpler way I suggested is to not track exclude per field, but use quantity to mark the number of characteristic IDs that have been used so far, eg if it's 0 you take 0th, if it's n you take the nth, if it's greater than the length of the characteristics we've used all the IDs. This is definitely semantically abusing it, but we're already doing that.

[pld]

@allan-on I took some time on this. I think the Group resource works well though I feel like we are semantically abusing it.

I would have suggested we use the ValueSet and CodeSystem. but they do not have ideal ways to link them to a Practitioner

I like the idea of using ValueSet, looking into that a bit...

[pld]

Looking through ValueSet and CodeSystem both of them have a relatedArtifact field that we can use to link either of these resource to a Practitioner through .relatedArtifact.resource = [the Practitioner]

if we use ValueSet

• we can store the IDs in .expansion.contains, this can store the list of IDs where the ID is in the .code field.
• we can use .expansion.offset to function the same way that I suggested group.quantity function.

if we use CodeSystem

• we can store the IDs in .concept, this can store the list of IDs where the ID is in the .code field.
• we can use .count to function the same way that I suggested group.quantity function.

CodeSystem seems like the simpler option, less nesting. Overall this is an improvement in semantic alignment, but not entirely aligned.

Thoughts?

[dubdabasoduba]

@pld I think the relatedArtifact attribute is available for the FHIR R5 versions of ValueSets and CodeSystems

If we had this working on R4B then I would say we choose from the ValueSet or CodeSystem. I would favour the ValueSet because of the ease of marking the used IDs as inactive.

cc: @allan-on

[pld]

Agree, @allan-on to you have any questions on this approach? This should be able to be a content only approach, let us know if you encounter any challenges with that

[allan-on]

Looking through ValueSet and CodeSystem both of them have a relatedArtifact field that we can use to link either of these resource to a Practitioner through .relatedArtifact.resource = [the Practitioner]

if we use ValueSet

* we can store the IDs in `.expansion.contains`, this can store the list of IDs where the ID is in the `.code` field.

* we can use `.expansion.offset` to function the same way that I suggested `group.quantity` function.

if we use CodeSystem

* we can store the IDs in `.concept`, this can store the list of IDs where the ID is in the `.code` field.

* we can use `.count` to function the same way that I suggested `group.quantity` function.

CodeSystem seems like the simpler option, less nesting. Overall this is an improvement in semantic alignment, but not entirely aligned.

Thoughts?

@pld @dubdabasoduba After comparing the three resources i.e. Group, CodeSystems and ValueSets vis a vis (a) which resource best holds a collection of entities and (b) if that resource supports the OpenSRP use cases/workflows:

• Semantically, the Group resource is fits better IMHO
• CodeSystems and ValueSets (are interrelated) and would be best if we wanted to define coded values that would be referenced or derived from a code system.

[allan-on]

FHIR Core sync updates

    The SyncListenerManager will be updated to populate the practitioner ID in the SearchParameter expression.

how will it know when to populate this / can it always populate it? it seems useful

Server-side sync updates

    The FHIR Gateway will be updated to allow syncing of these Group resources by managing-entity by adding a sync filter ignored query entry.

What's the filter we're ignoring?

ID assignment via FHIRPath and Code

in ii.a. why are we setting exclude to false? would that make a used ID usable again? I'd think IDs should only be going from unused (false) to used (true).

Challenges with this updating consumed using StructureMaps

    The major challenge is with marking and ID as used and updating a count/index
        Retrieving the specific Group resources for updating in a StructureMap isn’t trivial. Updating the resources will require recreating the Group before doing the required update.
        We’ll need the implementation replicated in multiple StructureMaps for the different projects.

Aren't we updating and saving the group, is that what you mean by "recreating"?

But yea, complex, the simpler way I suggested is to not track exclude per field, but use quantity to mark the number of characteristic IDs that have been used so far, eg if it's 0 you take 0th, if it's n you take the nth, if it's greater than the length of the characteristics we've used all the IDs. This is definitely semantically abusing it, but we're already doing that.

How will it know when to populate this / can it always populate it? it seems useful

In the SyncListenerManager we have implementation for substituting the #expression from the sync_config with the actual expression value (using a when expression). So we can add substituting the managingEntity with the logged in practitioner reference as the value.

What's the filter we're ignoring?

Here the idea was to by-pass filtering by Location sync strategy and allow us to use the search param defined in the sync configs.

in ii.a. why are we setting exclude to `false`? would that make a used ID usable again? I'd think IDs should only be going from unused (`false`) to used (`true`). That's correct. Typing mistake. We should set the characteristic.exclude to true to mark it as used.

Aren't we updating and saving the group, is that what you mean by "recreating"? Updating resources using StructureMaps involves defining a new resource in the StructureMap, then reusing the original resource ID i.e. assigning the original ID to the new resource (with updated values) to achieve an update.

But yea, complex, the simpler way I suggested is to not track exclude per field, but use `quantity` to mark the number of characteristic IDs that have been used so far, eg if it's 0 you take 0th, if it's n you take the nth, if it's greater than the length of the characteristics we've used all the IDs. This is definitely semantically abusing it, but we're already doing that.

Using the Group.quantity to mark the number of available characteristic IDs isn't necessarily abusing it. According to the note in the http://hl7.org/fhir/R4B/group-definitions.html:

The quantity may be less than the number of members if some of the members are not active.

We can combine using both the Group.characteristic.exclude to identify which IDs are usable (i.e. give me the first() ID with exclude false) and Group.quantity to show/track the actual number.

cc @pld

[pld]

On whether we use Group, ValueSet, or CodeSystem, let's make this decision based on what is easier to implement and we expect to be easier maintain. Let's target the architecture so that we can use a different resource type in the future, e.g. if we implement it with Group now, and want to change to use ValueSet later, that is not a significant rewrite of code.

Also, on using Group, would we have to do anything special to handle that we use Groups for other purpose now (I guess only to define families), or is that not an issue since the links in the Group will make the 2 different uses of Group clear?

I misunderstood what Group.quantity was used for, but that makes sense. But, if we are using that to refer to the item in the list of characteristics that is the next available ID, why do we need to both with Group.characteristic.exclude? Any characteristic less than quantity we now is used, any greater than is available.

[pld]

I'm closing this, my opinion is that the docs cover the initial version of what I was thinking for this, we can create a new issue with specifics on additional documentation