Fix data update using scopes

[anjomro]

Fixes Issue

Closes #773

In the current implementation publishing a data update using scopes doesn't work. This is due to issues with a missing prefixing of the topic.

Changes proposed

Current Faulty Setup

• Topic as part of DataSourceEntry: data:topic (Missing prefix with scope id)
• Pubsub manager topic published to: scope:data:topic (correct topic)

Corrected Setup

• Topic as part of DataSourceEntry: scope:data:topic
• Pubsub manager topic published to: scope:data:topic

Both topics need to be correct. The topic published to needs to be correct as the client only subscribes to the topic in the correct format and would ignore publications to the incorrect topic (format). The topic in the DataSourceEntry needs to be correct since with an incorrect topic the client will receive the message but discard it later due to an unknown topic in the entry.

To accomplish the corrected setup there are two changes necessary. The first is to prefix the topic with the scope id in the scope data update api endpoint. However in the ScopedDataPublisher the topic is prefixed with the scope id again. To avoid publishing to a topic with a duplicate prefix (scope:scope:data:topic) the ServerSideTopicPublisher is used instead of the ScopedTopicPublisher.

Check List (Check all the applicable boxes)

• [x] I sign off on contributing this submission to open-source
• [x] My code follows the code style of this project.
• [ ] My change requires changes to the documentation.
• [ ] I have updated the documentation accordingly.
• [ ] All new and existing tests passed.
• [x] This PR does not contain plagiarized content.
• [x] The title of my pull request is a short description of the requested changes.

Screenshots

n/a

Note to reviewers

n/a

EDIT: Used non scoped data topic publisher to avoid duplicate prefixing instead of modifying ScopedTopicPublisher to avoid interference with policy publishing.

[netlify[bot]]

✅ Deploy Preview for opal-docs canceled.

Name	Link
🔨 Latest commit	4ed51460adebe64aac3b643d3d0e9273e6db9334
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/68b06864a325740008778766

[vorbidan]

IMO, while this addresses the topic validation mismatch for update events on scopes, there are other issues:

1. Data manipulation in transit. The data passed through pub-sub is not the same as passed via POST. This creates a "smart" pipe which manipulates the data while it is also being transferred.
2. Data manipulation happens in multiple places - first we add "data:" prefix in the scopes/api.py (line 350), then we expand topics (which effectively completely changes the list of topics from the original one) in DataUpdatePublisher.get_topic_combos().
3. This data is never persisted in the scopes database, so when client restarts and pulls the initial data (default) it gets the old data entries in addition to mismatched topics.

IMO the /scopes/{scopeId}/update should be abolished and replaced with proper scopes CRUD api, since it is backed by the DB:

1. Create scope: POST to /scopes -> Manipulate topics for every data.entry to add prefixes and expand combos -> Persist in Redis -> Read from Redis -> Pub data and policy events to topics (already manipulated, returned by Read).
2. Read scope(s): GET to /scopes(/{scopeId}) -> Fetch one or more objects from Redis (returns an already manipulated topics field)
3. Update scope: PUT to /scopes/{scopeId} -> Same as Create presently - full object replacement
4. Delete scope: DELETE to /scopes/{scopeId} -> Remove scope from Redis

Why I think it may be a better approach - even though there is a "kludge" with storing not what users posted, we could apply a potentially sophisticated logic which would "normalize" topics to make sure they always have a prefix and are always expanded to combos, so even when users POST an already prefixed/expanded topcis we would recognize it and not double-prefix/double-expand. What it gives us - a uniformity in topics values across the DB and data in events published-to/subscribed-from topics.