Allow pattern matching in setting the snapshot and delta bundle resource in configure

[alizand1992]

What part of OPA would you like to see improved?

Delta Bundle functionality

Allow pattern matching in setting the snapshot and delta bundle resource in configuration so that there can be separate resource files for each.

Use Case: In cases with no bundle service where S3 is being used to serve the snapshot and delta bundles. Currently Delta bundles have the same name as the snapshot bundle requiring to overwrite the same file with each delta.

Problem: It is not simple to scale up additional instances of OPA because it requires republishing a snapshot to provide the initial bundle state for the new instance.

Describe the ideal solution

Allowing pattern matching in config file where the snapshot follows a different pattern than the delta files. Delta files can be numbered and uploaded to S3 and OPA will read snapshot followed by delta files sequentially which will allow OPA to rebuild the data.

example: snapshot:

• bundle-snapshot.tar.gz Delta:
• bundle-delta-001.tar.gz
• bundle-delta-002.tar.gz
• etc...

Additional Context

OPA is setup with a 10s short poll cycle. Bundle gets generated and uploaded to S3 by a separate service Bundle generator is not aware of when the OPA's polls are happening

[ashutosh-narkar]

The idea here is that when OPA initially starts the bundle server sends a snapshot and they deltas for the next updates. The server would be responsible for maintaining the order of snapshot/delta deliveries. Adding pattern matching in the OPA config won't help. You will have to implement the server code that returns the "right" bundle whenever OPA reaches out.

Bundle generator is not aware of when the OPA's polls are happening

You would need to build a service that is aware of the state of the OPA. OPA will simply reach out for a bundle and your service decides what to send.

[alizand1992]

A little more context on what is going on: We are currently using a single bundle that gets refreshed on an interval. We are using S3 to serve the bundle as it gets refreshed and we do not have a Bundle Service. S3 does not have a functionality to serve the "right" bundle based on the OPAs status; however, the 5 9s availability provided out of the box by S3 is very crucial to our application. Adding this feature to OPA will allow us to continue using S3 and leverage its high availability.

[ashutosh-narkar]

You'll have to build out a service that is aware of the bundle version each OPA currently has and then create the right bundle and serve via S3. You basically need a management plane for your OPAs to make this work. Using vanilla S3 in this case would not serve your purpose I think. OPA takes care of the activation part of the bundle be it snapshot and delta. The order is which these bundles are served is left to the server to decide as each implementation may differ and have different ordering mechanisms.

[anderseknert]

I can see the appeal in a solution like this, as it would allow any "dumb" server to serve delta bundles.

I think a problem with this would be that a new OPA client connecting would need to download both the snapshot bundle and all delta bundles produced since the snapshot was created... this could potentially be thousands of bundle files if running for a long time. Eventually, that would be much slower than just pulling a fresh snapshot bundle.

Any thoughts on that, @alizand1992 ?

[alizand1992]

Yes that was one of the concerns, we decided to solve this by refreshing the snapshot with variable frequency based on the delta count.

The idea is that the bundle generator would know how many deltas it has generated and once it hits a specific limit, it clears them out and starts with a new snapshot.

The second concern is new updates between when the DB read happens for the snapshot and the snapshot is ready would be missed. We think we can solve for that by doing the following: During a refresh, the snapshot itself will be via a DB read, the deltas will catch up by pulling events from a kafka topic using the latest DB record timestamp and by starting from the immediate timestamp after the DB timestamp in the kafka topic.

This approach has a few benefits:

1. Reduced load on the DB
2. Delays limited to OPA poll frequency
3. One step closer to a "smart" server without having to commit to all the dev work at once
4. Minimal work will be done during off peak hours

Please let me know what you think, @anderseknert.

[anderseknert]

Yes that was one of the concerns, we decided to solve this by refreshing the snapshot with variable frequency based on the delta count.

The idea is that the bundle generator would know how many deltas it has generated and once it hits a specific limit, it clears them out and starts with a new snapshot.

Right, so this would require quite some logic in the generator, but fair enough, I suppose :)

The second concern is new updates between when the DB read happens for the snapshot and the snapshot is ready would be missed. We think we can solve for that by doing the following: During a refresh, the snapshot itself will be via a DB read, the deltas will catch up by pulling events from a kafka topic using the latest DB record timestamp and by starting from the immediate timestamp after the DB timestamp in the kafka topic.

Sounds reasonable to me, but just so I understand correctly — this would not be a concern to OPA but your bundle generator, no? Sounds like there is some POC code available somewhere? 🙂 Did you work on having this supported in OPA as well?

[alizand1992]

@anderseknert sorry for the late response. That is correct. They both would be all on the bundle generator side and not the OPA. To make all of this work we would need OPA to be able to read a separate snapshot and also read deltas using a pattern and the bundle generator can take care of the rest.

The bundle generator and OPA with single bundle is implemented and live at the moment but the deltas are not a thing yet. We implemented a POC for deltas but quickly realized it needed to be the same name and here we are :) So no POC yet and it doesn't seem useful for us to implement a it until we know that this is a feature that OPA is interested in. No work has been done in OPA. Unfortunately no one in our team has experience in Go.

[anderseknert]

Thanks! I could definitely see this as useful.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[ashutosh-narkar]

Feel free re-open this but providing a single bundle endpoint to OPA and having the server taking care of ordering of bundles etc. seems like the right approach.