nfcompose

nfcompose copied to clipboard

Implement compose_cli client support for Tenant and global User management

1

- [ ] Add support for Tenant diff and creation - [ ] Add support for global User management (without diff though?) - in the same swoop we should probably...

s4ke

With our old backends DYNAMIC_SQL_V1 and DYNAMIC_SQL_MATERIALIZED we still store data in a fully materialized manner. Both backends are deprecated as of 2.0.1 and should be removed completely in 2.1.0....

s4ke

see https://www.citusdata.com/blog/2023/07/18/citus-12-schema-based-sharding-for-postgres/

s4ke

Right now we emulate migrations via the compose_cli. These are not transactional, though and if something goes wrong we leave the dataseries broken and might have to do data recovery...

s4ke

- [ ] Prototype UX for a compose_client command and library that polls a dataseries using ?changes_since (without deletion events) - [ ] library functionality - [ ] Design Interface...

s4ke

Currently the compose client cli is intended to be used as a component of bash scripts to handle the logic around diffing then pushing changes of dataseries for migrations. While...

s4ke

In 2.1.0 we left in some code that should allow for read only access to removed backend data. This code was left untested as all test code related to that...

s4ke

We currently don't have an opensourced example project that shows how to use NF Compose and especially with the compose_client.

s4ke

Currently we only support the rather wasteful pattern of generating canonical ids for DataPoints using this logic: https://github.com/neuroforgede/nfcompose/blob/main/skipper/skipper/dataseries/storage/uuid.py ```python def _gen_uuid(data_series_id: Union[uuid.UUID, str], external_id: str) -> str: computed_id = hashlib.sha256(external_id.encode('UTF-8')).hexdigest()...

s4ke

Improve throughput of event delivery

3

https://github.com/neuroforgede/nfcompose/pull/149#issue-2647126878 Follow up here

s4ke