Data Engine: Design principles

[varl]

Data Engine: Design principles

➡️ Split off from #56

This issue is about

This topic is about design principles that drive the design of the API and informs the architecture of the library.

The intent is to yield a collective understanding about the underlying design principles of the Data Engine, to facilitate a joint ownership of it and make it easier for everyone to contribute to the Data Engine.

For example, to understand how to contribute to ui-core a developer must understand the Atomic Design principles. If a developer does not understand those principles it is impossible to contribute to the overall direction of the library, and is limited to contributing with the simplest bug fixes.

The application platform is owned by the entire @dhis2/front-end team, and everyone should be able to contribute to the initiative through a deep understanding of the design principles. The Data Engine is the most sensitive part of the application platform and as such requires a much more careful pace than e.g. the app-shell or the app-scripts.

This issue is not about

• Technical decisions such as which libraries for e.g. caching/normalization to use internally

Principles

Let's use what @amcgee wrote in #56 as a starting point:

1. We are building a framework, not a library. The data access piece is actually a small (albeit important) part of the much bigger holistic re-architecture of "what it means to be a DHIS2 app". Incremental adoption is obviously important, but not if it gets in the way of the platform effort.
2. React is the basis for our modern application ecosystem. While there are some legacy apps and 3rd party applications which do not use React, we only officially support React for all of our developer tools and libraries. We may want to think about officially supporting, say, Angular or Vue in the future, but that's not a primary goal.
3. Redux is not a requirement for DHIS2 apps. As such, it should not be an explicit or implicit dependency of any frontend libraries we build. We should allow DHIS2 applications to use Redux for their own state management, but we shouldn't augment the functionality of our tools outside of redux just to support it.

[ismay]

To me a clear scope would help in establishing what belongs in the data engine and what doesn't. Plus I think it's important that we specify our long term goals clearly, so that we all know what we're trying to achieve with all of our libraries. Currently that is also not exactly clear to me, I have an idea of both things, but I think it's important we establish a very clear and shared understanding of both.

My question would be if we can:

1. Document our long-term goals clearly in the notes repo
2. Establish a clear scope for our new tooling/libraries and put that somewhere where it can't be missed (like top of the readme of the project). So for the data-engine that could be supply an adapter that maps queries to the supplied dhis2 core version, so that queries won't have to be changed but only the version number has to be changed (I assume there's things missing and that it could be expressed much more clearly)

To me that would ease contributions to these projects, prevent scope creep and make discussing these projects in an objective way easier.

[amcgee]

Thanks for clarifying these two topics @varl!

These are the Design Goals from my opening comment in #56 (modified slightly), I think I still stand by them as the Data Engine's raison d'être -

1. Eventually, no DHIS2 application should ever have to make a custom HTTP request to the DHIS2 server - all interaction with the backend should be able to happen through the Data Engine
2. The consumer of the Data Query APIs should not need to know or care about the HTTP requests which might be triggered under the hood. Any queries might be merged into a single request, or multiple requests, or no request at all if the data exists in the local cache.

I think it's also important to restrict the scope of the Data Engine, at least initially, to only DHIS2 server interactions dealing with the API - no external requests or non-API requests (icons, for instance) should be supported. The one minor exception to this is legacy dhis-web-commons/*.action API-like things.

And a new one:

3. It should be possible to create generic, functionally isolated components without knowledge of the application context - things like server discovery, authentication, cache location and policy, etc. should be automatically shared across all components of an application even if those components are defined and implemented by a third-party libary. Also, we shouldn't require that applications explicitly pass this context to the components they include, which would lead to lots of insidious deep prop-passing (a la d2)

[amcgee]

1. Document our long-term goals clearly in the notes repo
2. Establish a clear scope for our new tooling/libraries and put that somewhere where it can't be missed (like top of the readme of the project).

@ismay I 100% agree, let's do this!

[ismay]

I 100% agree, let's do this!

I could create a PR for the app-runtime readme, so we can work on defining the goals/scope of the project in that (and if we do that it'd be nice for other projects as well).

And maybe then we could have an issue in notes for the long-term goals (which we could eventually consolidate to notes). How does that sound?

[Mohammer5]

Relates to dhis2/app-runtime#128

[amcgee]

@ismay that would be great, and great to get your perspective too! We could also add it to the docs (/docs) if we think it would be relevant for end-users (I think it would!)