faro-web-sdk
faro-web-sdk copied to clipboard
grafana
Persistent Local Storage for Telemetry with Configurable Limits
Description
I'd like to propose the development of an enhanced telemetry buffering mechanism for applications running on various devices (tablets, phones, laptops, etc.) that may have an intermittent network connection. This feature aims to ensure the continuous collection and preservation of telemetry data even in offline scenarios or when the telemetry endpoint is unavailable.
Proposed solution
Offline Functionality: The client application should continue to capture telemetry data even when it is offline or the telemetry endpoint is inaccessible.
Persistent Storage: The buffered telemetry data should be stored in a persistent storage medium, thereby enabling it to survive application restarts.
Configurable Limits: Provide options to limit the amount of locally stored telemetry data, ideally based on the number of events or other configurable metrics.
Justification This feature would substantially improve the resilience of our telemetry capabilities. It is particularly crucial for maintaining high data fidelity in environments with intermittent network connectivity, ensuring that we can reliably analyze user interactions and application performance over time.
Context
This is to support an application that seamlessly continues to function in an offline mode on the device. Received data is persisted locally when a network connection is available. Changes to data are transmitted to a remote endpoint when it is accessible.
Hello @bdschaap @codecapitano
Allowing my self to some comments as my team and I already worked a little bit on offline solution. Local Storage has a limit of 10Mb and end user application may already use a big part of it. So local storage may be a dangerous solution event with the configurable limits.
If you do not want any complex dependencies I would recommend using indexedDB. If you can/want add external dependencies RxDB can be a great feat to let the end user choose what kind of storage he want to this offline support.
Here is our simple ADR on offline solutions just to let you know. It is specifically written for our use case so do not take anything the same way than for faro.
Architecture Decision Record: Offline-First solution
- Status: proposed
- Deciders: All developers of Socotec.io
- Date: 08/08/2023
Subject: Architecture Decision Record for Offline First solution with synchronization functionnality
Context and problem statement
In XXXX project, new feature was asked for an entire offline app, while keeping same functionalities (for the most part) as the online version.
We already have some offline functionalities like cacheAPI in XXXX and DexieJS in XXXXX but we wanted to challenge these decisions to find the best tool for these use cases
Considered Options
- DexieJS
- PouchDB
- WatermelonDB
- RxDB
- IndexedDB
- SQL.js
Decision outcome
RxDB is an online first framework agnostic technology written in TypeScript. It uses DexieJS in-browser database as its storage by default.
Positive Consequences
- Extensive API to define and populate schemas that represent storage collections (tables) and its documents (rows)
- Observable-based reactivity system for real time UI
- Extensive query language for complex queries (MongoDB-like)
- Support for different types of in browser storages, including in-memory storage
- Synchronization (replication)
Negative Consequences
- More performance considerations (querying in memory vs in a DB table)
- Important development overhead (Need to implement special API Calls)
Pros and cons of the options
DexieJS
DexieJS is widely used browser storage solution which is also a practial layer on top of IndexedDB
- Good, because widely used, reliable system with small overhead
- Bad, because lacks OOTB tooling (table relationships, replication, limited query language, schemas)
PouchDB
PouchDB is a browser storage solution based on Apache CouchDB
- Bad, because not well maintained
- Bad, requires CouchDB (proprietary)
- Good, because easy, automatic sync system
WatermelonDB
Watermelon is a reactive and asynchronous JS database, originally made for react but now framework reactive
- Good, because it performs well with large datasets
- Bad, because most community use cases are focused on its implemenetation with React
IndexedDB
IndexedDB is a native browser API to store data
- Good, because no overhead
- Bad, because no tooling whatsoever. user must implement custom architecture depending on his usecases
SQL.js
SQLjs is a JS library to run SQLite on the browser
- Good, because SQL is a universal query language that is easy to understand
- Bad, because it's deprecated
Links:
Thank you so much @AMontagu for the ADR. A very nice comparison about the available options. That's awesome!
When deciding for external dependencies we need to consider a bunch of tings, for example the additional size they may add to the Faro bundle and so on.
If possible we try to keep payload low and stick to what the browser offers.
But nevertheless we now have a nice list of options we can take into account when planning the feature.
Thanks and cheers, Marco
