kairosdb
kairosdb copied to clipboard
Add BigQueue to buffer data on kairos node
The implementation of this will be in two parts.
- A generic support class that lets a data store queue and dequeue datapoints. This class can be used by data stores to queue events if they need to.
- Modify CassandraDatastore to use the above queue to pull data points from.
Another alternative is to put the queue in the Kairos layer above the datastore but, this would require a change to the data store interface potentially break to many things so I'm going with the above solution.
Lets use this issue for further design discussions.
Great. So with this we would get the confirmation that data was stored? or something still can go wrong between bigqueue and cassandra?
There are multiple issues this solves but knowing the data is in C* is not one of them. Here are the goals
- Data is on disk before Kairos returns
- Better deal with bursts of data from clients
- Deal with temporary C* outages
Because we have to batch data to C* I'm not sure how to provide confirmation of when it is there.
I've created this branch for changes related to this issue: https://github.com/kairosdb/kairosdb/tree/feature/bigqueue
All updates for this feature should be made as pull request to the above mentioned branch.