qbeast-spark
qbeast-spark copied to clipboard
Qbeast-io
Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!
QbeastOptions is constructed by a Map of Strings containing (or not) different information for writing in a Qbeast table. The structure of the builder is a Case Class with a...
Qbeast Spark supports reading files not indexed with Qbeast Metadata. There's different situations that can cause a table to have a hybrid state. - Different set of writers. One writes...
## Description Part 1/4 from issue #327. ## Type of change Change on Documentation. ## Checklist: Here is the list of things you should do before submitting this pull request:...
## What went wrong? Running `ALTER TABLE table_name SET TBLPROPERTIES ()` does not persist the changes in the `_delta_log/.` ## How to reproduce? ```scala import org.apache.spark.sql.delta.actions.Action import org.apache.spark.sql.delta.DeltaLog import org.apache.spark.sql.delta.actions.CommitInfo...
## Description We noticed that we were not managing the metadata most efficiently: 1. The delta_log was read multiple times to rebuild the same data structure 2. We started to...
After analyzing the efficiency of distribution functions for indexing (see issue #336 ), we can start implementing the HistogramTransformation. The idea is to build it as another type of transformation,...
As discussed in issue #294 , we should remove the `compact()` operation from the QbeastTable API. Also, right now replication is out of the scope. (See issue #282 ) But...
## Description Fixes issue #441 by setting `dataChange = False` during `optimize()` ## Type of change Bug Fix
Since PR [Issue #405: DataWriter refactory #402](https://github.com/Qbeast-io/qbeast-spark/pull/402), a ConcurrentAppendException started appearing during a write() and a optimize() ``` io.delta.exceptions.ConcurrentAppendException: Files were added to the root of the table by a...
## Description Adds #294 ## Type of change New Feature. The Unindexed Files of a Qbeast Table were only optimizable from the `StagingDataManager` component. After thinking about structure and use...