qbeast-spark
qbeast-spark copied to clipboard
Qbeast-io
Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!
## Description Solves issue #42 . The problematic with `saveAsTable()` goes beyond a simple **`override`** method. It requires a lot of reworking in the design and implementation of `QbeastDataSource` and...
## Description Adds new feature #98 . :raised_hands: Compaction of small files is coming! ## Type of change In this PR, we present a new feature: compaction of small files....
To be more compatible with underlying Table Formats and set up an easier conversion to Qbeast, we should be able to process files that do not have any Qbeast Metadata...
**What went wrong?** `Table implementation does not support writes` while attempting to save a dataframe to the `metastore_db` using the `saveAsTable` method. **How to reproduce?** ``` val spark = SparkSession...
## Description Changes version for Delta, Hadoop, and Spark, and solves compatibility issues. This is for setting up the build for the upcoming changes in #98 and #4. ## Type...
**What went wrong?** When a sample() is performed against a qbeast dataset, the qbeast sql extension changes the Sample operation into a Filter to: - Pushdown the filter to the...
## What went wrong? Recently, Delta contributors added the functionality to Optimize tables through SQL on the Open Source version. :raised_hands: You can read everything in the issue related: https://github.com/delta-io/delta/commit/e366ccd6179c70dd603c2093a912aacfe719ed00...
Now Github allows embedding videos on the README and other documentation pages. I think it's a good opportunity to include here the jupyther notebook demo. :cat:
**What went wrong?** The following exception should be thrown when you load data that is not in qbeast format or when the path does not exist. It works well when...
The only way of writing in Qbeast Format is to load your data and write it again with Spark Dataframes API. It could be good to have some more easy...