datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Pass CreateView options from sqlparser to logical_plan

Open JanKaul opened this issue 1 year ago • 0 comments

Is your feature request related to a problem or challenge?

I'm trying to implement Iceberg (Materialized) Views with Datafusion. For the 'CREATE VIEW' statement I require additional information about the storage location etc. Currently the LogicalPlan node for CreateView looks like this:

/// Creates a view.
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct CreateView {
    /// The table name
    pub name: TableReference,
    /// The logical plan
    pub input: Arc<LogicalPlan>,
    /// Option to not error if table already exists
    pub or_replace: bool,
    /// SQL used to create the view, if available
    pub definition: Option<String>,
}

It doesn't contain any options passed to the view. This makes sense because Datafusion doesn't support any options. The ast node from the sqlparser crate looks like this:

    /// ```sql
    /// CREATE VIEW
    /// ```
    CreateView {
        or_replace: bool,
        materialized: bool,
        /// View name
        name: ObjectName,
        columns: Vec<ViewColumnDef>,
        query: Box<Query>,
        options: CreateTableOptions,
        cluster_by: Vec<Ident>,
        /// Snowflake: Views can have comments in Snowflake.
        /// <https://docs.snowflake.com/en/sql-reference/sql/create-view#syntax>
        comment: Option<String>,
        /// if true, has RedShift [`WITH NO SCHEMA BINDING`] clause <https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_VIEW.html>
        with_no_schema_binding: bool,
        /// if true, has SQLite `IF NOT EXISTS` clause <https://www.sqlite.org/lang_createview.html>
        if_not_exists: bool,
        /// if true, has SQLite `TEMP` or `TEMPORARY` clause <https://www.sqlite.org/lang_createview.html>
        temporary: bool,
    }

It contains an "options" field where additional options can be defined which I could use to implement the Iceberg Views.

Describe the solution you'd like

It would be great if we could introduce a new "options" field for the CreateView LogicalPlan node, like so:

/// Creates a view.
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct CreateView {
    /// The table name
    pub name: TableReference,
    /// The logical plan
    pub input: Arc<LogicalPlan>,
    /// Option to not error if table already exists
    pub or_replace: bool,
    /// SQL used to create the view, if available
    pub definition: Option<String>,
    /// CreateView options
    pub options: Option<Vec<SqlOption>>
}

This way we could pass the SQL options from the ast to the LogicalPlan node.

Describe alternatives you've considered

The alternative would be to define a new UserDefinedLogicalNode and implement a new SQL planner around SqlToRel. However, these could not be used with the Datafusion SessionContext and/or SessionState limiting the ability to integrate it with Datafusion.

Additional context

No response

JanKaul avatar Aug 29 '24 07:08 JanKaul