API REST source type

[csansoon]

Feature Description

A new source type that allows to use Latitude on API REST endpoints.

Proposal

Usage

An API source type can be configured with the the following schema:

type: api
config:
  method: GET
  headers:
    Authorization: <custom auth header>
    (... other custom headers)

Then, the sql query can be used as follows:

{@config method = 'POST'} -- Optional configuration overrides
SELECT *
FROM {api('https://api.example.com/v1/data')}
WHERE id = 1

Implementation

This api function is a exclusive method for this source type. In compile time, it will make a request based on the function parameters and source configuration, save the response as a temporary parquet file, and include it in the query. Then, the query will be executed as usual by a DuckDB connector.

To be able to parse the response, currently it must follow a given JSON schema previously defined by us.

Improvements

The user should be able to define how to parse the response to a table that we can save into a parquet file. This should be done in the config section of the source configuration, since then it could be replaced for specific queries simply adding a {@config} tag.