verified-sources icon indicating copy to clipboard operation
verified-sources copied to clipboard

rest_api: accept `DltResource` instances in resource list

Open rudolfix opened this issue 9 months ago • 1 comments

Source name

rest_api

Describe the data you'd like to see

We could combine instances of regular resources with a. Such config could look like this:

@dlt.resource(name="stories")
def read_stories(limit: int = 10):
    ...


@dlt.source(name="hacker_news_source", max_table_nesting=2)
def hacker_news_source(
    base_url: str = dlt.config.value,
) -> List[DltResource]:

    # source configuration
    source_config: RESTAPIConfig = {
        "client": {
            "base_url": base_url,
        },
        "resources": [
            {
                "name": "item",
                "table_name": "item",
                "endpoint": {
                    "path": "/item/{id}.json",
                    "data_selector": "$",
                    "paginator": "single_page",
                    "params": {
                        "id": {
                            "type": "resolve",
                            "field": "$",
                            "resource": "stories"
                        }
                    },
                },
            },
            read_stories(20)
        ],
    }

    return rest_api_source(source_config)

Above we add resource instance with the name "stories" and also attach it to a transformer that is fully declared

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

Yes, I'm ready.

dlt destination

No response

Additional information

No response

rudolfix avatar May 21 '24 15:05 rudolfix