rest_api_resources should support windows with passed incremental.

[valkum]

Feature description

When creating rest_api_resources you should be able to derive a end_value for a given incremental start_value.

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

I am using an API with rest_api_resources that needs both end and start to be defined. The incremental support only allows passing of the start date, you can't dynamically get an end date for a window.

"endpoint": {
            "path": f"api-foo",
            "params": {
                "ActionDateStart": "{incremental.start_value}",
                "ActionDateEnd": n/a,
            },
            "incremental": {
                "cursor_path": "EventDate",
                "initial_value": pendulum.parse("2025-06-16T00:00:00Z"),
                "convert": lambda epoch: epoch.to_date_string(),
                "primary_key": "Id",
                "row_order": "asc",
            },
            "data_selector": "Actions",
            "paginator": PageNumberPaginator(
                total_path="['@total']",
                page_param="Page",
            ),
        },

Proposed solution

Unsure what the best way would be. Having dynamic end_values on Incremental based on a window sounds like a good and generic solution too me.

Other solutions: You should be able to either set a request hook or another way of converting/adding params or body items after the incremental placeholder code got executed.

Related issues

No response

[jukiewiczm-marketer]

I'm missing this too. My rest API resource doesn't allow pulling more than X days per call, and it's kind of impossible to set currently

[jukiewiczm-marketer]

I don't want to ask for too much, but what would be even better is to be able to "replicate" the resource multiple times, so if e.g. I have 100 days to pull but can only pull 20 at once, multiple yields with different parameter sets would actually be created. That would allow for a dynamic "backfill" that is actually stateful with regard to the incremental. Seems like the only way to get this working right now would be to implement my own paginate_resource function and overwrite the __wraps in the resource once it's created. A simple params_transformer parameter that would get the resolved parameter set and return one or many resolved parameter sets would suffice