dlt icon indicating copy to clipboard operation
dlt copied to clipboard

rest_api: allow any type in incremental cursor values

Open willi-mueller opened this issue 5 months ago • 0 comments

Feature description

Right now, we require the initial_value, end_value, and last_value_func in IncrementalArgs to be str.

Changing existing test cases to int also proves that this is supported. Probably datetime is supported too.

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

No response

Proposed solution

The types would look as follows:

class IncrementalArgs(TypedDict, Generic[TCursorValue], total=False):
    cursor_path: str
    initial_value: Optional[TCursorValue]
    last_value_func: Optional[LastValueFunc[TCursorValue]]
    primary_key: Optional[TTableHintTemplate[TColumnNames]]
    end_value: Optional[TCursorValue]
    row_order: Optional[TSortOrder]
    allow_external_schedulers: Optional[bool]
class IncrementalRESTArgs(IncrementalArgs[Any], total=False):
    convert: Optional[Callable[..., Any]]

The validation for the RESTAPIConfig needs to be adapted to support generic types.

Related issues

No response

willi-mueller avatar Aug 28 '24 12:08 willi-mueller