dlt
dlt copied to clipboard
rest_api: allow any type in incremental cursor values
Feature description
Right now, we require the initial_value, end_value, and last_value_func in IncrementalArgs to be str.
Changing existing test cases to int
also proves that this is supported. Probably datetime
is supported too.
Are you a dlt user?
Yes, I'm already a dlt user.
Use case
No response
Proposed solution
The types would look as follows:
class IncrementalArgs(TypedDict, Generic[TCursorValue], total=False):
cursor_path: str
initial_value: Optional[TCursorValue]
last_value_func: Optional[LastValueFunc[TCursorValue]]
primary_key: Optional[TTableHintTemplate[TColumnNames]]
end_value: Optional[TCursorValue]
row_order: Optional[TSortOrder]
allow_external_schedulers: Optional[bool]
class IncrementalRESTArgs(IncrementalArgs[Any], total=False):
convert: Optional[Callable[..., Any]]
The validation for the RESTAPIConfig needs to be adapted to support generic types.
Related issues
No response