dlt icon indicating copy to clipboard operation
dlt copied to clipboard

Could not set next_item_mode based on documenation

Open waterworthd-cim opened this issue 1 month ago • 0 comments

I'm not sure if I'm doing something wrong but the documentation for fifi vs round_robin (https://dlthub.com/docs/reference/performance#resources-extraction-fifo-vs-round-robin) states

[extract] # global setting
next_item_mode="round_robin"

[sources.my_pipeline.extract] # setting for the "my_pipeline" pipeline
next_item_mode="fifo"

But this didn't seem to have any effect, and I verified by breaking in from_pipes method of the PipeIterator class, the next_item_mode is still set to "round_robin"

What did work was

[sources.my_pipeline_filename.my_pipeline.extract]
next_item_mode="fifo"

Where my_pipeline_filename is the name of the script (excluding the .py extensions). I'm not sure if I'm doing something wrong, my pipeline is created as follows:

my_pipeline_filename.py

    pipeline = dlt.pipeline(
        pipeline_name="my_pipeline",
        destination="bigquery",
        dataset_name="stg_aced_history",
        staging="filesystem",
        progress="log",
    )

I'm using dlt.config.get("pipelines.my_pipeline.batch_size", int) in the same script - I'm wondering if there's an issue with the documentation?

waterworthd-cim avatar Nov 13 '25 03:11 waterworthd-cim