Endless Loop with Daily History Requests for Future Contracts

[chris0x42]

When retrieving historical data with daily resolution, an infinite loop may occur for future contracts if the data is only available after the specified start date.

Expected Behavior

The data download aborts because there is no older data for the future contract, even if the oldest data point is newer than the specified start date.

Actual Behavior

The downloader attempts to retrieve the same data from the same oldest data point in an endless loop. Because IBKR always returns the same result, the loop never terminates.

Potential Solution

In the loop, the end time is set to the time of the oldest data point. It should not be converted to the exchange time zone for daily resolution, as otherwise the data from the oldest data point would be retrieved endlessly from IBKR.

// moving endTime to the new position to proceed with next request (if needed)
if (request.Resolution == Resolution.Daily)
{
    endTime = oldestDataPoint.Time;
}
else
{
    endTime = oldestDataPoint.Time.ConvertToUtc(exchangeTimeZone);
}

Additionally, a check could be added to prevent the endless loop if the current oldest data point is not older than the previous oldest data point.

Reproducing the Problem

lean data download --data-provider-historical "Interactive Brokers" --data-type Trade --resolution Daily --security-type Future --ticker ZR --market CBOT --start 20240101 --end 20250905

Console output:

20250910 05:59:54.365 TRACE:: DownloaderDataProvider.Main(): Starting download Symbol: ZR MD, Resolution:
Daily, StartUtc: 1/1/2024 12:00:00 AM, EndUtc: 9/5/2025 12:00:00 AM, TickType: Trade
20250910 05:59:54.369 TRACE:: InteractiveBrokersBrokerage.LookupSymbols(): Requesting symbol list for ZR ...
20250910 05:59:56.346 TRACE:: InteractiveBrokersBrokerage.LookupSymbols(): Returning 19 contract(s) for ZR
20250910 05:59:56.350 TRACE:: InteractiveBrokersBrokerage::GetHistory(): Submitting request: ZR14U23 (FUT ZR USD CBOT  20230914 0 ): Daily/Trade 1/1/2024 12:00:00 AM UTC -> 9/5/2025 12:00:00 AM UTC
[..]
20250910 06:00:04.841 TRACE:: InteractiveBrokersBrokerage::GetHistory(): Download completed: ZR14F25 (FUT ZR USD CBOT  20250114 0 )
20250910 06:00:04.841 TRACE:: InteractiveBrokersBrokerage::GetHistory(): Submitting request: ZR14H25 (FUT ZR USD CBOT  20250314 0 ): Daily/Trade 1/1/2024 12:00:00 AM UTC -> 9/5/2025 12:00:00 AM UTC
[endless loop]

Checklist

• [x] I have completely filled out this template
• [x] I have confirmed that this issue exists on the current master branch
• [x] I have confirmed that this is not a duplicate issue by searching issues
• [x] I have provided detailed steps to reproduce the issue