Customized calendar cannot work with zipline 1.3.0

[tibetjungle]

Dear Zipline Maintainers,

I am trying to extend zipline( version 1.3.0) to work with Hong Kong Exchange, whose market opens at 9:30 AM and closes at 16:00 PM, with lunch break from 12:00 AM to 13:00 PM, which is not similar to NYSE. when call data.history in handle_data, the following exception threw:

File "zipline/_protocol.pyx", line 647, in zipline._protocol.BarData.history
  File "/usr/local/lib/python3.6/site-packages/zipline/data/data_portal.py", line 971, in get_history_window
    "close")
  File "/usr/local/lib/python3.6/site-packages/zipline/data/data_portal.py", line 906, in _get_history_minute_window
    minutes_for_window,
  File "/usr/local/lib/python3.6/site-packages/zipline/data/data_portal.py", line 1063, in _get_minute_window_data
    False)
  File "/usr/local/lib/python3.6/site-packages/zipline/data/history_loader.py", line 549, in history
    is_perspective_after)
  File "/usr/local/lib/python3.6/site-packages/zipline/data/history_loader.py", line 431, in _ensure_sliding_windows
    array = self._array(prefetch_dts, needed_assets, field)
  File "/usr/local/lib/python3.6/site-packages/zipline/data/history_loader.py", line 595, in _array
    assets,
  File "/usr/local/lib/python3.6/site-packages/zipline/data/dispatch_bar_reader.py", line 120, in load_raw_arrays
    for t in asset_types if sid_groups[t]}
  File "/usr/local/lib/python3.6/site-packages/zipline/data/dispatch_bar_reader.py", line 120, in <dictcomp>
    for t in asset_types if sid_groups[t]}
  File "/usr/local/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1258, in load_raw_arrays
    start_idx, end_idx)
  File "/usr/local/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1054, in _exclusion_indices_for_range
    itree = self._minute_exclusion_tree
  File "/usr/local/lib/python3.6/site-packages/trading_calendars/utils/memoize.py", line 49, in __get__
    self._cache[instance] = val = self._get(instance)
  File "/usr/local/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1035, in _minute_exclusion_tree
    start_pos = self._find_position_of_minute(early_close) + 1
  File "/usr/local/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1227, in _find_position_of_minute
    False,
  File "zipline/data/_minute_bar_internal.pyx", line 85, in zipline.data._minute_bar_internal.find_position_of_minute (zipline/data/_minute_bar_internal.c:1778)
ValueError: Given minute is not between an open and a close

The calendar used to backtest:

class HKExchangeCalendar(TradingCalendar):

    def __init__(self, start=start_default, end=end_default):
        with warnings.catch_warnings():
            warnings.simplefilter('ignore')
            _all_days = date_range(start, end, freq=self.day, tz=pytz.utc)

        self._lunch_break_starts = days_at_time(_all_days, lunch_break_start, self.tz, 0)
        self._lunch_break_ends = days_at_time(_all_days, lunch_break_end, self.tz, 0)

        TradingCalendar.__init__(self, start=start, end=end)


    @property
    def name(self):
        return "HKEX"

    @property
    def tz(self):
        return pytz.timezone("Asia/Shanghai")

    @property
    def open_time(self):
        return time(9, 30)

    @property
    def close_time(self):
        return time(16, 0)

    @property
    def adhoc_holidays(self):
        return [Timestamp(t, tz=pytz.UTC) for t in get_cached(use_list=True)]

    @property
    def _minutes_per_session(self):
        diff = self.schedule.market_close - self.schedule.market_open
        diff = diff.astype('timedelta64[m]')
        return diff + 1 - 60

    @property
    @remember_last
    def all_minutes(self):
        """
            Returns a DatetimeIndex representing all the minutes in this calendar.
        """
        opens_in_ns = \
            self._opens.values.astype('datetime64[ns]')

        closes_in_ns = \
            self._closes.values.astype('datetime64[ns]')

        lunch_break_start_in_ns = \
            self._lunch_break_starts.values.astype('datetime64[ns]')
        lunch_break_ends_in_ns = \
            self._lunch_break_ends.values.astype('datetime64[ns]')

        deltas_before_lunch = lunch_break_start_in_ns - opens_in_ns
        deltas_after_lunch = closes_in_ns - lunch_break_ends_in_ns

        daily_before_lunch_sizes = (deltas_before_lunch / NANOS_IN_MINUTE) + 1
        daily_after_lunch_sizes = (deltas_after_lunch / NANOS_IN_MINUTE) + 1

        daily_sizes = daily_before_lunch_sizes + daily_after_lunch_sizes

        num_minutes = np.sum(daily_sizes).astype(np.int64)

        # One allocation for the entire thing. This assumes that each day
        # represents a contiguous block of minutes.
        all_minutes = np.empty(num_minutes, dtype='datetime64[ns]')

        idx = 0
        for day_idx, size in enumerate(daily_sizes):
            # lots of small allocations, but it's fast enough for now.

            # size is a np.timedelta64, so we need to int it
            size_int = int(size)

            before_lunch_size_int = int(daily_before_lunch_sizes[day_idx])
            after_lunch_size_int = int(daily_after_lunch_sizes[day_idx])

            #print("idx:{}, before_lunch_size_int: {}".format(idx, before_lunch_size_int))
            all_minutes[idx:(idx + before_lunch_size_int)] = \
                np.arange(
                    opens_in_ns[day_idx],
                    lunch_break_start_in_ns[day_idx] + NANOS_IN_MINUTE,
                    NANOS_IN_MINUTE
                )

            all_minutes[(idx + before_lunch_size_int):(idx + size_int)] = \
                np.arange(
                    lunch_break_ends_in_ns[day_idx],
                    closes_in_ns[day_idx] + NANOS_IN_MINUTE,
                    NANOS_IN_MINUTE
                )

            idx += size_int
        return DatetimeIndex(all_minutes).tz_localize("UTC")

Can I change the code line number 84 of file zipline/data/_minute_bar_internal.pyx from

        if not forward_fill and ((minute_val - market_open) >= minutes_per_day):
            raise ValueError("Given minute is not between an open and a close")

to

     if not forward_fill and minute_val > market_close:
            raise ValueError("Given minute is not between an open and a close")

so to avoid the exception?

Sincerely, $ whoami

[josephcclin]

Hi,

I changed the code line number 84 as you proposed and it doesn't work. The exception is still there.

Yours, Joseph

[CapitalZe]

I have the same problem.

[davidplumridge]

I'm facing a similar issue using a 24/7 calendar (the built-in one). Was there any solution or workaround that had a positive result?

[moseshassan]

I'm facing a similar issue using a 24/7 calendar (the built-in one). Was there any solution or workaround that had a positive result?

If the conditional ((minute_val - market_open) >= minutes_per_day) is using the default for minutes_per_day and your custom calendar differs, that could be raising the exception.

In that case, when registering the bundle you can specify the number of minutes as an argument (eg minutes_per_day=1440).