pathway icon indicating copy to clipboard operation
pathway copied to clipboard

[QUESTION] Why does `asof_join` line take long to spin up?

Open dxtrous opened this issue 1 year ago • 0 comments

I noticed that when running in Colab building up a compute graph for an asof_join takes several seconds, regardless of table size.

In the example below, taken from API documentation, it takes 4s, then if you duplicate the line in the cell, it takes 8s, etc.

Why is this the case? Does this only happen in interactive mode?

image

# -*- coding: utf-8 -*-
"""Colab_test_asof_join.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/1pWox7vvDoSuohRZ1EpWR2QDGSRSOmgsV
"""

!pip install pathway

import pathway as pw

t1 = pw.debug.table_from_markdown(
    '''
    value | event_time | __time__
      2   |      2     |     4
      3   |      5     |     6
      4   |      1     |     8
      5   |      7     |    14
'''
)
t2 = pw.debug.table_from_markdown(
    '''
    value | event_time | __time__
      42  |      1     |     2
       8  |      4     |    10
'''
)

result_join = t1.join(t2, t1.event_time ==t2.event_time, how=pw.JoinMode.LEFT).select(event_time = t1.event_time)

result_asof_join = t1.asof_join(
    t2, t1.event_time, t2.event_time, how=pw.JoinMode.LEFT
).select(
    left_value=t1.value,
    right_value=t2.value,
    left_time=t1.event_time,
    right_time=t2.event_time,
)```

dxtrous avatar Mar 14 '24 16:03 dxtrous