pathway
pathway copied to clipboard
Published
20 hours ago •
pathwaycom
pathwaycom
[Bug]: Tutorial on Defining behaviors for streaming windows
Steps to reproduce
I try to reproduce your tutorial : https://pathway.com/developers/user-guide/exploring-pathway/from-jupyter-to-deploy/#part-2-from-static-data-exploration-to-interactive-dashboard-prototyping
I use only copy/paste your code and I obtain an error of input type when I execute the cell bellow "Please add the behavior argument to window definition as in the code snippet below."
python version : 3.11.8 pathway version : 0.8.2
Relevant log output
~~~python
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[23], line 3
1 minute_20_stats = (
2 data
----> 3 .windowby(
4 pw.this.t,
5 window=pw.temporal.sliding(
6 hop=datetime.timedelta(minutes=1),
7 duration=datetime.timedelta(minutes=20)
8 ),
9 # Wait until the window collected all data before producing a result
10 behavior=pw.temporal.exactly_once_behavior(),
11 instance=pw.this.ticker
12 )
13 .reduce(
14 ticker=pw.this._pw_instance,
15 t=pw.this._pw_window_end,
16 volume=pw.reducers.sum(pw.this.volume),
17 transact_total=pw.reducers.sum(pw.this.volume * pw.this.vwap),
18 transact_total2=pw.reducers.sum(pw.this.volume * pw.this.vwap**2)
19 )
20 .with_columns(
21 vwap=pw.this.transact_total [/](http://localhost:8888/) pw.this.volume
22 )
23 .with_columns(
24 vwstd=(pw.this.transact_total2 [/](http://localhost:8888/) pw.this.volume - pw.this.vwap**2)**0.5
25 ).with_columns(
26 bollinger_upper=pw.this.vwap + 2 * pw.this.vwstd,
27 bollinger_lower=pw.this.vwap - 2 * pw.this.vwstd
28 )
29 )
31 minute_1_stats = (
32 data.windowby(
33 pw.this.t,
(...)
44 .with_columns(vwap=pw.this.transact_total [/](http://localhost:8888/) pw.this.volume)
45 )
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/trace.py:129](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/trace.py#line=128), in trace_user_frame.<locals>._pathway_trace_marker(*args, **kwargs)
127 return func(*args, **kwargs)
128 except Exception as e:
--> 129 _reraise_with_user_frame(e)
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/trace.py:109](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/trace.py#line=108), in _reraise_with_user_frame(e, trace)
106 if user_frame is not None:
107 add_pathway_trace_note(e, user_frame)
--> 109 raise e
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/desugaring.py:341](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/desugaring.py#line=340), in desugar.<locals>.wrapper(*args, **kwargs)
334 args = tuple(
335 desugaring_context._desugaring.eval_expression(arg) for arg in args
336 )
337 kwargs = {
338 key: desugaring_context._desugaring.eval_expression(value)
339 for key, value in kwargs.items()
340 }
--> 341 return func(*args, **kwargs)
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/arg_handlers.py:20](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/arg_handlers.py#line=19), in arg_handler.<locals>.wrapper.<locals>.inner(*args, **kwargs)
17 @wraps(func)
18 def inner(*args, **kwargs):
19 args, kwargs = handler(*args, **kwargs)
---> 20 return func(*args, **kwargs)
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/arg_handlers.py:20](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/arg_handlers.py#line=19), in arg_handler.<locals>.wrapper.<locals>.inner(*args, **kwargs)
17 @wraps(func)
18 def inner(*args, **kwargs):
19 args, kwargs = handler(*args, **kwargs)
---> 20 return func(*args, **kwargs)
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py:19](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py#line=18), in check_arg_types.<locals>.with_type_validation(*args, **kwargs)
14 """Hides beartype dependency by reraising beartype exception as TypeError.
15
16 Should not be needed after resolving https://github.com/beartype/beartype/issues/234
17 """
18 try:
---> 19 return beartype.beartype(f)(*args, **kwargs)
20 except beartype.roar.BeartypeCallHintParamViolation as e:
21 raise TypeError(e) from None
File <@beartype(pathway.stdlib.temporal._window.windowby) at 0x14cdad6c0>:108, in windowby(__beartype_func, __beartype_conf, __beartype_get_violation, __beartype_object_140551755614240, __beartype_object_140551766098816, __beartype_object_140551790108176, __beartype_object_5625875136, __beartype_object_5638154688, __beartype_object_140551766513840, *args, **kwargs)
File ~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/stdlib/temporal/_window.py:910, in windowby(self, time_expr, window, behavior, instance)
858 @trace_user_frame
859 @desugar
860 @arg_handler(handler=shard_deprecation)
(...)
869 instance: pw.ColumnExpression | None = None,
870 ) -> pw.GroupedTable:
871 """
872 Create a GroupedTable by windowing the table (based on `expr` and `window`),
873 optionally with `instance` argument.
(...)
908 1 | 1 | 16 | 2
909 """
--> 910 return window._apply(self, time_expr, behavior, instance)
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py:19](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py#line=18), in check_arg_types.<locals>.with_type_validation(*args, **kwargs)
14 """Hides beartype dependency by reraising beartype exception as TypeError.
15
16 Should not be needed after resolving https://github.com/beartype/beartype/issues/234
17 """
18 try:
---> 19 return beartype.beartype(f)(*args, **kwargs)
20 except beartype.roar.BeartypeCallHintParamViolation as e:
21 raise TypeError(e) from None
File <@beartype(pathway.stdlib.temporal._window._SlidingWindow._apply) at 0x14cdac180>:98, in _apply(__beartype_func, __beartype_conf, __beartype_get_violation, __beartype_object_140551755614240, __beartype_object_140551766098816, __beartype_object_5625875136, __beartype_object_5638154688, __beartype_object_140551766513840, *args, **kwargs)
File ~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/stdlib/temporal/_window.py:340, in _SlidingWindow._apply(self, table, key, behavior, instance)
332 @check_arg_types
333 def _apply(
334 self,
(...)
338 instance: pw.ColumnExpression | None,
339 ) -> pw.GroupedTable:
--> 340 check_joint_types(
341 {
342 "time_expr": (key, TimeEventType),
343 "window.hop": (self.hop, IntervalType),
344 "window.duration": (self.duration, IntervalType),
345 "window.origin": (self.origin, TimeEventType),
346 }
347 )
349 key_dtype = eval_type(key)
350 assign_windows = self._window_assignment_function(key_dtype)
File [~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/stdlib/temporal/utils.py:79](http://localhost:8888/lab/workspaces/~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/stdlib/temporal/utils.py#line=78), in check_joint_types(parameters)
75 else:
76 expected_types_string = " or ".join(
77 repr(tuple(ex_types.values())) for ex_types in expected_types
78 )
---> 79 raise TypeError(
80 f"Arguments ({', '.join(parameters.keys())}) have to be of types "
81 + f"{expected_types_string} but are of types {tuple(types.values())}."
82 )
TypeError: Arguments (time_expr, window.hop, window.duration) have to be of types (INT, INT, INT) or (FLOAT, FLOAT, FLOAT) or (DATE_TIME_NAIVE, DURATION, DURATION) or (DATE_TIME_UTC, DURATION, DURATION) but are of types (INT, DURATION, DURATION).
Occurred here:
Line: .windowby(
File: [/var/folders/2l/p4vvj_3j3fq2h1l38s2dgtq00000gn/T/ipykernel_9280/4248455004.py:3](http://localhost:8888/var/folders/2l/p4vvj_3j3fq2h1l38s2dgtq00000gn/T/ipykernel_9280/4248455004.py#line=2)
~~~
What did you expect to happen?
A tutorial up to date ;)
Version
0.8.2
Docker Versions (if used)
No response
OS
MacOS
On which CPU architecture did you run Pathway?
x86-64
Thanks for the bug report - we'll get back to you shortly with a resolution.
I think I found this problem.
In part 1, the data are loaded and the t column is converted to date time format :
data = data.with_columns(t=data.t.dt.utc_from_timestamp(unit="ms"))
It's not present in part 2 after loading shown in "Switching to streaming data" section.
With conversion in datetime format, minute_20_stats and minute_1_stats appear to be correctly calculated.
Looking at your charts, the t column is indeed INT on your side. Unfortunately it doesn't work on my side. Applying conversion to datetime format, it's good for me.
I would like to share my experience on this part 2. You should send a message to the installation page of the panel module because it requires the pyviz_comms module ( https://panel.holoviz.org/getting_started/installation.html )
After the stupid and nasty execution of the cells that you propose, the output of pn.Row(...) leaves the static mode. I first need to recalculate minute_20_stats and minute_1_stats then re-execute the cell which calculates joint_stats (present in part 1 and not mentioned in part 2) to obtain streaming mode.
Then I can execute the pn.Row(...) command that you suggest for a nice display ready for streaming and finally the pw.run() command to start streaming.
When streaming, I have this warning which appears but which you must surely already be working on or have even corrected:
~/miniconda3/envs/mixtral_ollama/lib/python3.11/site-packages/pathway/stdlib/viz/table_viz.py:140:
FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
df = df.applymap(_format_types) # type: ignore