Example Algorithmic Trading bot - broken - crashes the server

[123mitnik]

when running the Example Algorithmic Trading bot

and the server is started with:

RUST_BACKTRACE=full ./tdb-server -vv -a -f "/tectonicdb/test/test-data" -i 10000

from tectonic import TectonicDB
import json
import asyncio

async def subscribe(name):
    db = TectonicDB(host="localhost", port=9001)
    _success, _text = await db.subscribe(name)
    while 1:
        _, item = await db.poll()
        if b"NONE" == item:
            await asyncio.sleep(0.01)
        else:
            yield json.loads(item)

class TickBatcher(object):
    def __init__(self, db_name):
        self.one_batch = []
        self.db_name = db_name

    async def batch(self):
        async for item in subscribe(self.db_name):
            self.one_batch.append(item)

    async def timer(self):
        while 1:
            await asyncio.sleep(1)     # do work every n seconds
            print(len(self.one_batch)) # do work here
            self.one_batch = []        # clear queue

if __name__ == '__main__':
    loop = asyncio.get_event_loop()

    proc = TickBatcher("bt_btceth")
    loop.create_task(proc.batch())
    loop.create_task(proc.timer())

    loop.run_forever()
    loop.close()

where the bt_btceth - is the file located inside the test-data directory

when running the code it crashes the server with:

[2021-01-08][21:58:28:763015000][tdb_server_core::server][INFO] Accepting from: 127.0.0.1:54466
thread 'async-std/executor' panicked at 'range end index 1398096467 out of range for slice of length 1048576', /Users/megalodon/hook/tectonicdb/crates/tdb-server-core/src/server.rs:97:32
stack backtrace:
   0:        0x100fb9bb4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha0848bb2602b5d05
   1:        0x100fd91e0 - core::fmt::write::h9f3ccac2ef682b93
   2:        0x100fb35e6 - std::io::Write::write_fmt::h0a47673aab280496
   3:        0x100fbb9b9 - std::panicking::default_hook::{{closure}}::h850c6aaf5e80c2f5
   4:        0x100fbb67d - std::panicking::default_hook::h037801299da6e1c6
   5:        0x100fbc03b - std::panicking::rust_panic_with_hook::h76436d4cf7a368ac
   6:        0x100fbbb65 - std::panicking::begin_panic_handler::{{closure}}::h516c76d70abf04f6
   7:        0x100fba028 - std::sys_common::backtrace::__rust_end_short_backtrace::h653290b4f930faed
   8:        0x100fbbaca - _rust_begin_unwind
   9:        0x100fe790f - core::panicking::panic_fmt::hde9134dd19c9a74f
  10:        0x100fe79e6 - core::slice::index::slice_end_index_len_fail::h1abfffb7603f7340
  11:        0x100e2b7a5 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::ha8ee21bb24b9c334
  12:        0x100e1b2db - async_task::raw::RawTask<F,R,S,T>::run::h3048d31c3e65fa36
  13:        0x100f19d9b - std::thread::local::LocalKey<T>::with::h22726983dfed64e7
  14:        0x100f1e039 - std::sys_common::backtrace::__rust_begin_short_backtrace::h751ab56d9cd682d2
  15:        0x100f1969d - core::ops::function::FnOnce::call_once{{vtable.shim}}::hef7b80dd4fa7d5bb
  16:        0x100fbf4cd - std::sys::unix::thread::Thread::new::thread_start::hedb7cc0d930a8f40
  17:     0x7fff2036a950 - __pthread_start
[1]    21857 abort      RUST_BACKTRACE=full ./tdb-server -vv -a -f  -i 10000

rustc 1.49.0 (e1884a8e3 2020-12-29)

the order book seems ok.. running the ffi.py with big limit, gives:

                    ts    seq  is_trade  is_bid     price       size
0        1509862963012   7050      True    True  0.040190  81.000000
1        1509862963012   7050      True    True  0.040190  28.550123
2        1509862963012   7050      True    True  0.036790   0.000000
3        1509862963012   7050      True    True  0.032791   0.000000
4        1509862963012   7050      True    True  0.040313   0.990000
...                ...    ...       ...     ...       ...        ...
6104686  1510340961171  16144      True    True  0.052534   0.000000
6104687  1510340961171  16144      True    True  0.060045   0.045147
6104688  1510340961171  16144      True    True  0.060048   0.350615
6104689  1510340961171  16144      True    True  0.060054   0.382365
6104690  1510340961171  16144      True    True  0.060059   0.067691

[6104691 rows x 6 columns]
53.56917119026184