streaming preview

[DaveSkender]

Description

A preview of streaming use cases for EMA only, for feedback.

# initialize indicator series baseline
var emaBase = quotes.InitEma(14);
var results = emaBase.Results;

# add streaming / repeating updates
emaBase.Add(quote);

# feedback appreciated

To do

• [ ] investigate potential use of IObservable per @danbopes recommendation
• [ ] experiment with LinkedList, also possibly for SMA lookback, for example
• [ ] interface for base + add with common params and properties
• [ ] register in a global dictionary, separately, can have multiple registries (e.g chart profile)
• [ ] global Add(quote) to update all registered indicators
• [ ] enable init empty (handle insufficient quotes case, calc when ready)
• [ ] refactor time-series to use stream algos, if more performant
• [ ] return full batch of registered indicator results, api payload
• [ ] add streaming example project with websocket (consider updating demo site)

Checklist

• [ ] My code follows the existing style, code structure, and naming taxonomy
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have performed a self-review of my own code and included any verifying manual calculations
• [ ] I have added or updated unit tests that prove my fix is effective or that my feature works, and achieves sufficient code coverage. New and existing unit tests pass locally and in the build (below) with my changes
• [ ] My changes generate no new warnings and running code analysis does not produce any issues
• [ ] I have added or run the performance tests that depict optimal execution times
• [ ] I have made corresponding changes to the documentation

[codebeaulieu]

This would be super handy for use with web sockets. Reactive Extensions internals would be a major plus... possibly consider Dynamic Data as well, which is what I'm currently using.

[DaveSkender]

This would be super handy for use with web sockets. Reactive Extensions internals would be a major plus... possibly consider Dynamic Data as well, which is what I'm currently using.

For sure. I'm thinking of WebSocket specifically for this one.

Thanks for the reference on Dynamic Data -- it might be easier than simply using the native IObservable alone, something I will explore further. I'll have to weigh my appetite for third-party packages; I like the idea of continuing without any.

homework completed

• [x] enable init empty (handle insufficient quotes case, calc when ready)
• [x] experiment with LinkedList, also possibly for SMA lookback, for example --> not using, slow performance
• [x] review Reactive Extension and Dynamic Data --> both of these have some nice advanced Rx things, but seem like overkill or likely performance-limiting overhead for the library (may still be good for library users). The out-of-the-box IObservable and general Rx design patterns seem to work fine for what I'm planning, no need for extra package dependencies IMO.

[DaveSkender]

I've come back to this and am currently working through the complexities ... like dealing with out-of-sequence late quote arrivals through chains of dependencies 😮. @codebeaulieu and @danbopes, I'd love some early feedback on the public interface UX depicted in the description above if you have any.

[martonb]

Just my humble opinion: you started to solve too many issues at once? Personally, i would just have started with creating indicators that don't need to recalculate the full sequence every time a new quote comes in. Only later on create the extra code needed to connect to web sockets, or leave people to make their own hook-up connections, maybe by implementing an interface provided by your API?

Then, for my use case, I would be able to:

• request historic quotes from an arbitrary exchange, using whatever method (eg: REST request), from this moment, backwards N + warmup periods count quotes
• create the indicator, initialized with these quotes
• now I already have a valid indicator value calculated
• when a new quote notification is received from the exchange (via pure TCP/IP (protobuff) or WebSocket (JSON/XML), etc) your API user can process that and pass it to the indicator
• the indicator might delete the oldest quote from the sequence, then progressively calculate the new value
• the user can use the new calculated indicator value

Edit: I just found the "STREAM INITIALIZATION" marked functions in Ema.Api.cs and yes, those look exactly like what I meant, thanks! I assume they are internal, as not production ready?

[DaveSkender]

Just my humble opinion: you started to solve too many issues at once? Personally, I would just have started with creating indicators that don't need to recalculate the full sequence every time a new quote comes in.

This is some good general advice. Thank you. My goal here is to do a single tracer the whole way through with one chainable indicator before widely implementing across all other indicators. Once I get the basic concept working well, I'll think about how to apply some useful increments in smaller forms.

And you're right, I probably could have done the quote stream first or last; but found it hard to test the flow without at least one indicator and chain case. In isolation, the quote stream currently has these dimensions:

# initialize quotes provider,
# can be extended to integrate with your
# quote stream or WebSocket source
QuoteProvider provider = new();

# subscribe individual indicators as observers at any time

# you can add batch quotes
provider.Add(quotes);

# or incremental quotes at any time
provider.Add(quote);

# to pass quotes to subscribers

// stop providing quotes to everyone
// will unsubscribe all gracefully
provider.EndTransmission();

[DaveSkender]

I assume they are internal, as not production ready?

That’s right. It’s only in the feature branch right now. I’ll release a preview package soon, for people to try it out.