Event sourced backtesting

[sgflt]

As BFU I wanted to test some strategies. I had to grab BarSeries, code some data loader (CSV, JDBC, online API). After a few days I had some runnable code with copy pasted graphing.

But results are outcome for some static handmade set of parameters. Later I have implemented simulated annealing for parameter estimation.

Real exchange is somehow more complex. There are candle stream, tick stream, news stream, stream of profits for each position. Maybe other useful streams. I have added Apache Camel realtime candle aggregation to multiple timeframes.

And there is the point where I need something better. I need to push that time ordered streams into event sourced backtester that forwards apropriate candles to related time framed series. That forwards tick, profits and other stateful data to "trading context".

For this complexity backtesting is "iterate only for each bar and calculate indicators" not sufficient, because it covers small portion of real problem.

I usually use Rules that fetch data through Supplier<> from external source and are not even related to series.

  this.liveTrading = Map.of(
        "1m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "1m")).build(),
        "5m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "5m")).build(),
        "15m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "15m")).build(),
        "30m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "30m")).build(),
        "1h", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "1h")).build(),
        "4h", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "4h")).build()
    );

  private Strategy buildStrategy(final BarSeries series, final String resolution) {
    if (series == null) {
      throw new IllegalArgumentException("Series cannot be null");
    }


    return this.strategyFactoryFactory.createStrategy(
        series,
        this.symbol.symbol(),
        resolution,
        new RuntimeContext(
            this::getProfit,
            this::getMaxProfit,
            this::getTimeInTrade,
            this::getPriceDifference,
            this::getDifference,
            this.positions::size
        )
    );
  }

  public synchronized boolean shouldEnter() {
    return this.liveTrading.values().stream().allMatch(LiveTrading::shouldEnter);
  }

So I need event sourced backtesting.

1. Load time based events from data source.
2. Prepare chosen time frame series
3. Assign strategies to corresponding timeframes
4. Publish events
5. Consume events and update state of backtest (indicators, profits, positions, signals, bid/ask price)
6. compare results with best fitting strategy
7. evolute parameters of strategies
8. repeat from 2. until satisfied

Are you interested in such complex backtesting framework or is it far far out of scope of this library?

[TheCookieLab]

So as a best practice one's back-testing engine should be as close as possible - ideally the same - to the trading engine. Since most client apps these days are going to interface with their broker/exchange via web-socket I agree that event based back-testing would be valuable. If I'm not mistaken however, you are also asking us to adopt Apache Camel as a new framework correct?

As it happens I've used Camel quite extensively in a past life (integrating smart meters with back-office systems at scale) so I personally would have no issue, however it would certainly be a dramatic change for most users. Not only is there the learning curve with a new DSL, but most would likely need to extensively refactor/re-architect their client apps. This is a change that requires significant community engagement to weigh pros, cons, alternatives, etc.

On Tue, Oct 29, 2024 at 1:56 PM Lukáš Kvídera @.***> wrote:

As BFU I wanted to test some strategies. I had to grab BarSeries, code some data loader (CSV, JDBC, online API). After a few days I had some runnable code with copy pasted graphing.

But results are outcome for some static handmade set of parameters. Later I have implemented simulated annealing for parameter estimation.

Real exchange is somehow more complex. There are candle stream, tick stream, news stream, stream of profits for each position. Maybe other useful streams. I have added Apache Camel realtime candle aggregation to multiple timeframes.

And there is the point where I need something better. I need to push that time ordered streams into event sourced backtester that forwards apropriate candles to related time framed series. That forwards tick, profits and other stateful data to "trading context".

For this complexity backtesting is "iterate only for each bar and calculate indicators" not sufficient, because it covers small portion of real problem.

I usually use Rules that fetch data through Supplier<> from external source and are not even related to series.

this.liveTrading = Map.of( "1m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "1m")).build(), "5m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "5m")).build(), "15m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "15m")).build(), "30m", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "30m")).build(), "1h", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "1h")).build(), "4h", new LiveTradingBuilder().withStrategyFactory(series -> buildStrategy(series, "4h")).build() );

private Strategy buildStrategy(final BarSeries series, final String resolution) { if (series == null) { throw new IllegalArgumentException("Series cannot be null"); }

return this.strategyFactoryFactory.createStrategy(
    series,
    this.symbol.symbol(),
    resolution,
    new RuntimeContext(
        this::getProfit,
        this::getMaxProfit,
        this::getTimeInTrade,
        this::getPriceDifference,
        this::getDifference,
        this.positions::size
    )
);

}

public synchronized boolean shouldEnter() { return this.liveTrading.values().stream().allMatch(LiveTrading::shouldEnter); }

So I need event sourced backtesting.

1. Load time based events from data source.
2. Prepare chosen time frame series
3. Assign strategies to corresponding timeframes
4. Publish events
5. Consume events and update state of backtest (indicators, profits, positions, signals, bid/ask price)
6. compare results with best fitting strategy
7. evolute parameters of strategies
8. repeat from 2. until satisfied

Are you interested in such complex backtesting framework or is it far far out of scope of this library?

— Reply to this email directly, view it on GitHub https://github.com/ta4j/ta4j/issues/1209, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQJ6FLRV4J2B2OJYJHHKTDZ57D4FAVCNFSM6AAAAABQ2NU2JOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZDCOJXGEZDAMY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ek-ex]

IMO I vote to keep the library as clean as possible to allow ease of integration with whatever use-case one can have. Adding libraries would make integration inflexible. In my case, I listen to callback calls from the broker that would update the bars every 250ms, and my trading agent is responsible to interfacing the callback into a state and let my strategies run outside the live update thread loop.

My backtesting strategy plugs into the same callback functions so that the backtesting is as close to the realtime usecase as possible.

[sgflt]

you are also asking us to adopt Apache Camel as a new framework correct?

No, no. just the event based idea, that could better reflect the real implementation of entry/exit rules, which may not depend only on indicator data based on bars, but on something outside of bar series.

[sgflt]

Draft without methods.

[TheCookieLab]

Please see new road-map and project "vision": https://github.com/ta4j/ta4j-wiki/blob/master/Roadmap-and-Tasks.md

Gave this a lot of thought, going back and forth multiple times. Ultimately back-testing is a subject that has a multitude of assumptions and opinions baked in for Ta4j to support. If you can generalize/genericize this such that it can support a variety of implementations then I'll give it an earnest re-evaluation. Otherwise not scope we should take on, and better left to your client app.

On Sat, Nov 2, 2024 at 11:56 AM Lukáš Kvídera @.***> wrote:

Draft without methods.

ta4j.png (view on web) https://github.com/user-attachments/assets/bd530dac-c0ca-4e1b-9f08-a8c15953ccac

— Reply to this email directly, view it on GitHub https://github.com/ta4j/ta4j/issues/1209#issuecomment-2453030479, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQJ6FO2UH3SDDONLFQMJZLZ6TY3RAVCNFSM6AAAAABQ2NU2JOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTGAZTANBXHE . You are receiving this because you commented.Message ID: @.***>

[sgflt]

If you can generalize/genericize this such that it can support a variety of implementations then I'll give it an earnest re-evaluation.

Okay. For now I am going to do OpenOffice/LibreOffice thing and if event sourced streams are deal breaker, then I would kindly ask for re-evaluation :).

[sgflt]

Concept of events has interesting impact on testability. I coud disentangle Indicators from series. It simplifies loading data, in which case I do not even need NumFactory.

I like this kind of hackaton because I like refactorings and this looks promising. It does not mean it will be really usable, but at start it allowed me clean API even more.

XlsTestUtils:

NumericIndicator:

I do not like global NumFactory as it stands on expectation that signle JVM will use defined precision, but now it simplifies decoupling.

Indicator:

Assuming bar is immutable. Mutable implementation deserves different method and approach.

Testing draft:

[sgflt]

See python self... assert and forwards methods could belong to TestContext.

context.fastForward(10)
context.assertNext(63.6948)
...

[sgflt]

  @ParameterizedTest(name = "EMA with barCount 10 [{index}] {0}")
  @MethodSource("provideNumFactories")
  void testEmaWithBarCount10(final NumFactory factory) {
    this.context.withNumFactory(factory)
        .withIndicator(NumericIndicator.closePrice().ema(10))
        .fastForwardUntilStable()
        .assertCurrent(63.6948)
        .assertNext(63.2648)
        .assertNext(62.9457);
  }

[sgflt]

  @ParameterizedTest
  @MethodSource("numFactories")
  void calculateOnlyWithProfitPositions(final NumFactory numFactory) {
    final var context = new TradingRecordTestContext()
        .withNumFactory(numFactory)
        .withCriterion(new ProfitLossCriterion());

    context.operate(50).at(100)
        .operate(50).at(110)
        .operate(50).at(100)
        .operate(50).at(105)
        .assertResult(500 + 250)
    ;
  }

[sgflt]

I was able to reimplement BacktestExecutor, a few indicators and ProfitLossCriterion as proof of concept.

Results:

• Testing is gorgeously simple and clean (at least on code what I have migrated)
• It enables possibility to backtest even ticks (as there are tick traders)
• It decouples Indicators from BarSeries. BarSeries is only event transformer and lookup holder for historical data if needed for backtesting.
  • push principle instead of pull, indicator does not pull data from series, but "event processor" pushes bar into indicator

:warning: Current PoC does not allow mutable bars.

Reopening for possible re-evaluation. :bulb:

[sgflt]

TBD mutable bar use case

[sgflt]

Related https://github.com/ta4j/ta4j/issues/1227#issuecomment-2496496156

[sgflt]

Notable changes: https://github.com/sgflt/ta4j/blob/%231209-eventsourced-reactive-backtesting/ta4j-core/src/main/java/org/ta4j/core/StrategyFactory.java

Position tracks its own price history. MaxDrawdownCriterion logic is now part of Position. https://github.com/sgflt/ta4j/blob/%231209-eventsourced-reactive-backtesting/ta4j-core/src/main/java/org/ta4j/core/Position.java

New BarListener interface for classes that are interested in received bars. https://github.com/sgflt/ta4j/blob/%231209-eventsourced-reactive-backtesting/ta4j-core/src/main/java/org/ta4j/core/BarListener.java

IndicatorContext - listens to bars and pushes them into indicators. State may be inscpected from outside for monitoring purposes. RuntimeContext - api will be different, but it allow tracking of variables, that comes from broker.

CashFlow and other analysis clases are now time based, instead of indexed.

Modeled events

Test APIs: TradingRecordTestContext - simplifies testing of trade execution for criterions MarketEventTestContext - simulates data from broker, simplifies testing of indicators

Test example for citerions: https://github.com/sgflt/ta4j/blob/52aeb0966207ad1b54141d13815cc2823be13f4c/ta4j-core/src/test/java/org/ta4j/core/criteria/TimeInTradeCriterionTest.java#L52-L69

And some other changes that I have forgot.

[sgflt]

Currently I get 12043 indicators per second, but database is the bottleneck.

public class AssetRelatedSeries {
  private static final Logger LOG = LoggerFactory.getLogger(AssetRelatedSeries.class);

  private final IndicatorContexts indicatorContexts = IndicatorContexts.empty();
  private final LiveTrading barSeries;


  AssetRelatedSeries(final ResolutionDependentIndicatorChangeListener changeListener) {
    createIndicatorContext(changeListener);
    this.barSeries = new LiveTradingBuilder()
        .withIndicatorContexts(this.indicatorContexts)
        .build();
  }


  private void createIndicatorContext(final ResolutionDependentIndicatorChangeListener changeListener) {
    for (final var timeFrame : getTimeFrames()) {
      final var ictx = IndicatorContext.empty(timeFrame);
      ictx.register((tick, indicatorId, indicator) -> {
            if (indicator instanceof final NumericIndicator numericIndicator) {
              changeListener.accept(
                  tick,
                  timeFrame,
                  indicatorId,
                  numericIndicator
              );
            }
          }
      );

      this.indicatorContexts.add(ictx);


      IntStream.of(5, 15, 20, 50, 100, 200)
          .forEach(barCount -> {
            createHighestIndicators(barCount, ictx);
            createLowestIndicators(barCount, ictx);
            createNormalizedPriceIndicators(barCount, ictx);
            createNormalizedSMA(barCount, ictx);
            createNormalizedEMA(barCount, ictx);
            createNormalizedTREMA(barCount, ictx);
            createNormalizedZLEMA(barCount, ictx);
            createNormalizedLWMA(barCount, ictx);
            createRsi(barCount, ictx);
            createAroonIndicators(barCount, ictx);
            createBollingerBandsIndicators(barCount, ictx);
            createKeltnerChannelIndicators(barCount, ictx);
            createAtrIndicators(barCount, ictx);
            createHighestAtrIndicators(barCount, ictx);
            createAtrRatioIndicators(barCount, ictx);
            createSMAIndicators(barCount, ictx);
            createEmaIndicators(barCount, ictx);
            createTripleEmaIndicators(barCount, ictx);
            createLWMAIndicators(barCount, ictx);
            createZLEMAIndicators(barCount, ictx);
            createAwesomeIndicators(barCount, ictx);
            createVarianceIndicators(barCount, ictx);
            createStandardDevieationIndicators(barCount, ictx);
          });
    }
  }


  private static List<TimeFrame> getTimeFrames() {
    return List.of(
        new TimeFrame("1m"),
        new TimeFrame("5m"),
        new TimeFrame("15m"),
        new TimeFrame("30m"),
        new TimeFrame("1h"),
        new TimeFrame("4h"),
        new TimeFrame("1d")
    );
  }


  private void createHighestIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var highest = indicator.indicator().highest(barCount);
      indicatorContext.add(
          highest,
          new IndicatorIdentification("%s_highest|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createLowestIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var highest = indicator.indicator().lowest(barCount);
      indicatorContext.add(
          highest,
          new IndicatorIdentification("%s_lowest|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createNormalizedPriceIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var high = getHighestHighIndiator(barCount, indicatorContext);
    final var low = getLowestLowIndicator(barCount, indicatorContext);
    for (final var indicator : getPriceIndicators()) {
      final var price = indicator.indicator();
      final var normalized = price.minus(low).dividedBy(high.minus(low));
      indicatorContext.add(
          normalized,
          new IndicatorIdentification("%s_normalized|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private static NumericIndicator getLowestLowIndicator(final int barCount, final IndicatorContext indicatorContext) {
    return indicatorContext.getNumericIndicator(new IndicatorIdentification("low_lowest|%d".formatted(barCount)));
  }


  private static NumericIndicator getHighestHighIndiator(final int barCount, final IndicatorContext indicatorContext) {
    return indicatorContext.getNumericIndicator(new IndicatorIdentification("high_highest|%d".formatted(barCount)));
  }


  private static List<NamedIndicator> getPriceIndicators() {
    return List.of(
        new NamedIndicator(Indicators.closePrice(), "close"),
        new NamedIndicator(Indicators.openPrice(), "open"),
        new NamedIndicator(Indicators.medianPrice(), "median"),
        new NamedIndicator(Indicators.lowPrice(), "high"),
        new NamedIndicator(Indicators.medianPrice(), "low")
    );
  }


  private void createNormalizedSMA(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var high = getHighestHighIndiator(barCount, indicatorContext);
    final var low = getLowestLowIndicator(barCount, indicatorContext);
    for (final var indicator : getPriceIndicators()) {
      final var price = indicator.indicator();
      final var normalizedSma = price.minus(low).dividedBy(high.minus(low)).sma(barCount);
      indicatorContext.add(
          normalizedSma,
          new IndicatorIdentification("%s_normalized_sma|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createNormalizedEMA(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var high = getHighestHighIndiator(barCount, indicatorContext);
    final var low = getLowestLowIndicator(barCount, indicatorContext);
    for (final var indicator : getPriceIndicators()) {
      final var price = indicator.indicator();
      final var normalizedEma = price.minus(low).dividedBy(high.minus(low)).ema(barCount);
      indicatorContext.add(
          normalizedEma,
          new IndicatorIdentification("%s_normalized_ema|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createNormalizedTREMA(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var high = getHighestHighIndiator(barCount, indicatorContext);
    final var low = getLowestLowIndicator(barCount, indicatorContext);
    for (final var indicator : getPriceIndicators()) {
      final var price = indicator.indicator();
      final var normalizedTREma = price.minus(low).dividedBy(high.minus(low)).tripleEma(barCount);
      indicatorContext.add(
          normalizedTREma,
          new IndicatorIdentification("%s_normalized_tripleema|%d".formatted(
              indicator.name(),
              barCount
          ))
      );
    }
  }


  private void createNormalizedZLEMA(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var high = getHighestHighIndiator(barCount, indicatorContext);
    final var low = getLowestLowIndicator(barCount, indicatorContext);
    for (final var indicator : getPriceIndicators()) {
      final var price = indicator.indicator();
      final var normalizedZLEma = price.minus(low).dividedBy(high.minus(low)).zlema(barCount);
      indicatorContext.add(
          normalizedZLEma,
          new IndicatorIdentification("%s_normalized_zlema|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createNormalizedLWMA(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var high = getHighestHighIndiator(barCount, indicatorContext);
    final var low = getLowestLowIndicator(barCount, indicatorContext);
    for (final var indicator : getPriceIndicators()) {
      final var price = indicator.indicator();
      final var normalizedLWMA = price.minus(low).dividedBy(high.minus(low)).lwma(barCount);
      indicatorContext.add(
          normalizedLWMA,
          new IndicatorIdentification("%s_normalized_lwma|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createVarianceIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var variance = indicator.indicator().variance(barCount);
      indicatorContext.add(
          variance,
          new IndicatorIdentification("%s_var|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createStandardDevieationIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var dev = indicator.indicator().stddev(barCount);
      indicatorContext.add(
          dev,
          new IndicatorIdentification("%s_stddev|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createRsi(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var rsi = Indicators.closePrice().rsi(barCount);
    indicatorContext.add(rsi, new IndicatorIdentification("rsi|%d".formatted(barCount)));
  }


  private void createBollingerBandsIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var bollingerBandFacade = Indicators.bollingerBands(barCount, 2);
    final var bbu = bollingerBandFacade.upper();
    final var bbm = bollingerBandFacade.middle();
    final var bbl = bollingerBandFacade.lower();
    indicatorContext.add(bbu, new IndicatorIdentification("bbu|%d".formatted(barCount)));
    indicatorContext.add(bbl, new IndicatorIdentification("bbl|%d".formatted(barCount)));
    indicatorContext.add(bbm, new IndicatorIdentification("bbm|%d".formatted(barCount)));
  }


  private void createKeltnerChannelIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var kf = new KeltnerChannelFacade(barCount, barCount, 2);
    final var kcu = kf.upper();
    final var kcm = kf.middle();
    final var kcl = kf.lower();
    indicatorContext.add(kcu, new IndicatorIdentification("kcu|%d".formatted(barCount)));
    indicatorContext.add(kcm, new IndicatorIdentification("kcm|%d".formatted(barCount)));
    indicatorContext.add(kcl, new IndicatorIdentification("kcl|%d".formatted(barCount)));
  }


  private void createAtrIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var atr = Indicators.atr(barCount);
    indicatorContext.add(atr, getAtrIdentification(barCount));
  }


  private static IndicatorIdentification getAtrIdentification(final int barCount) {
    return new IndicatorIdentification("atr|%d".formatted(barCount));
  }


  private void createHighestAtrIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var atr = indicatorContext.getNumericIndicator(getAtrIdentification(barCount)).highest(barCount);
    indicatorContext.add(atr, getAtrHighestIdentification(barCount));
  }


  private static IndicatorIdentification getAtrHighestIdentification(final int barCount) {
    return new IndicatorIdentification("atr_highest|%d".formatted(barCount));
  }


  private void createAtrRatioIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var atrHighest = indicatorContext.getNumericIndicator(getAtrHighestIdentification(barCount));
    final var atrRatio = indicatorContext.getNumericIndicator(getAtrIdentification(barCount)).dividedBy(atrHighest);
    indicatorContext.add(atrRatio, new IndicatorIdentification("atrRatio|%d".formatted(barCount)));
  }


  private void createSMAIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var sma = indicator.indicator().sma(barCount);
      indicatorContext.add(sma, new IndicatorIdentification("%s_sma|%d".formatted(indicator.name(), barCount)));
    }
  }


  private void createEmaIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var ema = indicator.indicator().ema(barCount);
      indicatorContext.add(ema, new IndicatorIdentification("%s_ema|%d".formatted(indicator.name(), barCount)));
    }
  }


  private void createTripleEmaIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var tripleEma = indicator.indicator().tripleEma(barCount);
      indicatorContext.add(
          tripleEma,
          new IndicatorIdentification("%s_tripleema|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createZLEMAIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var zlema = indicator.indicator().zlema(barCount);
      indicatorContext.add(zlema, new IndicatorIdentification("%s_zlema|%d".formatted(indicator.name(), barCount)));
    }
  }


  private void createLWMAIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var lwma = indicator.indicator().lwma(barCount);
      indicatorContext.add(lwma, new IndicatorIdentification("%s_lwma|%d".formatted(indicator.name(), barCount)));
    }
  }


  private void createAroonIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    for (final var indicator : getPriceIndicators()) {
      final var aroonOscillator = Indicators.aroonOscillator(barCount);
      final var aroonUp = aroonOscillator.getAroonUpIndicator();
      final var aroonDown = aroonOscillator.getAroonDownIndicator();
      indicatorContext.add(
          aroonOscillator,
          new IndicatorIdentification("%s_aroon|%d".formatted(indicator.name(), barCount))
      );
      indicatorContext.add(
          aroonUp,
          new IndicatorIdentification("%s_aroon_up|%d".formatted(indicator.name(), barCount))
      );
      indicatorContext.add(
          aroonDown,
          new IndicatorIdentification("%s_aroon_down|%d".formatted(indicator.name(), barCount))
      );
    }
  }


  private void createAwesomeIndicators(
      final int barCount,
      final IndicatorContext indicatorContext
  ) {
    final var awesomeOscilator = Indicators.awesomeOscillator(barCount / 3, barCount);
    indicatorContext.add(awesomeOscilator, new IndicatorIdentification("awesome|%d".formatted(barCount)));
  }


  public synchronized void addCandle(final Candle candle) {
    LOG.debug("addCandle(candle={})", candle);

    try {
      this.barSeries.onCandle(
          CandleReceived.builder()
              .timeFrame(new TimeFrame(candle.resolution()))
              .beginTime(candle.startTime())
              .endTime(candle.endTime())
              .openPrice(candle.open())
              .highPrice(candle.high())
              .lowPrice(candle.low())
              .closePrice(candle.close())
              .volume(candle.volume())
              .build()
      );
    } catch (final IllegalArgumentException e) {
      LOG.warn("Candle already present. {}", candle);
    } catch (final NullPointerException e) {
      LOG.error("Missing mapping for resolution {}", candle.resolution(), e);
    }
  }


  private record NamedIndicator(NumericIndicator indicator, String name) {

  }
}

[sgflt]

I have almost finished my vision of event driven architecture.

There is benchmark for decision whether is caching needed.

https://github.com/sgflt/ta4k/blob/eabeb30b5b47a939e6518cdaf990436e258bc8fb/ta4j-core/src/test/kotlin/org/ta4j/core/performance/LiveTradingPerformanceTest.kt#L51-L281

And there are examples of how it can be used.

https://github.com/sgflt/ta4k/tree/master/ta4j-examples/src/main/kotlin/ta4jexamples/strategies/sma

https://github.com/sgflt/ta4k/tree/master/ta4j-examples/src/main/kotlin/ta4jexamples/strategies/ai

There are examples of testing framework.

https://github.com/sgflt/ta4k/blob/eabeb30b5b47a939e6518cdaf990436e258bc8fb/ta4j-core/src/test/kotlin/org/ta4j/core/indicators/numeric/average/SMAIndicatorTest.kt#L33-L106

https://github.com/sgflt/ta4k/blob/eabeb30b5b47a939e6518cdaf990436e258bc8fb/ta4j-core/src/test/kotlin/org/ta4j/core/criteria/NumberOfLosingPositionsCriterionTest.kt#L33-L111

I personally like how it can be plugged into bot just by implementing a few interfaces.

[TheCookieLab]

Looks interesting. You must have picked up a lot of insights and lessons learned throughout the course of development. What are some you think could be applied back to Ta4j?

On Fri, Jun 20, 2025 at 4:42 PM Lukáš Kvídera @.***> wrote:

sgflt left a comment (ta4j/ta4j#1209) https://github.com/ta4j/ta4j/issues/1209#issuecomment-2992775336

I have almost finished my vision of event driven architecture.

There is benchmark for decision whether is caching needed.

https://github.com/sgflt/ta4k/blob/eabeb30b5b47a939e6518cdaf990436e258bc8fb/ta4j-core/src/test/kotlin/org/ta4j/core/performance/LiveTradingPerformanceTest.kt#L51-L281

And there are examples of how it can be used.

https://github.com/sgflt/ta4k/tree/master/ta4j-examples/src/main/kotlin/ta4jexamples/strategies/sma

https://github.com/sgflt/ta4k/tree/master/ta4j-examples/src/main/kotlin/ta4jexamples/strategies/ai

There are examples of testing framework.

https://github.com/sgflt/ta4k/blob/eabeb30b5b47a939e6518cdaf990436e258bc8fb/ta4j-core/src/test/kotlin/org/ta4j/core/indicators/numeric/average/SMAIndicatorTest.kt#L33-L106

https://github.com/sgflt/ta4k/blob/eabeb30b5b47a939e6518cdaf990436e258bc8fb/ta4j-core/src/test/kotlin/org/ta4j/core/criteria/NumberOfLosingPositionsCriterionTest.kt#L33-L111

I personally like how it can be plugged into bot just by implementing a few interfaces.

— Reply to this email directly, view it on GitHub https://github.com/ta4j/ta4j/issues/1209#issuecomment-2992775336, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQJ6FNM5OOLYFWOWS4KAG33ERW2DAVCNFSM6AAAAABQ2NU2JOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSOJSG43TKMZTGY . You are receiving this because you commented.Message ID: @.***>

[sgflt]

I would pick:

1. Fluent test framework that encapsulates wiring of indicators and series. Tester just call builder methods and provide data for testing. Then manage assert flow with assertNext(Num), assertNextNan(), assertNextFalse(), assertNextTrue() It may be combined with excel testing.
2. NumericIndicator as parent of all numeric indicators. It enables nice fluent API, but it requires that each indicator to be statefull and maintains its updates. Stateful indicator may use streaming algorithms for sequential access and boost performance. Where ta4j with looping and recalculations is O(N^2), ta4k streaming is O(1) => the more bars in window, the bigger impact and time saving. This change requires immutable bars. I cant tell whether tick traders may emulate current mutability through choice of smaller timeframe.
3. MultipleTimeframe support. I have introduced IndicatorContexts, and RuntimeContext. Event routing mechanism forwards candle event to appropriate IndicatorContext and single strategy may combine them. RuntimeContext provides data outside of ta4j, like positions from broker, account balance, risk management data etc. It highly relates to https://github.com/ta4j/ta4j/issues/742
4. Indicators facade that acts like entrypoint for indicator creation.

Other changes are dependent on event driven principle and they are not portable without this core philosophy change.

Generated summary:

Java Backport Improvements from Kotlin Migration

This document describes architectural improvements made during the Kotlin migration of ta4j that could be backported to the original Java implementation. These improvements align with the event-sourced backtesting proposal in issue #1209.

1. Event-Driven Architecture

The Kotlin migration introduces a comprehensive event-driven system that decouples market data processing from strategy execution.

Key Components

MarketEventHandler Interface

interface MarketEventHandler {
    fun onCandle(event: CandleReceived)
}

Event Types

• CandleReceived: Complete OHLCV bar data with timeframe information

Benefits for Java ta4j

• Enables real-time and historical data processing with the same codebase
• Supports multiple data sources through event abstraction
• Facilitates testing through event replay
• Aligns with the proposed event-sourcing architecture

2. Enhanced Backtesting Framework

The new backtesting system implements event replay and realistic order execution.

BacktestExecutor

class BacktestExecutor {
    fun execute(
        backtestRun: BacktestRun,
        marketEvents: List<MarketEvent>,
        amount: Number
    ): TradingStatement {
        // Event replay mechanism
        replay(marketEvents, marketEventHandler)
    }
}

Key Improvements

• Event Replay: Chronological processing of market events
• PendingOrderManager: Manages order execution timing
• ExecutionMode:
  • CURRENT_CLOSE: Execute at current bar close
  • NEXT_OPEN: Execute at next bar open (more realistic)
• Separated Bar Types: BacktestBar vs regular Bar for clear separation

Benefits for Java ta4j

• More realistic backtesting results
• Better simulation of real-world trading delays
• Cleaner separation between backtesting and live trading code

3. IndicatorContext System

A major architectural improvement that decouples indicators from BarSeries.

IndicatorContext Features

class IndicatorContext {
    val isStable: Boolean  // Track when indicators have enough data
    fun enableHistory(historyWindow: Int)  // Historical value tracking
    fun register(changeListener: IndicatorChangeListener)
    fun register(updateListener: IndicatorContextUpdateListener)
}

Multi-Timeframe Support

class IndicatorContexts {
    operator fun get(timeFrame: TimeFrame): IndicatorContext
}

Benefits for Java ta4j

• Indicators become independent of specific BarSeries implementations
• Easier to test indicators in isolation
• Supports multi-timeframe strategies naturally
• Enables indicator value caching and history tracking

4. Enhanced Strategy Execution

The strategy system now supports runtime contexts and multi-timeframe analysis.

RuntimeContext

interface RuntimeContext {
    // Dynamic state management during execution
}

class CompoundRuntimeContext : RuntimeContext {
    // Combines multiple contexts (e.g., trading record + custom state)
}

BacktestStrategy

abstract class BacktestStrategy {
    val timeFrames: Set<TimeFrame>
    fun shouldOperate(): OperationType
}

Benefits for Java ta4j

• Strategies can maintain state between evaluations
• Support for complex multi-timeframe strategies
• Clear separation between strategy logic and execution

5. Improved Indicator Architecture

Indicators Facade Entry Point

The Indicators object serves as a centralized factory for creating indicators:

object Indicators {
    @JvmStatic
    fun closePrice() = ClosePriceIndicator(numFactory)
    
    @JvmStatic
    fun sma(barCount: Int) = extended().sma(barCount)
    
    // Factory pattern with NumFactory propagation
    fun extended(numFactory: NumFactory) = ExtendedIndicatorFactory(numFactory)
}

NumericIndicator Base Class with Fluent API

All numeric indicators extend from NumericIndicator which provides fluent methods:

abstract class NumericIndicator(val numFactory: NumFactory) {
    // Arithmetic operations
    fun plus(other: NumericIndicator)
    fun minus(other: NumericIndicator)
    fun multipliedBy(other: NumericIndicator)
    fun dividedBy(other: NumericIndicator)
    
    // Derived indicators
    fun sma(barCount: Int) = SMAIndicator(this, barCount)
    fun ema(barCount: Int) = EMAIndicator(this, barCount)
    fun rsi(barCount: Int) = RSIIndicator(this, barCount)
    
    // Comparison and rules
    fun crossedOver(other: NumericIndicator)
    fun isGreaterThanRule(other: NumericIndicator)
}

Removal of Complexity

• No RecursiveCachedIndicator: Removed complex recursive caching mechanism
• No CachedIndicator: State management simplified through updateState() pattern
• Direct State Updates: Each indicator manages its own state efficiently

Benefits for Java ta4j

• Centralized Creation: Single entry point for all indicators
• Fluent Chaining: Natural expression of complex calculations
• NumFactory Propagation: Consistent numerical precision throughout chains
• Simplified Caching: More predictable performance characteristics
• Better Composability: Indicators can be easily combined

6. Architectural Patterns

Observer Pattern

• BarListeners for series updates
• IndicatorChangeListeners for value changes
• MarketEventHandlers for event processing

Builder Pattern

 val liveTrading = LiveTradingBuilder()
            .withName("AI-Powered Bitcoin Trading")
            .withNumFactory(numFactory)
            .withStrategyFactory(strategyFactory)
            .withIndicatorContexts(indicatorContexts)
            .withConfiguration(StrategyConfiguration())
            .enableHistory(100) // Keep last 100 bars for indicators
            .build()

Benefits for Java ta4j

• More intuitive API for users
• Better testability through dependency injection
• Cleaner, more maintainable code

7. Improved Package Structure

Clear Separation of Concerns

org.ta4j.core/
├── api/           # Core interfaces
├── backtest/      # Backtesting framework
├── events/        # Event types
├── indicators/    # Technical indicators
├── strategy/      # Strategy components
└── trading/       # Live trading support

Benefits for Java ta4j

• Easier to understand and maintain
• Better modularity for custom implementations
• Clearer API boundaries

[TheCookieLab]

Thanks for the write-up. I'll take some time to digest and consider each of the points. In the meantime I encourage any and all in the Ta4j community to chime in with opinions.

On Thu, Jun 26, 2025 at 2:12 AM Lukáš Kvídera @.***> wrote:

sgflt left a comment (ta4j/ta4j#1209) https://github.com/ta4j/ta4j/issues/1209#issuecomment-3007224994

I would pick:

1. Fluent test framework that encapsulates wiring of indicators and series. Tester just call builder methods and provide data for testing. Then manage assert flow with assertNext(Num), assertNextNan(), assertNextFalse(), assertNextTrue() It may be combined with excel testing.
2. NumericIndicator as parent of all numeric indicators. It enables nice fluent API, but it requires that each indicator to be statefull and maintains its updates. Stateful indicator may use streaming algorithms for sequential access and boost performance. Where ta4j with looping and recalculations is O(N^2), ta4k streaming is O(1) => the more bars in window, the bigger impact and time saving. This change requires immutable bars. I cant tell whether tick traders may emulate current mutability through choice of smaller timeframe.
3. MultipleTimeframe support. I have introduced IndicatorContexts, and RuntimeContext. Event routing mechanism forwards candle event to appropriate IndicatorContext and single strategy may combine them. RuntimeContext provides data outside of ta4j, like positions from broker, account balance, risk management data etc. It highly relates to #742 https://github.com/ta4j/ta4j/issues/742

Other changes are dependent on event driven principle and they are not portable without this core philosophy change.

Generated summary: Java Backport Improvements from Kotlin Migration

This document describes architectural improvements made during the Kotlin migration of ta4j that could be backported to the original Java implementation. These improvements align with the event-sourced backtesting proposal in issue #1209 https://github.com/ta4j/ta4j/issues/1209#issuecomment-3005785192.

1. Event-Driven Architecture

The Kotlin migration introduces a comprehensive event-driven system that decouples market data processing from strategy execution. Key Components MarketEventHandler Interface

interface MarketEventHandler { fun onCandle(event: CandleReceived) }

Event Types

• CandleReceived: Complete OHLCV bar data with timeframe information

Benefits for Java ta4j

• Enables real-time and historical data processing with the same codebase
• Supports multiple data sources through event abstraction
• Facilitates testing through event replay
• Aligns with the proposed event-sourcing architecture

2. Enhanced Backtesting Framework

The new backtesting system implements event replay and realistic order execution. BacktestExecutor

class BacktestExecutor { fun execute( backtestRun: BacktestRun, marketEvents: List<MarketEvent>, amount: Number ): TradingStatement { // Event replay mechanism replay(marketEvents, marketEventHandler) } }

Key Improvements

• Event Replay: Chronological processing of market events
• PendingOrderManager: Manages order execution timing
• ExecutionMode:
  • CURRENT_CLOSE: Execute at current bar close
  • NEXT_OPEN: Execute at next bar open (more realistic)
• Separated Bar Types: BacktestBar vs regular Bar for clear separation

Benefits for Java ta4j

• More realistic backtesting results
• Better simulation of real-world trading delays
• Cleaner separation between backtesting and live trading code

3. IndicatorContext System

A major architectural improvement that decouples indicators from BarSeries. IndicatorContext Features

class IndicatorContext { val isStable: Boolean // Track when indicators have enough data fun enableHistory(historyWindow: Int) // Historical value tracking fun register(changeListener: IndicatorChangeListener) fun register(updateListener: IndicatorContextUpdateListener) }

Multi-Timeframe Support

class IndicatorContexts { operator fun get(timeFrame: TimeFrame): IndicatorContext }

Benefits for Java ta4j

• Indicators become independent of specific BarSeries implementations
• Easier to test indicators in isolation
• Supports multi-timeframe strategies naturally
• Enables indicator value caching and history tracking

4. Enhanced Strategy Execution

The strategy system now supports runtime contexts and multi-timeframe analysis. RuntimeContext

interface RuntimeContext { // Dynamic state management during execution } class CompoundRuntimeContext : RuntimeContext { // Combines multiple contexts (e.g., trading record + custom state) }

BacktestStrategy

abstract class BacktestStrategy { val timeFrames: Set<TimeFrame> fun shouldOperate(): OperationType }

Benefits for Java ta4j

• Strategies can maintain state between evaluations
• Support for complex multi-timeframe strategies
• Clear separation between strategy logic and execution

5. Improved Indicator Architecture Indicators Facade Entry Point

The Indicators object serves as a centralized factory for creating indicators:

object Indicators { @JvmStatic fun closePrice() = ClosePriceIndicator(numFactory)

@JvmStatic
fun sma(barCount: Int) = extended().sma(barCount)

// Factory pattern with NumFactory propagation
fun extended(numFactory: NumFactory) = ExtendedIndicatorFactory(numFactory)

}

NumericIndicator Base Class with Fluent API

All numeric indicators extend from NumericIndicator which provides fluent methods:

abstract class NumericIndicator(val numFactory: NumFactory) { // Arithmetic operations fun plus(other: NumericIndicator) fun minus(other: NumericIndicator) fun multipliedBy(other: NumericIndicator) fun dividedBy(other: NumericIndicator)

// Derived indicators
fun sma(barCount: Int) = SMAIndicator(this, barCount)
fun ema(barCount: Int) = EMAIndicator(this, barCount)
fun rsi(barCount: Int) = RSIIndicator(this, barCount)

// Comparison and rules
fun crossedOver(other: NumericIndicator)
fun isGreaterThanRule(other: NumericIndicator)

}

Removal of Complexity

• No RecursiveCachedIndicator: Removed complex recursive caching mechanism
• No CachedIndicator: State management simplified through updateState() pattern
• Direct State Updates: Each indicator manages its own state efficiently

Benefits for Java ta4j

• Centralized Creation: Single entry point for all indicators
• Fluent Chaining: Natural expression of complex calculations
• NumFactory Propagation: Consistent numerical precision throughout chains
• Simplified Caching: More predictable performance characteristics
• Better Composability: Indicators can be easily combined

6. Architectural Patterns Observer Pattern

   • BarListeners for series updates
   • IndicatorChangeListeners for value changes
   • MarketEventHandlers for event processing

Builder Pattern

val liveTrading = LiveTradingBuilder() .withName("AI-Powered Bitcoin Trading") .withNumFactory(numFactory) .withStrategyFactory(strategyFactory) .withIndicatorContexts(indicatorContexts) .withConfiguration(StrategyConfiguration()) .enableHistory(100) // Keep last 100 bars for indicators .build()

Benefits for Java ta4j

• More intuitive API for users
• Better testability through dependency injection
• Cleaner, more maintainable code

7. Improved Package Structure Clear Separation of Concerns

org.ta4j.core/ ├── api/ # Core interfaces ├── backtest/ # Backtesting framework ├── events/ # Event types ├── indicators/ # Technical indicators ├── strategy/ # Strategy components └── trading/ # Live trading support

Benefits for Java ta4j

• Easier to understand and maintain
• Better modularity for custom implementations
• Clearer API boundaries

— Reply to this email directly, view it on GitHub https://github.com/ta4j/ta4j/issues/1209#issuecomment-3007224994, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQJ6FLS254364NXYIOWU7D3FOFLHAVCNFSM6AAAAABQ2NU2JOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMBXGIZDIOJZGQ . You are receiving this because you commented.Message ID: @.***>