Results on Backtesting

[sf202020sf]

Hi,

First of all I would like to thank you for all of your effort you spend in this lib - great work. I have tested the backtesting - when you run a strategy and analyse tradingRecord i was wondering if you take the right bar for the entry:

• in the trading records the index of entry is the index where the indicator rule is true (index x)
• you save the close price(AssetPrice) of the bar x in the trade order

But in general that is not true for live trading:

• you get bar X in the stream and analyze your indicator and decide to enter the trade
• then you decide to enter the market and you will get the Open price of bar x+1

Do I get something wrong ? If this is the case then the backtesting results are not correct?

[nickmitchko]

I am having the same question. My backtest are slightly off from other platforms regadless of the exact same buy/sell times.

[Meier-Andre]

I see it the same as sf202020sf. I developed my own backtesting over the last months with this great lib, and changed it as the way you described it. When my program gets a new tick and recognizes the tick is not in the current bar and is in the next bar, it generates a new bar with index 10 for example in the barseries with my current price and calls the method strategy.shouldEnter(9), because bar nr. 9 is now complete/ended. If the shouldEnter method for bar 9 returns true, my program enters in bar 10 with my current price (open price), of bar 10.

Hopefully this is the right way, what is your opinion?

[nickmitchko]

@Meier-Andre That is what would happen in real life correct? Signal after bar at time t and then purchase at t+1. However, you could also assume that at t+1 you purchase at the t close price. So I'd make sure your using the t+1 open price.

EG:

t-1: shouldEnter() == false --> O: 10 L: 9 H: 11: C: 10
  t: shouldEnter() == true  --> O: 10 L:10 H: 12  C: 12
[You buy here] Buy(price = 12)
t+1: shouldEnter() == false --> O: 12 L: 9 H: 11: C: 10

[Meier-Andre]

@nickmitchko yes, correct. I buy (in livetrading and in backtesting) at the opening price of bar t (EG: 30min Bar from 10:30:00-10:59:59), because my current time (EG: 10:30:01) is already in the new bar t and then i look back if i had a shouldEnter signal in bar t-1 (30min Bar from 10:00:00-10:29:59). if shouldEnter() in bar t-1 returns true my stock entry is called in bar t, the following way: tradingRecord.enter(series.getEndIndex() [my current bar t 10:30 - 11:00], series.getLastBar().getOpenPrice(), 100.0, series.getLastBar().getSimpleDateName());

Thats the way it would work in real live and i developed my backtest the same way, because I have stock data in 2 min bars, in my database and combine them to the lager bars, like 30min bars like in this example, to test different bar sizes.

I take the opening price for trade enter, because the close price is already in the past, when my tradingRecord.enter Method is called and if you have day change between bar t and t-1 there can be a larger gap between close of t-1 and open of t

[sf202020sf]

yes I agree and the main problem is also the all statistics are wrong because of this issue - the openprice of the next bar is not equal of the price of the previous bar - I hope somebody can have a look to this issue.

[nimo23]

the main problem is also the all statistics are wrong because of this issue

With statistics, the criteria-package is ment?

@team172011 can you confirm this, that all statistics are wrong because of this issue described?

[marsteff]

I would like to give this thread a new heartbeat.

First aff all, thanks for this very cool framework, i love it. I played with it a bit and also came to the conclusion, that the backtesting statistics are not that accurate they have to be.

I´ve began with an other approach. On my Bar, i have OHLC Values. I can calculate, if the bar is a green or a red one. I now have to calculate the actual indicator for bar (for a green bar) with a close price which is the low price, then calculate the close price set to the high price and then calculate the original close price. With this approach, i have all values, where the candle is moving in real life (the shouldEnter would be not at the real price like in live trading, but nearly there, but i think this could be calculated too).

The Problem is, that all Indicators using the CachedIndicators, what will give me all the time the same value... Therefore there has to be a possibility to deactivate the caching for backtesting.

I hope my thoughts are clear.

I will create a PR later with this approach.

[team172011]

@marsteff

Therefore there has to be a possibility to deactivate the caching for backtesting.

• When you do backtesting you already have a set of known bars. You cannot travel back in time and change prices - so there should be no need for modifying an existing bar.

With this approach, i have all values, where the candle is moving in real life

• For real time trading we have the last bar uncached, that means you can update the price of the last bar by using the BarSeries#addPrice or BarSeries#addTrade methods

[marsteff]

@team172011 I think the point is, that for my understanding, i will emulate the "live-moving" of a candle in the back-testing, because my live-trading logic is also working with moving candles. So when i want to look if my strategy is working fine i also need a backtesting with "moving candles". Thats why i thought, when i have a list of Bars, i have to simulate the the moving, but i using now a different approach to get the backtesting running. I am using the same logic like i use in live-trading and feed it with historical data (1min candles in a 15min scale for example). works fine but of course, it is a bit slower.

[team172011]

@marsteff thanks for your feedback. This is a frequently asked question. I think we will update the FAQ section regarding this

[phong-phuong]

Would the community prefer the BarSeriesManager be updated (along with all the tests that get broken) for the correct logic of opening on the next bar, or would you prefer to have a new class, say. OpenOnNextBarSeriesManager for this?

[nimo23]

Would the community prefer the BarSeriesManager be updated (along with all the tests that get broken) for the correct logic of opening on the next bar, or would you prefer to have a new class, say. OpenOnNextBarSeriesManager for this?

Why would anyone want to open on the next bar when the decision was made in the last bar? If you open on the next bar, then the decision is out of scope and maybe stale because the reason for the decision lies in the last bar and not in the next bar. Actually, the close price of the last bar is the open price of the next bar, because the beginTime of the next bar is the endTime of the lastBar. So what is your intention or benefit when you open the trade on the next bar (with the opening price) instead of the actual bar (with the close price)?

[phong-phuong]

Because a bar is confirmed after it closes. You can only enter on the next bar. You can't go back in time and open on the the exact close of a closed bar. This is precisely why this topic was created.

On Thu, Jun 1, 2023 at 3:04 PM nimo23 @.***> wrote:

Would the community prefer the BarSeriesManager be updated (along with all the tests that get broken) for the correct logic of opening on the next bar, or would you prefer to have a new class, say. OpenOnNextBarSeriesManager for this?

Why would anyone want to open on the next bar when the decision was made in the last bar? If you open on the next bar, then the decision is out of scope and maybe stale because the reason for the decision lies in the last bar and not in the next bar.

— Reply to this email directly, view it on GitHub https://github.com/ta4j/ta4j/issues/598#issuecomment-1571344269, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP2SV4HXA6BFL6ELWGCO633XJAPGRANCNFSM4NOGBBQQ . You are receiving this because you commented.Message ID: @.***>

[phong-phuong]

Those that trade will know that markets such as Forex and Stock Markets are closed between sessions and/or on weekends. Hence, the term "gaps" in the market. So the close of the previous bar does not necessarily equal to the open of the next bar. The benefit of this change is for more accurate backtesting that more resembles what happens in real trading.

[nimo23]

Because a bar is confirmed after it closes. You can only enter on the next bar. You can't go back in time and open on the the exact close of a closed bar.

Sounds ok. However, if we are really convinced to change that ,then we should change it directly within BarSeriesManager and not by providing another class OpenOnNextBarSeriesManager. We could also add boolean openOnClosePrice-property within BarSeriesManager to configure the behaviour explicitly. But I'm afraid that this is also related to the other calculations (indicies, criteria, etc.), so it's probably not enough to just make the changes in the BarSeriesManager. We have to take a closer look why this was implemented in that way (= open on the closePrice of current bar instead of the openPrice of the next bar).

[nimo23]

In the case of a gap, the openPrice of the nextBar does not necessarily have to be the closePrice of the currentBar. Therefore, adding a trade in the currentBar to the closePrice could happen outside of market opening hours. However, the user will likely add a trade at (almost) the closePrice. However, if you use the nextBar instead, the indicator calculations can be completely different due to the possibility of an gap up/down. Thus, if we place a trade on the nextBar instead, the decision we made on the lastBar might no longer be valid. I think that was the reason why the origin implementer wanted us to add a possible trade at the end of the currentBar instead of the nextBar.

[marsteff]

In our implementation we don't wait for the closing of a candle .. when a rule is satisfied, don't matter at what time, I open a trade and also close a trade, when a rule is satisfied... When u are trading at day-scale (for example) you could only open/close one trade per day...

I think you miss much money, when you only trade on closing candles..

[nimo23]

In our implementation we don't wait for the closing of a candle .. when a rule is satisfied, don't matter at what time, I open a trade and also close a trade, when a rule is satisfied...

However, a rule can only be satisfied at the end of the current bar and not at the beginning of the next bar. At the beginning of the next bar, no decision can be made at that moment because the bar is not closed and has no finalized high/low/close. In ta4j, we don't have the concept of trades within an unfinalized bar (the bar would be unfinalized as we only have the openPrice and the high, low and close price of the next bar is not yet available).

I think you miss much money, when you only trade on closing candles..

This is not necessarily true and may be the opposite of what you think.

[team172011]

In our implementation we don't wait for the closing of a candle .. when a rule is satisfied, don't matter at what time, I open a trade and also close a trade, when a rule is satisfied... When u are trading at day-scale (for example) you could only open/close one trade per day... I think you miss much money, when you only trade on closing candles..

If you trade on a day-scale the required information is only valid at the end of the day. If you want to be able to enter or exit the market during the day, you are not trading on a day-scale. You should change your bar duration to another scale like hour-, minute- or second-scale

[phong-phuong]

The current implementation of the BarSeriesManager avoids look-ahead bias by only presenting completed bars at index t.

However, it executes trades based on the closing price of bar t, which is inaccurate since users are only able to execute trades based on the opening price of t + 1, particularly in markets where gaps frequently occur. Unless a strategy is programmed to wait and execute trades at the last second before the close, this discrepancy must be taken into account.

Given that the library is already well-established, I would propose the following changes:

1. Update BarSeriesManager's default behaviour to execute a trade at the open time and open price of bar t + 1 , but an additional bar is necessary for trade execution for the edge case.
2. Provide an option to revert back to the previous behaviour of opening on the close price at the close time of bar t.
3. Update and or/add additional tests as approriate

User can customise their strategy to account for gaps in their own code by applying their own checks. This is not the responsibility of BarSeriesManager which is a simple backtest.

[phong-phuong]

and not by providing another class OpenOnNextBarSeriesManager. We could also add boolean openOnCurrentBar-property within BarSeriesManager to configure the behaviour explicitly. But I'm afraid that this is also related to the other calculations (indicies, criteria, etc.), so it's probably not enough to just make the changes in the BarSeriesManager. We have to take a closer look why this was implemented in that way (= open on the current bar instead of the next).

If the proposed changes were implemented, you would need to carefully examine the impact given your backtest assumes that you can execute on the closing price, but the next tick that you get executed on could be very different price depending on volatility.

So, I can see there will be need for both situations, if a user wants their backtest to execute at the closing price, and their strategy explicity uses the closing price, though I am not convinced that this behaviour should be the default provided by ta4j, as users generally expect to open on the next candle at the open price. We can provide that option to open on the close, so your use-case is not affected.

[phong-phuong]

However, it executes trades based on the closing price of bar t, which is inaccurate since users are only able to execute trades based on the opening price of t + 1, particularly in markets where gaps frequently occur.

You're making an assumption here, and in doing so, you're inferring something to justify that assumption. However, if the assumption is already incorrect, then its conclusion is also incorrect. It would be nice if you read and understand my posts here in this issue, then you might not have made that assumption.

I'll have to disagree. I have read your posts already and it is you who doesn't understand. When a bar is closed, a new bar is formed when a trade takes place with the last price traded to form the open price (except when trading is closed). Assuming you can get any price you want is the wrong assumption too, unless you use limit orders.

It would be helpful if you learnt a thing or two about how trading actually works before forcing your incorrect assumptions on others. Having access to the next open price is NOT looking into the future, as the bar already opened, and if you were to place a trade, this is most accurate price you can get when working with ohlc bars. You can only trade at the price presented to you. When the bar opens, you can't go back to trade the price at the close, you have already agreed with me on this.

Ta4j's very simple backtest of presenting competed bars prices at index t is the cause of your confusion. In reality, you would be presented with an unfinished bar at index t, and the strategies would need to look back at bar t - 1 for the last completed bar.

However, I didn't want to modify the existing structure, which will require everyone to update their ta4j strategy to accommodate the shifted indexes.

There are a few other inaccuracies and bugs with ta4j I haven't even brought up yet...why would you even choose to add the cost basis as the default implementation for the ReturnCriterion is beyond me. Ta4j doesn't even calculate the return properly when using different sized orders due to developers making wrong assumption and taking short cuts to improve performance.

[phong-phuong]

Again you are confused, do re-read my previous post in detail.

[nimo23]

To make it clear: I know that the price of n (=closePrice of actual finished bar) can be different than n+1 (openPrice of next unfinished bar), especially for gap up/downs, but for your backtesting it would mean, we take a price (n+1) which is not involed in any indicator calculations. Maybe I don't get the real intent right, but the posts I've read haven't convinced me.

Again: please tell us more about the "few other inaccuracies and bugs with ta4j" by creating new issues for that. That would be very kind of you.

[phong-phuong]

You took it in the wrong context. I am referring to the next open price for the trade execution price for a stratety that trades on "the open". For "on close" strategies, aka strategies that open before the close, yes that is look ahead.

But have you even developed a live trading strategy? Do you even open on the close???

[phong-phuong]

I'm not talking about indicators, I'm talking about where the backtest will execute the price at.

[nimo23]

You took it in the wrong context. I am referring to the next open price for the trade execution price for a stratety that trades on the "on open". For "on close" strategies, aka strategies that open before the close, yes that is look ahead.

Oh, ok, I understand the intent better now. I have to think about it, I don't have much time at the moment. But now I know what you mean! Yes, I do have to take a closer look at that. Thanks.

[nimo23]

Thanks again @phong-phuong for your clarification!

Yes, we must use the openPrice of the next (finished) Bar instead of the closePrice of the current bar. I think, https://github.com/ta4j/ta4j/pull/984 is a good start to resolve this issue.

According https://github.com/ta4j/ta4j/issues/598#issuecomment-645202921:

the main problem is also the all statistics are wrong because of this issue

We also need to check if the change from "closePrice" to "openPrice" is also reflected in the criteria calculations. If so, then we must add those changes within https://github.com/ta4j/ta4j/pull/984.

[phong-phuong]

I've updated the criteria tests to continue to use the closing prices by setting bar series manager to use the execute on close model. That way, none of the tests will be affected. I don't have time to update all the tests to pass, nor time to understand how each different developer came up with some of those calculations.

[nimo23]

I've updated the criteria tests to continue to use the closing prices by setting bar series manager to use the execute on close model.

But if we introduce a ExecuteOnOpenModel, then users expect that criteria calculations are also adapted. Otherwise it is misleading for the user.. If you don't have time, I understand that, maybe we can help here..