These algoritmns are extrimly slow

[reza1615]

These algorithms are slow and in my opinion, there is space for performance improvement Extremely slow

• alma
• td_seq
• cg
• pwma
• cfo
• qqe
• inertia
• cti
• mcgd
• linreg

Slow

• hilo
• psar
• supertrend
• ttm_trend
• vidya
• jma

[reza1615]

version 0.3.14b0

for strategy in ['alma','td_seq','cg','pwma','cfo','qqe','inertia','cti','mcgd','linreg']+['hilo','psar','supertrend','ttm_trend','vidya','jma']:
  ta_list=[
  {"kind": strategy,  "prefix":f"foo@"}
  ]
  df.ta.cores = 31
  df.ta.strategy(ta.Strategy(
              name=f"{strategy}",
              description=f"{strategy}",
              ta=ta_list
          ),timed=True)

[reza1615]

for alma i found https://stackoverflow.com/questions/46990098/arnaud-legoux-moving-average-and-numpy

[twopirllc]

@reza1615,

Terms like "extremely slow" and "slow" are relative and not descriptive. Real numbers are more useful than vague terms.

Please provide a sample ohlcv csv of the data you are using for your code above as well as the execution times you get for:

1. only the extremely slow
2. the extremely slow without td_seq
3. the slow
4. both the extremely slow and slow without td_seq

Also I know td_seq is slow, it is why it is automatically excluded from df.ta.strategy. It needs a new implementation.

Thanks for the alma link.

Kind Regards, KJ

[reza1615]

Here is the code and parquet file df_file.txt and timing. Also, I added SMA to have a better comparison. The rest of the indicators which I didn't list here according to run performance are almost the same as SMA (around 1-5 s). For this try, I didn't continue to run "extremely slow" indicators. After 8 min I terminated running alma.

sma [i] Runtime: 194.6930 ms (0.1947 s) hilo [i] Runtime: 40416.3191 ms (40.4163 s) psar [i] Runtime: 52944.7372 ms (52.9447 s) supertrend [i] Runtime: 25181.5755 ms (25.1816 s) ttm_trend [i] Runtime: 4925.0239 ms (4.9250 s) vidya [i] Runtime: 22703.0376 ms (22.7030 s) jma [i] Runtime: 11367.1171 ms (11.3671 s)

Base on another experiment there is log.txt

import pandas_ta as ta
import pandas as pd
df=pd.read_parquet('df_file.txt') # because of Github I added .txt please remove it


df.ta.cores = 31

for indicator in ['sma', 'alma','td_seq','cg','pwma','cfo','qqe','inertia','cti','mcgd','linreg']+['hilo','psar','supertrend','ttm_trend','vidya','jma']:
    print(indicator)
    ta_list=[]
    for length in range(2,2000,100):
        ta_list.append({"kind": indicator, 'length': length, "prefix":f"{indicator}_{length}@"})
    df.ta.strategy(ta.Strategy(
              name=f"{indicator}",
              description=f"{indicator}",
              ta=ta_list
          ),timed=True)

[reza1615]

for the linereg the below library is a wrapper for TA-LIB based on Cython instead of SWIG. Maybe as an option, we can have these wrappers as high-performance codes. https://mrjbq7.github.io/ta-lib/

[twopirllc]

@reza1615,

for the linereg the below library is a wrapper for TA-LIB based on Cython instead of SWIG. Maybe as an option, we can have these wrappers as high-performance codes.

Actually that has been completed for most TA Lib indicators. On the main branch, these are the TA Lib indicators that are currently wrapped. The linreg TA Lib adaption is on the development branch if you want to test it out:

$ pip install -U git+https://github.com/twopirllc/pandas-ta.git@development

However, the TA Lib wrapper solution is not the solution to why you are getting such "slow" speeds. It is your misunderstanding of how df.ta.strategy works. Did you look at the examples in the PandasTA Strategy Examples Notebook?

Why did you give me different code and the 4 results that I requested from you? Stop moving the goal posts!

I will address your original code first with the parquet file you provided before I address your new goal post Issue that I did not ask for.

KJ

[reza1615]

Why did you give me different code and the 4 results that I requested from you? Stop moving the goal posts!

@twopirllc did you read https://github.com/twopirllc/pandas-ta/issues/367#issuecomment-896509585 ?

[twopirllc]

@reza1615,

Of course I read it . That is how I know you didn't provide the 4 timings that I clearly requested. Yet, you provided something else.

Anyhow, I will take them Issues in turn using the parquet data you provided.

[reza1615]

I sent you the full log of my run so you can simply compare these algorithm speed with each. The goal of this report is to show that these algorithms are slower that the rest algorithms. So I didnt go off the topic

[twopirllc]

@reza1615,

Pandas TA Indicator Performance

Since we are doing some indicators, might as do them all. Please screenshot the DataFrame of the performance results, like the first DataFrame below. Which is currently the Slowest 10 Indicators.

Be aware that this is not the correct nor best way to run your Multiple Window Combinations strategy that you also included in your comment here. I'll address that later using Pandas TA. However, I recommend checking out vectorbt, which also can easily hook up with Pandas TA, to do this Multiple Window Combinations strategy, like in his example notebook BitcoinDMAC.ipynb starting with section Multiple Window Combinations at cell 22.

import pandas as pd
import pandas_ta as ta

df = pd.read_parquet("df_file")
print(df.shape)

def performance(df:pd.DataFrame, excluded:list, other:list, top:int = None, sortby:str = "secs", ascending:bool = False, places:int = 5):
    if df.empty: return
    top = int(top) if isinstance(top, int) and top > 0 else None

    data = []
    df = df.copy()
    indicators = df.ta.indicators(as_list=True, exclude=excluded)
    
    header = "Quickest" if ascending else "Slowest"
    header = f"{header} {top} Indicators" if top is not None else f"{header} Indicators"

    if len(indicators):
        for indicator in indicators:
            result = df.ta(indicator, timed=True)
            ms = float(result.timed.split(" ")[0].split(" ")[0])
            data.append({"indicator": indicator, "secs": round(0.001 * ms, places), "ms": ms})

    if len(other) > 0:
        for indicator in other:
            result = df.ta(indicator, timed=True)
            ms = float(result.timed.split(" ")[0].split(" ")[0])
            data.append({"indicator": indicator, "secs": round(0.001 * ms, places), "ms": ms})
        
    pdf = pd.DataFrame.from_dict(data)
    pdf.set_index("indicator", inplace=True)
    pdf.sort_values(by=sortby, ascending=ascending, inplace=True)

    print(f"\n{header}\n{len(header) * '='}")

    if top is not None:
        return pdf.head(top)
    return pdf


excluded = ["above", "above_value", "below", "below_value", "cross", "cross_value", "long_run", "short_run", "td_seq", "tsignals", "vp", "xsignals", "ichimoku"]
other = ["long_run", "short_run", "vp", "td_seq"]
pr = performance(rdf, excluded, other, top=10, ascending=False, places=6)
pr.style.background_gradient("autumn_r")

For instance, with an Apple MacBoook M1 with 8 cores and 16GB RAM. The Slowest 10 Indicators, ichimoku excluded, in order given your ohlcv file "df_file" with shape: (57601, 9) are:

Please post the results of your 32 core machine and upload a "csv" of the pr DataFrame so I can see the performance contrast with a quicker machine. I do not need the stdout results.

Also, I already know that td_seq is slow as mentioned in the README and in my prior comment.

Pandas TA All Strategy Performance

Additionally, I would like you to share your output of running the ta.AllStrategy.

import pandas as pd
import pandas_ta as ta

df = pd.read_parquet("df_file")
df.ta.cores = 31
df.ta.strategy(ta.AllStrategy, timed=True, verbose=True)

Results from my MacBook:

Thanks, KJ

[twopirllc]

@reza1615,

The df.ta.strategy() method was built so you did not have build your own loop executor. All you really have to do to use df.ta.strategy() is setup a list of dicts for the ta argument of ta.Strategy() as illustrated in the Pandas TA Strategies Section of the README, like the "Custom" Strategy, and df.ta.strategy() will largely do the rest; help(df.ta.strategy) for arguments.

For example in the code below, for each indicator in the list, the OS has to initialize a Multiprocessing Pool 31 cores for one indicator and then tear down that Multiprocessing Pool. Which unfortunately is a huge waste of OS resources and consequentially adds processing time. One simple fix, but still inefficient for this use case, is to set df.ta.cores = 0; this allows df.ta.strategy() to not initialize and tear down the 31 core Multiprocessing Pool for each indicator.

for strategy in ['alma', 'sma', 'ema', 'supertrend']:
    ta_list=[{"kind": strategy,  "prefix":f"RA"}]
    # Use 0 cores when you have a few and use more cores when you have many
    df.ta.cores = 31
    df.ta.strategy(
        ta.Strategy(name=f"{strategy}", description=f"{strategy}",  ta=ta_list),
        timed=True
    )

It would have been easier to do:

for strategy in ['alma', 'sma', 'ema', 'supertrend']:
    df.ta(strategy, timed=True, verbose=True)

But all that is unnecessary and doesn't utilize the Multiprocessing Pool when you need it.

prefix = "RA"

# Create the ta list first
ta_list=[
    {"kind": "sma", "length": 10, "prefix": prefix},
    {"kind": "sma", "length": 20, "prefix": prefix},
    {"kind": "ema", "length": 8, "prefix": prefix},
    {"kind": "ema", "length": 21, "prefix": prefix},
    {"kind": "alma", "sigma": 3.5, "prefix": prefix},
    {"kind": "'supertrend'", "sigma": 3.5, "prefix": prefix},
]

# Create the Strategy
rastrat = ta.Strategy(name=f"RA", description=f"Custom RA", ta=ta_list)

# Execute the Strategy
df.ta.strategy(rastrat, cores=0, timed=True) # Few indicators => set cores = 0
print(df)

Multiple Window Combinations with Pandas TA

# Define Window Combinations
length_grid = [_ for _ in range(10, 60, 10)]
indicators = ['sma','alma','mcgd','linreg']

def window_combos(indicators:list, lengths:list, prefix:str = None, verbose:bool = True):
    result = []
    for kind in indicators:
        for length in length_grid:
            if isinstance(prefix, str) and len(prefix) > 0:
                result.append({"kind": kind, "length": length, "prefix": prefix})
            else:
                result.append({"kind": kind, "length": length})
    
    if verbose:
        print(f"Total indicators[{len(result)}]:\n{result}\n") # Verify arguments
    return result

# Build Window Combinations
wc = window_combos(indicators, length_grid, prefix="RA", verbose=False)

# Execute Strategy
df.ta.strategy(ta.Strategy("Reza TA", ta=wc), timed=True, verbose=True)
print(f"[i] Columns[{len(df.columns)}]: {', '.join(df.columns)}")

df.iloc[-16:,-7:].style.background_gradient() # Take a peak

Also looking forward to your Performance Report with the code from my prior comment on a 32 core machine (OS?).

Hope this helps! Let me know if anything is unclear.

Kind Regards, KJ

[reza1615]

KJ

Thank you for codes and explanations.

My system is AMD Ryzer 9 5950X 16-core and 32 Threads, Ram 64 Os: Ubuntu 20.04

Also here is the pr.csv pr.csv

[reza1615]

Be aware that this is not the correct nor best way to run your Multiple Window Combinations strategy that you also included in your comment here. I'll address that later using Pandas TA. However, I recommend checking out vectorbt, which also can easily hook up with Pandas TA, to do this Multiple Window Combinations strategy, like in his example notebook BitcoinDMAC.ipynb starting with section Multiple Window Combinations at cell 22.

In that comment i have two for loops in put the startegy in side the first one to have timing seperatly for each indicator. your multiple window solution shows timing for all indicatores in one shot. Thank you for you codes.

[twopirllc]

@reza1615,

Thank you for codes and explanations. 😎

No worries. I hope it is helpful.

My system is AMD Ryzer 9 5950X 16-core and 32 Threads, Ram 64 Os: Ubuntu 20.04

Excellent! Thank you! Does the code provided help the processing speed of your chosen indicators and use cases?

your multiple window solution shows timing for all indicatores in one shot.

Yes. The setup is the most important part to utilize the benefit of df.ta.strategy()'s multiprocessing capabilities when you have a large set of indicators. Also depending on the number of indicators in a Strategy, you will have to determine, through some trial and error, the appropriate number of cores and possibly chunksize to maximize your speed.

Nevertheless, time permitting, I am trying to speed up slower indicators as well. Any contribution is largely accepted, even if it is as simple as some documentation. 😎

I appreciate your patience in this matter (and dealing with me at times).

Kind Regards, KJ

[reza1615]

I found the issue. Alma and others (in this topic) for longer lengths like 1200 or 12000 are slower than shorter like 3-10. It seems some part of the algorithms which is related to the window defining is slow.

[reza1615]

length = 10

length = 1000

length = 2000

• fwma shows error
• alma was so slow I excluded

[twopirllc]

@reza1615,

I found the issue. Alma and others (in this topic) for longer lengths like 1200 or 12000 are slower than shorter like 3-10.

That goes without saying. Longer lengths require long processing times in comparison to shorter lengths. Furthermore, what use is an indicator length greater than 500 anyhow? I would like to see convincing evidence that 1,200 or 12,000 or even 12,000,000 length indicators provide significant value to the humongous AI/ML matrices that you planning to push it through.

Additionally some indicators, sma for example, with have leading NaNs as long as it's length that will have to either be dropped entirely and thus removing potential useful data points or backfilled with values that could throw of the results of your AI/ML process.

It seems some part of the algorithms which is related to the window defining is slow.

Yes, like some of algorithms in the "slow" list. They are on my To Do List. I am certain these would be fixed sooner if more people would be willing to contribute. Unfortunately, I spend more time addressing Issues than I get to add new features and indicators or fix slow indicators.

KJ

[twopirllc]

length = 10

length = 1000

length = 2000

* **fwma** shows error

* **alma** was so slow I excluded
  ![image](https://user-images.githubusercontent.com/2202483/129439744-00a3cc98-8e7f-4255-8e0d-4efa5dd95536.png)

I am trying to fix alma. Also you do realize that fwma calculates the Fibonacci numbers up to it's length... so obviously it will error out at some point and Sterling's calculation will not help either.

Furthermore I fail to see how indicators with lengths greater than 500 are useful. If you have evidence to the contrary, I would like proof.

KJ

[reza1615]

I created a grid search for all indicators between 2-16000 length and selected the useful features. the feature selector chose many features that are more than 8000 almost 10% of features have length more than 8000. around 82 unique algorithms was selected It seems combination of short length and medium and long length has good impact on ml predictions. Also more than 12000 doesn't make sense

[twopirllc]

I created a grid search for all indicators between 2-16000 length and selected the useful features. the feature selector chose many features that are more than 8000. almost 10% of features have length more than 8000. around 82 unique algorithms was selected

Are these 10% of features with lengths greater than 8000 consistent among different tickers/symbols? Or do you have to consistently grid search per ticker to mine these 10%? If that's the case, this Kitchen Sink approach is very costly and overkill in my opinion. But if it's profitable for you 🍀.

It seems combination of short length and medium and long length has good impact on ml predictions.

Of course combos of short, medium, and long lengths are necessary. That goes without saying.

Also more than 12000 doesn't make sense

I know, it was hyperbole. However it's funny because TA Lib's maximum period/lengths are capped at 100,000.

[reza1615]

comparing to just having some combination of features kitchen sink had Round 10-20% improvements in F1 score of my model. As you said it is costly and seeking for less costly approach. do you have any combination suggestion?

[reza1615]

I created a grid search for all indicators between 2-16000 length and selected the useful features. the feature selector chose many features that are more than 8000. almost 10% of features have length more than 8000. around 82 unique algorithms was selected

Are these 10% of features with lengths greater than 8000 consistent among different tickers/symbols? Or do you have to consistently grid search per ticker to mine these 10%? If that's the case, this Kitchen Sink approach is very costly and overkill in my opinion. But if it's profitable for you 🍀.

I just tested on a ticker. I expanded the kitchen sink 😅 to close, volume, high, low ,.. and used these indicators for all prices of the candles. It was processing 400Gig features.

[twopirllc]

@reza1615,

do you have any combination suggestion?

You should have a minimum of two models. A baseline model that has the standard defaults, since most people do not understand TA indicators and the math behind them. It gives you an idea of what the mainstream individual sees. It may also be useful to contrast against non-standard values you find from your Kitchen Sink.

For example like the Common Strategy (below) that is frequently used on Financial Market Programs/TV like Bloomberg TV, MSNBC, et al

CommonStrategy = Strategy(
    name="Common Price and Volume SMAs",
    description="Common Price SMAs: 10, 20, 50, 200 and Volume SMA: 20.",
    ta=[
        {"kind": "sma", "length": 10},
        {"kind": "sma", "length": 20},
        {"kind": "sma", "length": 50},
        {"kind": "sma", "length": 200},
        {"kind": "sma", "close": "volume", "length": 20, "prefix": "VOL"}
    ]
)

Of course you will need to extend it to include things like rsi et al since the average user typically uses a more than those.

I just tested on a ticker. I expanded the kitchen sink 😅 to close, volume, high, low ,.. and used these indicators for all prices of the candles.

What do you mean? Like you used macd with different inputs like volume, high, et al?

It was processing 400Gig features.

That's massive! Of course, that depends on your trading time frame and asset class, crypto in your case. The lower timeframes like tick, 1min, et al is a larger data set than high timeframes like 1H, 1D, 1W, ....

[reza1615]

What do you mean? Like you used macd with different inputs like volume, high, et al? Yes, I created the below columns and generated all indicators by grid search on the below columns for the close argument and different lengths and std,....

• original data: close
• diff data: high-low
• diff with last data: close.shift(1)- close
• ... For the indicators which get high, low, close, open (more than the close argument for price) I didn't touch them. This kitchen sink was just for cases like alma which just get close and length arguments.

[reza1615]

@reza1615,

do you have any combination suggestion?

You should have a minimum of two models. A baseline model that has the standard defaults, since most people do not understand TA indicators and the math behind them. It gives you an idea of what the mainstream individual sees. It may also be useful to contrast against non-standard values you find from your Kitchen Sink.

For example like the Common Strategy (below) that is frequently used on Financial Market Programs/TV like Bloomberg TV, MSNBC, et al

CommonStrategy = Strategy(
    name="Common Price and Volume SMAs",
    description="Common Price SMAs: 10, 20, 50, 200 and Volume SMA: 20.",
    ta=[
        {"kind": "sma", "length": 10},
        {"kind": "sma", "length": 20},
        {"kind": "sma", "length": 50},
        {"kind": "sma", "length": 200},
        {"kind": "sma", "close": "volume", "length": 20, "prefix": "VOL"}
    ]
)

Of course you will need to extend it to include things like rsi et al since the average user typically uses more than those.

I created the baseline and Its F1 was 0.3 after that kitchen sink and feature selection the F1 became 0.55 with cross-validation but this feature selection seems over-fitted on the current dataset after getting a new dataset with 1-2 days lag, the features selections performance goes down. It means every day I should do this kitchen sink which is not possible. I wish I could find a good combination and just get that indicator combination but it seems the feature selection also can over-fit on the market nature.

[twopirllc]

Hey @reza1615,

I have a quicker alma on the development branch if you want to test it out first.

$ pip install -U git+https://github.com/twopirllc/pandas-ta.git@development

Thanks, KJ

[xmatthias]

@twopirllc sorry to chip in here (hope it's not too much off-topic) ... are there any plans (or reasons not to) use cython for certain indicators?

Some indicators (e.g. Supertrend) (have to) use loops to iterate through the whole dataframe, as the calculation of "current row" uses the output of the same calculation for "previous row".

obviously, starting with a certain candle size, performance improvements wil be pretty huge. While not for supertrend, something similar was shown here: https://github.com/freqtrade/technical/issues/148#issuecomment-890877054 (unfortunately it's currently missing the cython code).

[twopirllc]

Hello @xmatthias,

sorry to chip in here (hope it's not too much off-topic) ... are there any plans (or reasons not to) use cython for certain indicators?

No worries. Do you have indicators in mind that need a Cython implementation? Also Pandas TA has wrapped most of TA-Lib indicators if they have it installed in their environment.

I could at some point, I just don't know when. 🤷🏼‍♂️

obviously, starting with a certain candle size, performance improvements wil be pretty huge.

True

While not for supertrend, something similar was shown here: freqtrade/technical#148 (comment) (unfortunately it's currently missing the cython code).

Would've been nice.

KJ

[xmatthias]

From memory (and looking at the list above, i'd say supertrend and vidya) - but in theory every indicator which can't be calculated in a vectorized manner would/should be a candidate for this.

I'd be curious of this myself, as i've never used cython before (at least not for anything serious) - and i'm not sure what that'd mean for installability (i assume ci would have to build wheels for all supported/major platforms/combinations ... ?)

Obviously it'd be pointless for SMA or other "easy to calculate" indicators (indicators where the calculation can be vectorized).

[twopirllc]

@xmatthias

I'd be curious of this myself, as i've never used cython before (at least not for anything serious) - and i'm not sure what that'd mean for installability (i assume ci would have to build wheels for all supported/major platforms/combinations ... ?)

Same. I don't know either.

From memory (and looking at the list above, i'd say supertrend and vidya) - but in theory every indicator which can't be calculated in a vectorized manner would/should be a candidate for this. Obviously it'd be pointless for SMA or other "easy to calculate" indicators (indicators where the calculation can be vectorized).

Right. It's something to consider. Plus there is (recent) GPU interest #375 with cuDF.

[xmatthias]

Plus there is (recent) GPU interest #375 with cuDF.

I'd evaluate that VERY closely if it's worth the effort (both to implement and maintain). Considering that most users will probably not work with "very huge" dataframes (10s of millions of lines) - i'm not sure if gpu will give the boost people hope for.

The library would become rather difficult to install (think ta-lib - but potentially worse as you'll have to have the right Grafics card drivers) - which will increase the support burden of people raising issues that are unable to install.

The code will most likely become rather complex (-> probably no longer fun to work in) - and setup / teardown of the GPU context might have an impact on timing as well, so in many cases you'd not be much faster in the GPU (would need evaluation though).

For a (probably very) limited set of indicators, sure, it can be a boost in certain usecases - but maybe these should be done in a seperate, standalone "gpu-ta" library.

pandas-ta can then optionally depend on that, and offer "non-gpu" fallback code if that library is not available or fails to initialize (grafic card drivers are fun ...) - but it'd be fully optional - and logically seperate libraries.

[twopirllc]

@xmatthias,

I'd evaluate that VERY closely if it's worth the effort (both to implement and maintain). Considering that most users will probably not work with "very huge" dataframes (10s of millions of lines) - i'm not sure if gpu will give the boost people hope for.

Thanks for the insight.

The library would become rather difficult to install (think ta-lib - but potentially worse as you'll have to have the right Grafics card drivers) - which will increase the support burden of people raising issues that are unable to install.

The code will most likely become rather complex (-> probably no longer fun to work in) - and setup / teardown of the GPU context might have an impact on timing as well, so in many cases you'd not be much faster in the GPU (would need evaluation though).

Good point. That's what I was thinking as well.

For a (probably very) limited set of indicators, sure, it can be a boost in certain usecases - but maybe these should be done in a seperate, standalone "gpu-ta" library.

pandas-ta can then optionally depend on that, and offer "non-gpu" fallback code if that library is not available or fails to initialize (grafic card drivers are fun ...) - but it'd be fully optional - and logically seperate libraries.

Good idea. Pretty sure I don't want a combined library but perhaps an independent addon.

Thanks for the insight KJ