deep-trading-agent
deep-trading-agent copied to clipboard
samre12
any plan to add SMA, RSI, ATR indices?
All these additional indicators are essentially functions of timeseries of Bitcoin prices and that too specifically closing prices. A convolutional network is able to compute these functions and thus providing them explicitly is redundant. Rather, introducing highest and lowest prices in an interval and the volume of currency traded gives a better view of the changing trends.
However, I have an earlier commit in the master branch which contains calculation of these indicators and passes them as state representation to the Q network.
I disagree. There is a contextual LSTM model which I think would fair better results if the context of price were included in training. https://arxiv.org/pdf/1602.06291.pdf
a successful agent needs to decouple from training data and level up in risk / reward plays. the agent should generalize on other data. If you take an Elliot wave trader on any stock - they can make/ predict the counts / predict a couple of scenarios and place the bet accordingly. these bot agents need to grasp these contexts. If RSI is bearish - then in that context - we're anticipating price swings downwards. If the training is over fit here - sure - then the bot - if given the same prices - it will get the bet right. but it would be better to anticipate if there's a high risk reward scenario to put the farm on bet (like George Soros did by borrowing billions of dollars to bet against the pound) becoming one of the richest men in the world. how did he do this? he took a bet that had very little risk / he only had to pay interest on loan.
we could branch off into competing forks? the idea being the best code wins to get merged into master.
this preprocessing code expands out features to a very wide selection of indicators. This could be a curse on dimensionality - but we could iron it down
eg. cci is useful indicator for bearish / bullish channels. cci = CCI(inputs)
if you fast forward to a finished training sample / given any time + price - you going to get a hodgepodge of signals on different time frames. 5 min / 1 hr / 2 hr etc. we need to incorporate more granularity on risk / reward into strategy. https://github.com/maxpaulus/alexis_maxime/blob/bc40344627011f191a5f137fb7988c59355b179c/DataProcessing/strategy_with_prediction.py
https://github.com/bfmill253/ForexDL/blob/3fcbd8cc92ff98363c478b5cdae5c0a2959ef3fc/ForexDL/Data.py
def expandData(startTime="0", instrument="EUR_USD", endTime="0"):
close, high, low, volume, openMid, time = getData(startTime=startTime, instrument=instrument, endTime=endTime)
inputs = {
'open': openMid,
'high': high,
'low': low,
'close': close,
'volume': volume
}
#print (type(inputs.get('open')))
#Overlap Studies
sma = SMA(inputs)
bbuper, bbmiddle, bblower = BBANDS(inputs)
dema = DEMA(inputs)
ema = EMA(inputs)
htTrend = HT_TRENDLINE(inputs)
kama = KAMA(inputs)
ma = MA(inputs)
mama, fama = MAMA(inputs)
midpoint = MIDPOINT(inputs)
midprice = MIDPRICE(inputs)
sar = SAR(inputs)
sarext = SAREXT(inputs)
t3 = T3(inputs)
trima = TRIMA(inputs)
wma = WMA(inputs)
#Momentum Indicators
adx = ADX(inputs)
adxr = ADXR(inputs)
apo =APO(inputs)
aroonDown, aroonUp = AROON(inputs)
aroon = AROONOSC(inputs)
bop = BOP(inputs)
cci = CCI(inputs)
cmo = CMO(inputs)
dx = DX(inputs)
macd, macds, macdh = MACD(inputs)
macdext, macdexts, macdexth = MACDEXT(inputs)
macdfix, macdfixs, macdfixh = MACDFIX(inputs)
mfi = MFI(inputs)
minusdi = MINUS_DI(inputs)
minusdm = MINUS_DM(inputs)
mom = MOM(inputs)
plusdi = PLUS_DI(inputs)
plusdm = PLUS_DM(inputs)
ppo = PPO(inputs)
roc = ROC(inputs)
rocp = ROCP(inputs)
rocr = ROCR(inputs)
rocr100 = ROCR100(inputs)
rsi = RSI(inputs)
slowk, slowd = STOCH(inputs)
fastk, fastd = STOCHF(inputs)
rsifastk, rsifastd = STOCHRSI(inputs)
trix = TRIX(inputs)
ultosc = ULTOSC(inputs)
willr = WILLR(inputs)
#volume indicators
ad = AD(inputs)
adosc = ADOSC(inputs)
obv = OBV(inputs)
#cycle indicators
ht_dcperiod = HT_DCPERIOD(inputs)
ht_dcphase = HT_DCPHASE(inputs)
ht_phasor_inphase, ht_phasor_quadrature = HT_PHASOR(inputs)
ht_sine, ht_leadsine = HT_SINE(inputs)
ht_trendmode = HT_TRENDMODE(inputs)
#price transform
avgprice = AVGPRICE(inputs)
medprice = MEDPRICE(inputs)
typrice = TYPPRICE(inputs)
wclprice = WCLPRICE(inputs)
#volatility ind
atr = ATR(inputs)
natr = NATR(inputs)
trange = TRANGE(inputs)
#pattern rec
twocrows = CDL2CROWS(inputs)
threeblackcrows = CDL3BLACKCROWS(inputs)
threeinside = CDL3INSIDE(inputs)
threelinestrike = CDL3LINESTRIKE(inputs)
threeoutside = CDL3OUTSIDE(inputs)
threestarsinsouth = CDL3STARSINSOUTH(inputs)
threeWhiteSoldiers = CDL3WHITESOLDIERS(inputs)
abandonBaby = CDLABANDONEDBABY(inputs)
advanceBlock = CDLADVANCEBLOCK(inputs)
beltHold = CDLBELTHOLD(inputs)
breakAway = CDLBREAKAWAY(inputs)
closingM = CDLCLOSINGMARUBOZU(inputs)
consealBabySwall = CDLCONCEALBABYSWALL(inputs)
counterAttack = CDLCOUNTERATTACK(inputs)
darkCloudCover = CDLDARKCLOUDCOVER(inputs)
doji = CDLDOJI(inputs)
dojiStar = CDLDOJISTAR(inputs)
dragonFlyDoji = CDLDRAGONFLYDOJI(inputs)
engulfing = CDLENGULFING(inputs)
eveningDojiStar = CDLEVENINGDOJISTAR(inputs)
eveningStar = CDLEVENINGSTAR(inputs)
gapSideSideWhite = CDLGAPSIDESIDEWHITE(inputs)
graveStoneDoji = CDLGRAVESTONEDOJI(inputs)
hammer = CDLHAMMER(inputs)
hangingMan = CDLHANGINGMAN(inputs)
harami = CDLHARAMI(inputs)
haramiCross = CDLHARAMICROSS(inputs)
highWave = CDLHIGHWAVE(inputs)
hikkake = CDLHIKKAKE(inputs)
hikkakeMod = CDLHIKKAKEMOD(inputs)
homingPigeon = CDLHOMINGPIGEON(inputs)
identical3Crows = CDLIDENTICAL3CROWS(inputs)
inNeck = CDLINNECK(inputs)
invertedHammer = CDLINVERTEDHAMMER(inputs)
kicking = CDLKICKING(inputs)
kickingByLength = CDLKICKINGBYLENGTH(inputs)
ladderBottom = CDLLADDERBOTTOM(inputs)
longLeggedDoji = CDLLONGLEGGEDDOJI(inputs)
longLine = CDLLONGLINE(inputs)
maruBozu = CDLMARUBOZU(inputs)
matchingLow = CDLMATCHINGLOW(inputs)
matHold = CDLMATHOLD(inputs)
morningDojiStar = CDLMORNINGDOJISTAR(inputs)
morningStar = CDLMORNINGSTAR(inputs)
onNeck = CDLONNECK(inputs)
piercing = CDLPIERCING(inputs)
rickShawman = CDLRICKSHAWMAN(inputs)
riseFall3Methods = CDLRISEFALL3METHODS(inputs)
seperatingLines = CDLSEPARATINGLINES(inputs)
shootingStar = CDLSHOOTINGSTAR(inputs)
shortline = CDLSHORTLINE(inputs)
spinningTop = CDLSPINNINGTOP(inputs)
stalledPattern = CDLSTALLEDPATTERN(inputs)
stickSandwhich = CDLSTICKSANDWICH(inputs)
takuri = CDLTAKURI(inputs)
tasukiGap = CDLTASUKIGAP(inputs)
thrusting = CDLTHRUSTING(inputs)
triStar = CDLTRISTAR(inputs)
unique3River = CDLUNIQUE3RIVER(inputs)
upsideGap2Crows = CDLUPSIDEGAP2CROWS(inputs)
xSdieGap3Methods = CDLXSIDEGAP3METHODS(inputs)
data = np.array([close, high, low, volume, openMid, time, sma, bbuper, bbmiddle, bblower, dema, ema, htTrend, kama, ma, mama, fama, midpoint, midprice, sar, sarext, t3, trima, wma, adx, adxr, ppo, apo, aroonDown, aroonUp, aroon, bop, cci, cmo, dx, macd, macds, macdh, macdext, macdexts, macdexth, macdfix, macdfixs, macdfixh, mfi, minusdi, minusdm, mom, plusdi, plusdm, roc, rocr, rocp, rocr100, rsi, slowk, slowd, fastk, fastd, rsifastk, rsifastd, trix, ultosc, willr, ad, adosc, obv, ht_dcperiod, ht_dcphase, ht_phasor_inphase, ht_phasor_quadrature, ht_sine, ht_leadsine, ht_trendmode, avgprice, medprice, typrice, wclprice, atr, natr, trange, twocrows, threeblackcrows, threeinside, threelinestrike, threeoutside, threestarsinsouth, threeWhiteSoldiers, abandonBaby, advanceBlock, beltHold, breakAway, closingM, consealBabySwall, counterAttack, darkCloudCover, doji, dojiStar, dragonFlyDoji, engulfing, eveningDojiStar, eveningStar, gapSideSideWhite, graveStoneDoji, hammer, hangingMan, harami, haramiCross, highWave, hikkake, hikkakeMod, homingPigeon, identical3Crows, inNeck, invertedHammer, kicking, kickingByLength, ladderBottom, longLeggedDoji, longLine, maruBozu, matchingLow, matHold, morningDojiStar, morningStar, onNeck, piercing, rickShawman, riseFall3Methods, seperatingLines, shootingStar, shortline, spinningTop, stalledPattern, stickSandwhich, takuri, tasukiGap, thrusting, triStar, unique3River, upsideGap2Crows, xSdieGap3Methods])
return data
N.B. all these functions are out of the box with TALib.
Competing forks seems to nice idea but currently I don't have much ideas remaining to improve the performance of the agent. In that scenario may be collaboration and sharing ideas seems as a better alternative. Also, these are a hell lot of indicators in the preprocessing code, currently the input is closing, high, low and volume normalised by their values at previous time steps. Maybe 5-6 more features may steal the deal. What say, which of these should be selected?
stepping back - rather than adding these as features into data set why not have q learning action decide which action (indicator) to take/use? eg. BUY
- BUYONMOMENTUM
- BUYONVOLATILITY -BUYONVOLUME -BUYON... SELL
- SELLONMOMENTUM
- SELLONVOLATILITY -SELLONVOLUME -SELLON... HOLD
- HOLDONMOMENTUM
- HOLDONVOLATILITY -HOLDONVOLUME -HOLDON...
the action would need to observe the corresponding value of result & feedback into system. it reminds me of reinforcement learning for helicopters.
I skimmed through this code by @dandxy89 - but not sure how one could cherry pick this code. You need neural net to have a notion of a spread of observed values. RSI is high - (ai should learn - don't enter trade here). MACD is showing bearish divergence - (be careful - don't buy). MACD is showing bullish divergence - go all in.
https://github.com/dandxy89/rf_helicopter/blob/master/Model/Q_Learning_Agent.py
alternative could be create an array of indicators that ai will use and let them learn to trade based off indicator. we could start with one - the SMA / simple moving average crossover strategy -- the ai should be able to leverage this for better trades. https://www.youtube.com/watch?v=3N8Pl26N2x4
then step 2- somehow cross bread / concatenate other signals into a super strategy. I want RSI ai + CCI ai + SMA ai. the training data - doesn't necessarily need to be authentic - just the reward function needs to be optimized for most profits. If you go to month 1 of btc trading and applied SMA without ai - you will get a base line of how much should be won (assuming you successfully entered / exited every trade).
Seems like a nice idea, need to do a bit more research on this topic. I was also considering using Google Trends per minute statistics as an input to the model as they serve as a proxy to the market sentiments. But currently I am having issue acquiring per minute trends due to rate limits of the API.
from @antkve, neuralnetcrypto.py is a trading bot for crypto markets (bitcoin, ethereum, etc). It trains on historical price data, learning to notice patterns in the Relative Strength Index and Moving Average indicators. It does pretty well atm but i'll be extending it to analyze more indicators soon:
https://github.com/antkve/neuralnet/blob/master/neuralnetcrypto.py
related - https://github.com/songyunfan/DeepRLTrading/blob/master/prediction-using-RL.pdf
I think if we sunset the buy / sell / hold actions - and simply make the code predict prices - then the scope of the problem will be more fitting. Then we use the trained ai indicator to trade.
the reinforcement paper use these components in vector.
we can use backtrader to test strategies for profitability. It's very simple. https://github.com/ianmobbs/Backtrader/blob/2b8b06f56a9530a86e3c08a7fa21495fe7c56643/main.py
from my understanding of the paper - and reading this graph
predictions of 1 day ahead - really sucked predictions 8 days out - worked well. predictions of 16 days and beyond sucked.
seems like the more indicators = the better - here author is using 18.
https://launchpad.ai/blog/trading-bitcoin
solved https://github.com/sahpat229/RL-Trading/blob/460f1d7944f5220b4d0a799546ed74c6765f02ac/DetPolicyGradeNoHistory/baselines/ddpg/datacontainer.py
there's a config file that's appending these features
eg. here acceleration in price
if conf['ac96/96'] is True:
ac96_per_asset = []
for asset in range(num_assets):
p = closes[asset, :].astype(np.float64)
pavgs = talib.MA(p, timeperiod=96) #avg(p, 12)
pavgsavgs = talib.MA(p / pavgs, timeperiod=96) # avg[p/avg(p, 12), 12]
item = (p / pavgs) / pavgsavgs
avgs = talib.MA(item, timeperiod=96)
stddevs = talib.STDDEV(item, timeperiod=96)
zScores = np.nan_to_num((item - avgs) / stddevs)
ac96_per_asset.append(zScores)
ac96_per_asset = np.array(ac96_per_asset).astype(np.float32)
features.append(ac96_per_asset)
So I think the state representation should include
- rate of change of closing, high and low prices and volume of the past say 180 minutes (history length)
- sma(5), sma(20) and sma(60)
- rsi Can you explain more about acceleration in price?
https://launchpad.ai/blog/trading-bitcoin are using 18 indicators but they are not providing history length of historical prices in the state representation, so I guess apart from all the historical prices, it is sufficient to provide certain technical indicators.
Also I am thinking of passing these technical indicators directly to the fully connected layers at the end of the model after the convolutional layer and recurrent layer on the historical prices.
recommend https://github.com/tensorflow/models/pull/1509/files we can use those indicates directly.
This is the direct link to the file extracting all the features from the historical price dataset as mentioned by @yunhaizhu . Also we need to first decide upon the time periods for which to calculate the indicators.
to throw a spanner in the works - consider we need an RNN to discriminate how long the time periods to use.
I think the approach is to somehow in prediction to be able to pass in time frame - rather than updating embeddings. trading is all about timing and direction.
At any point in technicals - there will be dozens on conflicting directions.
buy on monthly / sell on daily / buy on 5 minute.
Need to be able to discriminate which time has more weight.
https://www.tradingview.com/symbols/BTCUSD/technicals/
I am currently working in dev branch of this repository where I have integrated samre12/gym-cryptotrading as the environment. I am planning to add the decided technical indicators as a part of state representations first in the environment repository and then train the agent in it.
Another thing to consider is to how to normalise the input technical indicators. As @johndpope mentioned in an earlier comment, we can normalise them based on predefined maximum and minimum values of the indicators. We need to decide upon these values as well since they will largely affect the training process.
Working with predefined maximums and minimums raises an issue in the sense that the price of Bitcoin varies in extremes in the time period under consideration. Using fixed values for the entire time frame will render certain values very low in certain cases and very high in certain cases sending the network in shocks. Take as an example the case of SMA.
Regarding normalisation of the input to the network, I have found this which describes a general method to normalise the input in a reinforcement learning problem using online algorithms to calculate mean and variance.
Here I can also apply same technique as BN to adapt the transform with the learning using gamma and beta (refer here).
I did a lot of validation on technical indicators on a FCN net, and here are the main insights:
- Delta is key
Just adding the difference (delta) between the previous n-days close prices is key to increase model accuracy.
Also, change in momentum and change of change (second derivative) is key.
- Distance is key
Specifically, the distance between the closing price and key indicators such as
- Bollinger bands
- MACD
- 20/60 WMA
- 20/200 EMA
To some extent, these insights are almost too obvious to be true, but here you codify domain knowledge by making implicit observations explicit. Taking the infamous Bollinger bands as an example, when the closing price approaches the upper-band, usually a trend reversal is due. The distance between close and upper band captures exactly this and allows the net/agent to find a matching rule.
- A lot of garbage indicators are just garbage... RSI / ADX / CCI / all dojis and you name it. None of them actually improved model performance.
Calculating most commonly used indicators isn't terribly fun, so instead, I pull them together with the data from AlphaVantage and merge them over index. You may wonder what's the point of downloading indicator data, but as AlphaVantage offer tick resolution data for free, loading and caching them speeds things up noticeably.
I have stuffed all the above in a bunch of utils and released them here: https://github.com/marvin-hansen/StockUtils/blob/master/Main.py
Finally, in terms of RL, it seems that neuro evolution combined with Bayesian optimization seems to work quite well:
https://github.com/huseinzol05/Stock-Prediction-Models/blob/master/free-agent/evolution-strategy-bayesian-agent.ipynb
