FinRL
FinRL copied to clipboard
AI4Finance-Foundation
Paper Trading Demo does not handle tick level data
The paper trading demo expects to train on data in the form of OHLCV.
https://github.com/AI4Finance-Foundation/FinRL/blob/7c5deaff188176562309df600a86a5f89d4eb83a/tutorials/3-Practical/FinRL_PaperTrading_Demo.ipynb
This of course is great for any periodicity above 1 minute sampling.
However, it does not handle tick data i.e. time, bid price, ask price.
How can the demo be upgraded to cater for tick level data?
Much appreciated
Marc
Hi @marcipops In my opinion, 1-minute level trading is a viable way because with second-level trading your trading costs will overshadow your profits generated. So it may make our agent passive. But you can always train your model to trade data. In this project you can find the final_stream.py
There you can do the following in the main function to get real-time trade data
stream.subscribe_trades(trades,'TSLA')
Thanks @Athe-kunal
Appreciate the spread issue but I was hoping that at tick level it would give far more granularity to the agent, to allow it to observe the environment and take action (enter/exit) in a more timely way instead of waiting potentially a whole minute before taking action.
- Does the agent not take into account the spread in how often it executes trades?
- Does the agent cope with ticks properly? (they do not have a regular period of arrival e.g. 10 ticks can arrive in a second or none at all for seconds.)
Hi @marcipops My knowledge is constrained on this topic, so I think RL agents generally perform well with more data than granular data. Like including sentiment data in our state-space may give better results than having granular data. Also, the agent tries to capture the variation in data. That is if this is the current cash value and these are prices of assets, what actions would maximize rewards moving forward. Addressing your questions
- So, do you want to keep a constraint on trading frequency? Currently, I am not aware of any constraints based on the spread value.
- Second-level data comes with latency issues so we may have an erroneous dataset. Also, our main aim is to deploy the model live for real-time trading. It will be a bit infeasible with second-level trading as there are many I/O operations and computation overhead involved. So minute level data is a viable granular level for deploying models in real-time environment
Hi @Athe-kunal - yes agree - but I am not an expert at all in this area of RL. All I know is we are on the hunt for Alpha - aiming to build a richer data set to include e.g. fundamentals gives the RL more features to determine predictive patterns.
When you say "RL agents generally perform well with more data than granular data" - I am not so sure. Hopefully one of RL experts will advise. All I am suggesting is that we extend our periodicities to the next level of granularity to give more richness to the data and see if we can gain a performance advantage. Aren't the benefits we gain from moving from 30 min to 1 min the same as if we are moving from 1 min to tick level? We are getting to the individual trades, rather than use an arbitrary aggregation. I appreciate that this aggregation might have actually occurred as human traders needed a way of visualising patterns in the data, and they chose periodicities to do that - but here we are dealing with a machine doing that activity and I am concerned that we might miss out on an opportunity to feed RL with the best data we can get, to enable it to make the best predictions it can possibly achieve.
Your point 1. Actually, I am not suggesting to constrain anything - we constrain ourselves to 1 min at the moment, and I am looking to expand the frequencies we support.
Your point 2. By second-level do you mean 1-second periodicity? If you mean aggregation of ticks into 1 second samples, I am not suggesting that at all. My suggestion is to ingest the ticks in real-time. Tick data represents the timing of when trades have actually happened rather than being grouped together in 1 min samples. Ticks arrive irregularly e.g. 10 in one second during for example market open, or none over seconds, for example late a night when traders want a rest :). So what I am trying to say is Tick data reflects the absolute real-world granular profile of trading, as opposed to the aggregation of ticks into to an arbitrary 1-minute sample. We lose the profile of the trading over that 1 minute, in the same way, if we use 30 min data we lose the richness of the profile of 1 minute data. So all I am suggesting is that the more granular the environment, the more the RL model has to train to make even more precise predictions.
Sorry for the long explanation but thought the granularity might help :)
Does that logic work?
Hi @marcipops Thank you so much for explaining it in a detailed manner, this is extremely helpful If you go through this paper, here it is presented that increasing input dimensionality leads to an increase in performance. So from this observation, I extrapolated that instead of increasing the granularity, we can increase the state space size.
Also, as mentioned in this project, we are having tick level data. SO you are right in your description as the tick level data comes in, the dataset records and it is not forcibly clubbed together as a 1-min sample. Sorry, I misunderstood earlier. You can run the project in your local environment to see the values.
Actually, we can't conclude the types of data we need to feed to the RL. These all conclusions require empirical evidence. I am not much sure about training with bid/ask price data, because I was unable to figure out how to infer the model in real-time as making decisions in granular data incurs computational overhead.
We should definitely try out with more granular data to see if they are beneficial for our model. Looking forward to hearing from you to discuss this.
Hi @Athe-kunal thanks again for your insights and material to help inform the debate. Again, I must stress I am not an RL expert, so really appreciate any advice!
Reviewing the paper (as best as I could with limited RL knowledge!) I noted that the higher dimensionality features were learned by OFENet rather than imposed by a human (i.e. OHLCV). I have no issue with this at all if it gives an accurate higher-level representation of the raw data, rather than a human imposed dimensionality that does not preserve the characteristics of the raw data.
However, the financial industry introduced sampling (1 min and above) to aid human traders in understanding what was going on at different levels of abstraction. Unfortunately, in my view playing into the hands of the ‘curse of dimensionality’ i.e., making the data more sparse - reducing the number of data points - whilst increasing the dimensionality to represent the profile of the data points contained within the sample period.
Introducing the dimensionality of OHLCV is by nature lossy, i.e., with 1 min data one cannot generate the profile of all the trades that happened within the minute. So, it is not fully representative of raw data. I argue the need for raw data as input to the RL so it can learn from a more detailed environment AND give an ability to issue actions in a timelier manner than having to wait for the end of the sampling period before it can retrieve the state information to act on. It might present FinRL with an opportunity to generate the most accurate and timely predictions by using raw tick data.
The paper acknowledges the success of RL algorithms due to their ability to “learn complex tasks from raw sensory data” which is what we are debating here – taking raw data (i.e. tick level) input rather than an arbitrary aggregation. Also, it acknowledges the major driver of investigating increasing dimensionality was the challenge of high-frequency big-data collection and management (i.e. purview of dataops). "Whereas neural networks have the ability to automatically acquire task-specific representations from raw sensory data, learning representations usually requires a large amount of data. This is one of the reasons that the application of RL algorithms typically needs millions of steps of data collection. This limits the applicability of RL algorithms to real-world problems, especially problems in continuous control and robotics. This has driven tremendous research in the RL community to develop sample-efficient algorithms." This is entirely logical, but I argue that we should also address the challenge of big-data management, so we are then able to leverage high-frequency and thus high-volume granular data.
In your project, it looks like you are starting to address this issue – however, it only streams 1 min bars OHLCV, not tick-level? I am not sure whether Alpaca allows you to get access to tick data – it used to via a third-party Polygon.
On my part, I have had to create a streamer to capture tick data - I cannot use Alpaca as I am trading UK instruments, and my broker uses Lightstreamer. I am storing tickdata in daily files but only for a single instrument for now. I will need to create a dataprocessor to plug into FinRL and allow the training and testing to transparently extract the relevant span of data required from these files. My effort is primitive, and just to get something going. I will need to expand the pipeline processing and storage to collect and manage any instrument (the purview of FinRL-Meta). I am at this early stage in development and thus would be great to pool our minds as to how to deal with this big data management issue for the benefit of the FinRL effort.
There have been some major advances in dataops technologies of course, for example in my business we have used the managed variant of an open source dataops platform ‘Delta Lake’ (https://delta.io/) to enable us to collect and manage batch and streaming data in real-time through a data pipeline ready for data scientists to run their experiments through mlops. Also, it uses Spark to enable large-scale data processing. We will be able to manage unstructured data from social media, financial news etc, so we can feed into richer dimensionality. So I’m starting to develop an instance of Delta to handle the dataops part and encapsulate it in a FinRL dataprocessor.
Apologies again for the lack of brevity in my response but thought as it’s a foundational issue to FinRL it’s worth spending the time to collaborate! Appreciate your opinion and of course, if anyone else has insights on this topic, it would be most appreciated.
hi @Athe-kunal and @marcipops , I think this paper can help.
@marcipops Im doing like your work from a few months ago, Im appreciate you sharing plugin too FinRL. My idea is to train on tick data but due to the high transaction`s cost, the number of trades is daily or less(must be constrained)
Hi @skhonsha thanks - I will need to read that paper 3x :) Do those IPUs work under Spark?
Hi @skhonsha Those parameters would be from SB3. You can enable tensorboard logging and get the episode reward charts. Here is the code snippet to do the necessary changes for tensorboard logging.
Please would you point me to where FinRL has managed to work with tick-level data? All the examples I have seen so far have been for OHLCV bars, not ticks (datetime, bid price, ask price). Thanks!
Hi all, I'd like to offer some input as well.
I'm very interested in @marcipops's idea to somehow incorporate tick data into the project. This may be a very promising approach to take in an effort to milk any advantage we can get. For example, Astarag (@Athe-kunal), you may find this paper interesting, particularly on page 7, the authors show that training on minute-level data results in much stronger agents than training on daily data, simply exploiting the fact that minute-level resolution gives your models more data to train with, highlighting the "unreasonable effectiveness of data". It's possible that going a step further with tick data could produce even more of an edge.
Additionally, as @marcipops has said, the current usage of time-boxed data (1-min, 30-min, ...) is pretty arbitrary and possibly not optimal for statistical model training. If you think about it, the market doesn't close every 5 minutes and reopen for the next 5 minutes; these are just reductions we make to price data so that human traders can easily interpret patterns over a wide enough horizon. Using tick data allows machines to work with the time series data in its purest form, without introducing any human-level bias.
One possible way the input could be represented is by using the last N trades/ticks (N can be a hyperparameter tuned with cross-validation) along with a sort of speed/time-based factor that accounts for how long it took for this sequence to be generated. With tick data, trades can come in very fast or very slow, so it may be useful for an agent to know how fast/slow the market has recently been trading. Alternatively, this speed/time-based factor could be included at every step as the time or speed between the current tick and the previous tick. Sequence models like LSTMs may be more appropriate than the standard feed-forward multilayer perceptrons in making decisions here.
There are definitely a lot of exciting things that can be done with this. Feel free to ask me if I can clarify anything. - Momin
@hmomin you definitely appreciate the opportunity! :)
There will be the 'usual' computing problems to overcome:
- data management - streaming, storing, transformation (I have just started working on solving this problem by building a data pipeline with delta.io). I am currently streaming about 70K ticks a day, into daily files, but want to move this primitive solution into a data lake to make it available for training, testing and RL trading operations. Also, it presents an opportunity for the FinRL to use the SAME data source for the training and testing and operations. This is opposed to the present design where FinRL 'downloads' data to train, and then separately pulls data in on a periodic basis directly from Alpaca etc.
- compute - how can we run training fast enough (potentially daily?) on this data (presume this needs to be on Spark clusters). We should look at building on FinRL-Podracer work?
- RL model - to train, test, run and act on tick data irregularly, as it comes in, as opposed to the current regular periodic basis (sometimes ticks arrive in tens per second, and sometimes none at all for seconds). It also needs to be able to use the only available data - datetime, bid price, and ask price (i.e. no OHLCV). I've just found some previous work in this area: https://www.researchgate.net/publication/327464670_Deep_Reinforcement_Learning_in_High_Frequency_Trading/fulltext/5b909eea299bf114b7fccf13/Deep-Reinforcement-Learning-in-High-Frequency-Trading.pdf. As this was published in 2019, can we find whether this work has developed since then and see if we can build on it?
I am happy to lead on 1 and 2 above (with support!), if someone can please lead on 3, in a collaborative effort to build an RL model that can show the benefit of using tick data i.e. higher profitability in the trading performance? I am sure Wall St. would be very interested in a demo of this!
@hmomin just found more recent papers on DRL using the limit order book, that mentions the potential for LSTM which you suggested: https://arxiv.org/pdf/2101.07107.pdf and https://www.mdpi.com/1911-8074/13/4/78.
Hope that helps in exploring a potential solution.
Hi @marcipops and @hmomin. Thank you for such great insights. In RL, we generally have small models. So inference part is easy to handle, but the data aggregation part is challenging. So I would definitely like to collaborate with you in this field. Looking forward to it
Also, @marcipops and @hmomin, what are your slack handle names if you are a member of the FinRL Slack community. Would like to connect with you on this topic. Thanks in advance
Hi @Athe-kunal ,@marcipops and @hmomin. Great topic for me, thank you. Dears, Did you reproduce the results of the article I posted? @Athe-kunal, I test your notebook https://github.com/Athe-kunal/Reinforcement-learning-trading-agent-using-Google-trends-data on my data, unfortunately, they do not show good results compared to buy and hold! @marcipops, 1-Are you currently testing for one stock trading or portfolio allocation? 2- Do you think order books data can make modeling more accurate?
3-I get one stock tick data and orders book data, we have many rows in the orders book data that have not been traded. Do you think this data has an effect on modeling?
Also, @marcipops and @hmomin, what are your slack handle names if you are a member of the FinRL Slack community. Would like to connect with you on this topic. Thanks in advance
My slack is the same as my GitHub: @hmomin. Cheers!
Set up a slack channel and invited you @hmomin, @Athe-kunal , and tried add skhonsha but can't find Slack handle.
Hi @skhonsha Which notebook did you try to run from the repo https://github.com/Athe-kunal/Reinforcement-learning-trading-agent-using-Google-trends-data
Yes, in this article, I did some testing on this and it was able to get a better result for volatile stocks compared to B&H. What tickers did you use for testing? You have to be careful in choosing the keyword and category while downloading the google trends data.
Yeah, then trends data may not be an accurate feature for it. It didn't give results for Indian stocks too. So yes, for universally traded and known stocks with larger trading base can use Google trends data
@skhonsha can't find you to add you on Slack. @Athe-kunal can you try adding skhonsha to the slack channel I just set up for discussing this topic further please.
thank you, I just tested trends, will try other situations. May I delete these irrelevant messages to this issue?
Of course, thanks
@skhonsha can't find you to add you on Slack. @Athe-kunal can you try adding skhonsha to the slack channel I just set up for discussing this topic further please.
@marcipops maybe something was wrong, Pls try again. I sent message to @Athe-kunal .
Also interested in tick-level data, but for market simulation rather than live trading. Wondering if anyone has checked out this repo: https://github.com/ChuaCheowHuan/gym-continuousDoubleAuction? Perhaps some of the code could translate well?
I've recently joined the discord but I could join the slack channel as well.
