question: portfolio optimization

Hi,

It is not a feature request and more like a question. I check all the docs and try to find examples, but still not sure about. My usecase is simple to formulate: I have a portfolio with some weights (for example [0.25, 0.5, 0.25, 0.0]) and on each step I have new prediction of the weights and thus need to sell/buy several pairs to rebalance according to prediction. Previously I did this manually calculating limit prices and tradable amount and it works more or less correctly. And I try to understand if it possible to develop such approach here, because Octobot looks superuseful and web-interface is especially fun to work with.

If I can read a specific code part - link is appreciated. Thanks!

Hi, Thanks for you interest ! You can use OctoBot for that having evaluators to compute the weights and then make OctoBot balance your portfolio for you however this will require a bit of code since it is not integrated in the bot just yet. We are at the final phase of a major refactor and therefore I suggest you have a look at the 0.4 branch of this repo. You can see examples of evaluators here https://github.com/Drakkar-Software/OctoBot-Tentacles/blob/0.4.0/Evaluator/TA/momentum_evaluator/momentum_evaluator.py

Then the idea would be to create a trading mode that creates trades using the current portfolio balance. Example of a trading mode: https://github.com/Drakkar-Software/OctoBot-Tentacles/blob/0.4.0/Trading/Mode/daily_trading_mode/daily_trading_mode.py

GitHub

Drakkar-Software/OctoBot-Tentacles

This repository contains public tentacles (packages) for the OctoBot project - Drakkar-Software/OctoBot-Tentacles

GitHub

Drakkar-Software/OctoBot-Tentacles

This repository contains public tentacles (packages) for the OctoBot project - Drakkar-Software/OctoBot-Tentacles

Thanks @GuillaumeDSM for these examples! I start to investigate the possibilities to implement my strategy using 0.4.0 as a reference and will return with more questions.

One particular question: will it be possible to backtest my portfolio optimization using Octobot web UI or other options? I play with other tentacles recently downloading data for coins and backtest already implemented examples, but seems like this is suitable for one symbol at a time only.

Thanks @GuillaumeDSM for these examples! I start to investigate the possibilities to implement my strategy using 0.4.0 as a reference and will return with more questions.

One particular question: will it be possible to backtest my portfolio optimization using Octobot web UI or other options? I play with other tentacles recently downloading data for coins and backtest already implemented examples, but seems like this is suitable for one symbol at a time only.

Hi @Amarchuk, you mean backtesting on multiple symbols at the same time to simulate your full portfolio optimization ?

Hi @Herklos, yes, exactly as you formulate

Hi @Herklos, yes, exactly as you formulate

Backtesting on multiple symbol is not yet available but it's in our 0.4.0 roadmap.

It should be available very soon, probably this week.

Hi @GuillaumeDSM - can you please give an update on the status of Backtesting on multiple symbol?

Thanks

Hi @magick93, It's working in the latest versions of 0.4 !

Hi @GuillaumeDSM, is it correct that it is not yet available in the Web UI? And what does this actually mean - we can run same evaluator on just multiple symbols at once, or evaluator/strategy can now decide using prices from multiple symbols?

It's available on 0.4 from the web interface, you might need to re-install your tentacles to get the update.

Strategies and evaluators can chose to be evaluating every symbols or only one, same for strategies and trading modes, it's the purpose of this method:

@classmethod
def get_is_symbol_wildcard(cls) -> bool:

Ex: default is True for evaluators meaning that an evaluator will be waken up multiple times, once for each symbol. If it's returning False an instance of this evaluator will be created for each symbol and will only be waken up for its own symbol update

Thanks, good to know about - I can't find it by myself.

Hm, interesting - I try to pull 0.4.0 and different ways to install it. Which way you suggest to update to 0.4 and test these new features? Will latest launcher works with new code as well (I try it, but ui shows v0.3.9 and I actually not test if it contains new features or not)?

I suggest you have a look at this page of the wiki for how to install octobot 0.4.

The current 0.3 launcher is not displaying OctoBot 0.4 because 0.4 is in alpha tests for now, no public version has been released yet. You will find more info about 0.4 on this page if you are curious about it.

Hm, I did not touch docker yet, but experienced second time the same problem with python install (the first one was yesterday when I ask previous comment):

> python3.7 -m  pip install --prefer-binary -r requirements.txt

Collecting cython==0.29.17 (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/dc/6c/198da24bf0a1d5b5ee6119e891c5300bb1b0b8904832fac9ff62d4e381c9/Cython-0.29.17-cp37-cp37m-manylinux1_x86_64.whl
Collecting OctoBot-Commons<1.4,>=1.3.31 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/1f/6f/7a0e372063aa47b9e8d8a97ef43d8036dde54566405348c55412a48f2c40/OctoBot-Commons-1.3.33.tar.gz (556kB)
     |████████████████████████████████| 563kB 715kB/s 
Collecting OctoBot-Trading<1.7,>=1.6.20 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/c7/f4/400fbb6c1068dfe3943b4d339ab65f8983096974b30947977143f1353634/OctoBot-Trading-1.6.22.tar.gz (6.6MB)
     |████████████████████████████████| 6.7MB 6.9MB/s 

And after that last line nothing happens (I wait around 20 minutes or so). This sounds like a bug for separated issue

Hm, I did not touch docker yet, but experienced second time the same problem with python install (the first one was yesterday when I ask previous comment):

> python3.7 -m  pip install --prefer-binary -r requirements.txt

Collecting cython==0.29.17 (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/dc/6c/198da24bf0a1d5b5ee6119e891c5300bb1b0b8904832fac9ff62d4e381c9/Cython-0.29.17-cp37-cp37m-manylinux1_x86_64.whl
Collecting OctoBot-Commons<1.4,>=1.3.31 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/1f/6f/7a0e372063aa47b9e8d8a97ef43d8036dde54566405348c55412a48f2c40/OctoBot-Commons-1.3.33.tar.gz (556kB)
     |████████████████████████████████| 563kB 715kB/s 
Collecting OctoBot-Trading<1.7,>=1.6.20 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/c7/f4/400fbb6c1068dfe3943b4d339ab65f8983096974b30947977143f1353634/OctoBot-Trading-1.6.22.tar.gz (6.6MB)
     |████████████████████████████████| 6.7MB 6.9MB/s 

And after that last line nothing happens (I wait around 20 minutes or so). This sounds like a bug for separated issue

Hi @Amarchuk, if you're not using the current supported architecture and python version python will have to recompile some wheels (it can takes some hours depending on your computer CPU).

• The current supported python version is 3.7
• Current architectures are :
  • Linux : i386 and X86_64
  • Windows : 32bits and 64bits
  • MacOs : Not officially supported
  • Arm : None

@GuillaumeDSM Thanks for fast reply! I think that I actually use a supported architecture, but will give it another longer try anyway.

Hi there,

First, another thanks for the great project! I can finally successfully install 0.4.0 using fresh virtualenv and also implement my own tentacle using some tricky way for previous version. Now I have several independent questions:

1. I start octobot 0.3.9 using launcher and need some additional python packages in my tentacle, but can't find proper python version where to install them. I add a print in log and see that octobot starts with python 3.7.2, but I can't locate it on my machine. Can we control somehow the python version when start it using launcher? (note: bash python3 command will run 3.6.6, python3.7 command - 3.7.4)

2. How long can backtesting takes? It runs already for an hour for default 'DailyTradingMode' for one coin. There is Backtesting in progress sign, but the progressbar didn't move. Also is it expected behaviour that backtesting stops when you leave the /backtesting url?

3. In 0.4.0 if I need to backtest my tentacle for multiple currencies at once - what should I change? Can you point to some functions to pay attention and starts from? (for example I am interested if self.exchange.get_symbol_prices(symbol, TimeFrames.THIRTY_MINUTES, limit=None, return_list=True) will still work for obtaining previous prices)

4. About my own tentacle for 0.3.9: the code is tricky and kinda awkward because I need to manage portfolio as whole (so symbols are not independent from each other, because I need to sell some first and only then buy for other) and came so the solution when in my implementation of AbstractTradingModeCreator.create_new_order I create new orders not only for symbol parameter, but for other symbols as well. It is messy and I am not sure why it works, but seems like I get what I want. The question here is that for some symbols I can create all orders I want and they are executed correctly (so for symbol=ZEC/BTC I also add in created_orders orders for all other coins and return this list in create_new_order(ZEC/BTC, ...), and sometimes some of them (for other symbols) are silently cancelled. I am 100% sure that the orders are correct and can be executed. I know that it is difficult to understand what is going on here, but maybe you have a clue why this is happening.

Thanks

Addition to 2: I notice in Chrome console this one, maybe it is related

Uncaught ReferenceError: get_websocket is not defined
    at init_backtesting_status_websocket (backtesting_util.js:151)
    at HTMLDocument.<anonymous> (backtesting.js:90)
    at e (jquery-3.4.0.min.js:2)
    at t (jquery-3.4.0.min.js:2)

Hi there,

First, another thanks for the great project! I can finally successfully install 0.4.0 using fresh virtualenv and also implement my own tentacle using some tricky way for previous version. Now I have several independent questions:

1. I start octobot 0.3.9 using launcher and need some additional python packages in my tentacle, but can't find proper python version where to install them. I add a print in log and see that octobot starts with python 3.7.2, but I can't locate it on my machine. Can we control somehow the python version when start it using launcher? (note: bash python3 command will run 3.6.6, python3.7 command - 3.7.4)

To control the python version, the best is to use the code directly or a docker version. You can probably force a python version to run with the launcher by adding the python you want on the front of you PATH environment variable though.

2. How long can backtesting takes? It runs already for an hour for default 'DailyTradingMode' for one coin. There is Backtesting in progress sign, but the progressbar didn't move. Also is it expected behaviour that backtesting stops when you leave the /backtesting url?

If you get this error "Uncaught ReferenceError: get_websocket is not defined" it very likely means that your browsers is using js files from 0.3 as well as 0.4, you can use ctrl+f5 or shift+f5 (depending on your browser) to reset the local cache, it should fix this problem. Backtesting time should be a few seconds.

3. In 0.4.0 if I need to backtest my tentacle for multiple currencies at once - what should I change? Can you point to some functions to pay attention and starts from? (for example I am interested if self.exchange.get_symbol_prices(symbol, TimeFrames.THIRTY_MINUTES, limit=None, return_list=True) will still work for obtaining previous prices)

You just have to select select multiple data files with different currencies. Then if your evaluator get_is_symbol_wildcard(cls) returning True (default value), it will be called for each symbol. If returning false, an specific instance of this evaluator will be handling each symbol. Data will automatically come from ohlcv_callback, like on live mode.

4. About my own tentacle for 0.3.9: the code is tricky and kinda awkward because I need to manage portfolio as whole (so symbols are not independent from each other, because I need to sell some first and only then buy for other) and came so the solution when in my implementation of AbstractTradingModeCreator.create_new_order I create new orders not only for symbol parameter, but for other symbols as well. It is messy and I am not sure why it works, but seems like I get what I want. The question here is that for some symbols I can create all orders I want and they are executed correctly (so for symbol=ZEC/BTC I also add in created_orders orders for all other coins and return this list in create_new_order(ZEC/BTC, ...), and sometimes some of them (for other symbols) are silently cancelled. I am 100% sure that the orders are correct and can be executed. I know that it is difficult to understand what is going on here, but maybe you have a clue why this is happening.

If you are using the DailyTradingMode as a super class, it has await self.cancel_symbol_open_orders(symbol) in async def _set_state that can cancel orders. It might be the reason. If you want to know for sure from where a cancel is triggered, you can either add a debug breakpoint on each trader.py cancel methods or raise an exception to print the current traceback when a cancel is happening (add it in trader.py cancel methods):

Thanks

Thanks so much @GuillaumeDSM! I don't be able to take a look at everything yet, but this is definitely helps (and ctrl+f5 works, can backtest changes now).

Hi!

Another minor question (or probably bug report?) related to this issue (still for 0.3.9). I try to fill my portfolio with needed coins in Trading tab, but, for example, there is no way to add Etherium (ETH) from dropdown list (no such option). But I can go to user/config and add it manually there as one of the coins and it will appears in OctoBot after start. Is it ok?

Hi, Yes, to add currencies, add them directly into your user/config.json (web interface is not yet handling currency addition)

@GuillaumeDSM Thanks again for all clarifications! I have one minor new question: it seems like in 0.4.0 all Advanced folders were removed and when I try to create them and move files here anyway - I get a bunch of errors about missed metadata.json. So it seems like the way I test my code before is not applicable anymore. I try to get some insight about this from changelog and reading code, but only found some web-related changes. Can you please clarify this as well or suggest the proper way to test my tentacle in 0.4?

In 0.4 there is no advanced / default folders anymore, everything is bundled together in the related tentacle type, to integrate your code, you can just copy/paste an existing tentacle folder (like volatility_evaluatorfor ex) and replace everything (in every sub-file/folder) related to StochasticRSIVolatilityEvaluator and the names of the files with your tentacle

@GuillaumeDSM I have now two more:

1. So I continue trying to start using 0.4 and this is tough since there are so many changes. I finally replace almost every part and fill all configs, but actually can't turn on my strategy because I keep get errors about required_time_frames property missed, and I really don't know what to do about it because it is included in config files

 2020-06-25 22:31:38,849 INFO     OctoBot Launcher     Disclaimer accepted by user.
 2020-06-25 22:31:38,873 DEBUG    EvaluatorProducer    Pausing...
--- Logging error ---
Traceback (most recent call last):
  File "octobot4.0/OctoBot/octobot/commands.py", line 110, in start_bot
    await bot.initialize()
  File "octobot4.0/OctoBot/octobot/octobot.py", line 103, in initialize
    await self.start_producers()
  File "octobot4.0/OctoBot/octobot/octobot.py", line 113, in start_producers
    await self.evaluator_producer.run()
  File "octobot4.0/OctoBot/octobot/channels/octobot_channel.py", line 40, in run
    await self.start()
  File "octobot4.0/OctoBot/octobot/producers/evaluator_producer.py", line 39, in start
    self.matrix_id = await initialize_evaluators(self.octobot.config, self.tentacles_setup_config)
  File "p37/lib/python3.7/site-packages/octobot_evaluators/api/evaluators.py", line 167, in initialize_evaluators
    _init_time_frames(config, tentacles_setup_config)
  File "p37/lib/python3.7/site-packages/octobot_evaluators/api/evaluators.py", line 178, in _init_time_frames
    init_time_frames_from_strategies(config, tentacles_setup_config)
  File "p37/lib/python3.7/site-packages/octobot_evaluators/api/initialization.py", line 29, in init_time_frames_from_strategies
    for time_frame in strategies_eval_class.get_required_time_frames(config):
  File "p37/lib/python3.7/site-packages/octobot_evaluators/evaluator/strategy_evaluator.py", line 387, in get_required_time_frames
    raise Exception(f"'{STRATEGIES_REQUIRED_TIME_FRAME}' is missing in configuration file")
Exception: 'required_time_frames' is missing in configuration file

2. Ok, I turn off my strategy and use default one instead, and webUI can now start, but there is no trading and if I try to go into Trading section I get this one:

  File "p37/lib/python3.7/site-packages/octobot_tentacles_manager/loaders/tentacle_loading.py", line 55, in get_tentacle
    return _tentacle_by_tentacle_class[klass.get_name()]
KeyError: 'PGPDailyTradingMode'

It feels like I can't somehow register tentacle/mode, but really can't find out why because fill any config/init which found.

Hi @Amarchuk ! We will soon make a guide on how to create a 0.4 tentacle since it's different from 0.3 as you noticed. About 1.

Tentacles config are stored in your user/tentacles_config/specific_config folder. When installing a tentacle, its config is copied from the tentacle folder (config sub-dir) to there. json files in your tentacle/config folder are default values for your config that are re-applied when factory resetting the tentacle config (from web interface for example). Your issue is probably that you are editing the config file in your tentacle/config folder, which is not the one your bot will be using when starting (since it's the default values one). You can copy-paste it into your user/tentacles_config/specific_config folder and replace the previous version, that should fix your issue.

About 2.

The tentacle loading process is using the metadata.json file to register tentacles: it's registering tentacles according to their class name such as in this (from web_interface/metadata.json):

{
  "version": "1.2.0",
  "origin_package": "OctoBot-Default-Tentacles",
  "tentacles": ["WebInterface"],
  "tentacles-requirements": ["web_service"]
}

Here the data about the WebInterface tentacle class (which is located into the web_interface folder) will be stored under the WebInterface key into _tentacle_by_tentacle_class because "tentacles": ["WebInterface"] is in metadata.json. Then when OctoBot will require data about this WebInterface thing (for example to display details about it to the user, such as its version or any other thing), OctoBot will use this _tentacle_by_tentacle_class using the name of the tentacle class it is trying to get data from. Therefore in your tentacle metadata.json file, you need to have this syntax "tentacles": ["WebInterface"], with the tentacles class(es) that are located into your tentacle folder into the tentacles key list for them to be registered and available for later uses in OctoBot.
I hope that fixes your issue, this is the most likely reason you are having this 2. issue.

About 2, I'm updating the code to first not interrupt the page loading and the display an easier to understand error (in https://github.com/Drakkar-Software/OctoBot-Tentacles/pull/271)

Hi @GuillaumeDSM !

We will soon make a guide on how to create a 0.4 tentacle since it's different from 0.3 as you noticed.

Very good news!

It seems like for some reason configs were not copied into user/tentacles_config/specific_config and manual copying solves my first problem, thanks.

About second: I really appreciate the quality and fasteness of your answers, but unfortunately it is really difficult to understand this WebInterface example, because it is unusual one. I double-check all folders and still it is seems like I have metadata.json in every place I need about. So I have evaluator, strategy and mode, and each metadata looks like this:

{
  "version": "1.1.6",
  "origin_package": "OctoBot-Default-Tentacles",
  "tentacles": ["PGPTradingMode"],
  "tentacles-requirements": []
}

and the name for "tentacles" is the same as valuator/strategy/mode so they all are different. Note that in __init__.py I don't use check_tentacle_version('1.2.0',... and just import everything.

Maybe I miss something eventually and besides all tentacles_config.json I also need to additionally register tentacles somehow?

Your trading mode, eval and strategy classes have to be listed in tentacles_config.json for OctoBot to handle them properly yes. Updating tentacles_config.json, the __init__.py files (both inside the tentacle if missing and in the parent folder) and importing config are normally handled automatically by the tentacle manager (we also need to write a wiki article about this). To sum up, the required elements for a 0.4 tentacle are:

1. the tentacle folder in the right place in the tentacle architecture (next to similar tentacles)
2. in this folder:
   • a .py file with the code of your tentacle
   • importing the tentacle class of your .py file in __init__.py
   • listing this (or these) tentacle class(es) in metadata.json under "tentacles":
3. import * from this folder in the __init__.py from the parent folder (for python to see your tentacle)
4. add your tentacle class(es) into user/tentacles_config/tentacles_config.json (True for activation, False for not activation, please note that only one trading mode can be activated at a time)
5. copy your tentacle condig into user/tentacles_config/specific_config if any json config file

If you did all this and it still doesn't work, then I will need more info (startup errors / warnings / etc)

Hi @GuillaumeDSM,

Thanks again for the last answer - I re-check everything and finally was able to import and run my tentacle (indeed there was an error in one of json). After several hours of fixing compatibility issues I successfully run it and everything works as expected thus I switch my attention to the purpose doing it on the first place - backtesting. I try to collect new data for it, but get an error "Can't collect data for ETC/BNB on binance (data collector error: Please ensure your tentacles configuration file is valid and at least one trading mode is activated)". Unfortunately I can't backtrack the source of the error, can you please generously help me with this?

P.S. Workaround: I can switch to 0.3.9, download OHLC data from it and then import them using Import from data file, but of course it is not that useful.

Hi @Amarchuk !

There is an issue with the current version regarding data collection for exchange, you can fix it this way: Replace the content of your tentacles/Backtesting/collectors/exchanges/exchange_history_collector/exchange_history_collector.py with the content of this version (updated one): https://github.com/Drakkar-Software/OctoBot-Tentacles/blob/0.4.0/Backtesting/collectors/exchanges/exchange_history_collector/exchange_history_collector.py

You can import 0.3.9 data files into 0.4 one using the local file selector in data collector (this will transform 0.3.9 data into 0.4)

GitHub

Drakkar-Software/OctoBot-Tentacles

This repository contains public tentacles (packages) for the OctoBot project - Drakkar-Software/OctoBot-Tentacles