hummingbot icon indicating copy to clipboard operation
hummingbot copied to clipboard

Implement file based persistent command history.

Open WarrenFerrell opened this issue 9 months ago • 8 comments

Before submitting this PR, please make sure:

  • [x] Your code builds clean without any errors or warnings
  • [x] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

Added persistent command history to the Hummingbot Console with configurable command filtering. The changes include:

  1. Added configuration options in client_config_map.py:

    • command_history_file: Enable/disable file-based history (default: False)
    • command_history_file_path: History file location (default: ~/.hummingbot/cmd_history/command_history.txt)
    • command_history_exclusion_list: List of command prefixes to exclude (default: ["exit"])
  2. Modified layout.py to use FileHistory from prompt_toolkit for persistent command storage and added command filtering through a custom FilteredFileHistory class.

  3. Updated hummingbot_cli.py to pass client configuration to input field creation.

Tests performed by the developer:

  • Started the application and verified command history is retained between sessions
  • Confirmed that the "exit" command is not stored in command history
  • Verified that other commands are properly stored and can be recalled
  • Verified that setting command_history_file to false restores previous behavior

Tips for QA testing:

  1. Start Hummingbot and enter some commands
  2. Exit and restart Hummingbot
  3. Use up/down arrows to verify:
    • Previous commands are available
    • "exit" command is not in history
    • History file exists at ~/.hummingbot/cmd_history/command_history.txt

WarrenFerrell avatar Mar 21 '25 17:03 WarrenFerrell

~don't merge yet. broke a test.~ Never mind, ran them again and they're fine. Annoying because they reset conf_client but fine. It must have had something to do with my other PR.

WarrenFerrell avatar Mar 21 '25 22:03 WarrenFerrell

thanks for your contribution @WarrenFerrell ! QA will start testing this feature one thing that might be a concern is what happens if the user runs status --live, is the output stored every 0.3 secs, if so we should remove those live updates from this PR. for the config is good but what you can also do is to create config object for this specific thing instead of 3 new args, like for example the config for MQTT. btw, I like the idea of having this command --> text and you can use the broker to send this messages periodically if you want.

cardosofede avatar Apr 01 '25 14:04 cardosofede

@cardosofede This is what the command_history.txt file looks like. It only stores the input command and only when you use a command was not already the most recent command (I tested by using balance about 10 times in a row. I haven't looked into anything about prompt_toolkit customization except for excluding exit from history (since it was super annoying to accidentally press up after reopening the app and have it immediately close. We could include stop, status and/or balance as one of the default words to exclude as well as that would greatly reduce the number of entries that actually get written but since the config is off by default I don't think that's necessary. Went ahead and changed to a config Map.

# 2025-03-29 18:18:36.454425
+stop

# 2025-03-29 18:18:41.079173
+balance

# 2025-04-01 12:33:13.293644
+stop

WarrenFerrell avatar Apr 01 '25 18:04 WarrenFerrell

Test 35b491d897428bf57fc2327c4d62213331f3b7a7

  • Run tests on linux (ubuntu20.04) using source build
  • New commands command_history image- Set command_history.use_history_file to true ✅
    • Changed default path from ~/.hummingbot/cmd_history/command_history.txt to ~/hummingbot/cmd_history/command_history.txt
    • When set to false, does not fill any of stored history commands ✅
      • Then set to true, managed to pull all history commands when using up / down keys ✅
  • Observed that it does not create .txt file if we add new ❗ image
    • manually created test.txt and add a+rw permission ✅
    • loaded and worked ✅
  • Confirmed all config changes are stored on command_history.txt image
    • Relaunch hummingbot after changes
      • Confirmed that all history configs are stored and not exit (default) ✅
      • Run few more commands like balance paper ✅

Test command_history_execlusion_list: does not accept anything we want to add ❌

  • does not accept if command has space in between like gateway generate-certs
  • tried to add different types of variation separated by comma and getting same results as above. Observed that it does not consider exit as well ❌ image

Docker build

  • Enabled and confirmed updated on conf_client.yml
  • Does not register any of the commands on history ❌
    • Does not seem to recognize the path we set from history_file_path

rapcmia avatar Apr 23 '25 15:04 rapcmia

Moved the default path to conf/command_history.txt . I didn't test the docker build but I'm 90% sure that the issue here was having the user path (~) in the docker container. Fixed the issue with providing a list to exclusion list, it was missing a validator for splitting strings (I thought pydantic took care of this itself).

WarrenFerrell avatar Jun 10 '25 17:06 WarrenFerrell

Failing tests for xrpl appear to be failing in some other PRs as well with unrelated changes.

WarrenFerrell avatar Jun 10 '25 17:06 WarrenFerrell

commit 6a6a293

Steps to reproduce:

  1. set to # Command History configuration. command_history: use_history_file: true command_history_file_path: default conf command_history_exclusion_list:
  • exit
  1. use connect binance, connect kucoin, config rate oracle source commands
  2. exit
  3. start the client again
  4. review the issue below
12:38:20 - trading_pair_fetcher - An error occurred when fetching trading pairs for binance.Please check the logs (See log file for stack trace dump)
12:38:20 - trading_pair_fetcher - An error occurred when fetching trading pairs for binance_paper_trade.Please check the logs (See log file for stack trace dump)
12:38:20 - trading_pair_fetcher - An error occurred when fetching trading pairs for kucoin_paper_trade.Please check the logs (See log file for stack trace dump)
12:38:20 - trading_pair_fetcher - An error occurred when fetching trading pairs for kraken_paper_trade.Please check the logs (See log file for stack trace dump)
12:38:20 - trading_pair_fetcher - An error occurred when fetching trading pairs for gate_io_paper_trade.Please check the logs (See log file for stack trace dump)
Traceback (most recent call last):
  File "/home/hummingbot/nikita/hist7512/./bin/hummingbot_quickstart.py", line 172, in <module>
    main()
  File "/home/hummingbot/nikita/hist7512/./bin/hummingbot_quickstart.py", line 168, in main
    ev_loop.run_until_complete(quick_start(args, secrets_manager))
  File "/home/hummingbot/miniconda3/envs/hummingbot/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/./bin/hummingbot_quickstart.py", line 96, in quick_start
    hb = HummingbotApplication.main_application(client_config_map=client_config_map)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/hummingbot_application.py", line 68, in main_application
    cls._main_app = HummingbotApplication(client_config_map)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/hummingbot_application.py", line 117, in __init__
    self.app = HummingbotCLI(
               ^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/ui/hummingbot_cli.py", line 63, in __init__
    self.input_field = create_input_field(completer=completer, client_config_map=client_config_map)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/ui/layout.py", line 96, in create_input_field
    history=_configure_history(client_config_map),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/ui/layout.py", line 295, in _configure_history
    os.makedirs(history_dir, exist_ok=True)
  File "<frozen os>", line 225, in makedirs
FileNotFoundError: [Errno 2] No such file or directory: ''

image

nikspz avatar Jun 13 '25 12:06 nikspz

I added a min_length constraint to prevent setting of the path to '' . The makes it easier to set the default as pressing enter creates an invalid entry. Exclusion list needs to be able to accept an empty list so it can't have that constraint. Also specified that if using the top directory you need to use ./ . Overall not actually a bug as it's invalid config values that cause that. And The same lack of documentation occurs with log_file_path config.

WarrenFerrell avatar Jun 13 '25 16:06 WarrenFerrell

commit fcb13cb06c1ba1d7facbaf2b8fea1ff29357f343

Steps to reproduce:

  1. set to # Command History configuration.
  2. command_history:
  3. use_history_file: true
  4. command_history_file_path: default conf
  5. command_history_exclusion_list:
  6. exit
  7. use connect binance, connect kucoin, config rate oracle source commands
  8. exit
  9. start the client again with ./start -p or with bin/hummingbot.py
  10. review the issue below
11:29:30 - trading_pair_fetcher - An error occurred when fetching trading pairs for binance_paper_trade.Please check the logs (See log file for stack trace dump)
11:29:31 - trading_pair_fetcher - An error occurred when fetching trading pairs for kucoin_paper_trade.Please check the logs (See log file for stack trace dump)
11:29:31 - trading_pair_fetcher - An error occurred when fetching trading pairs for kraken_paper_trade.Please check the logs (See log file for stack trace dump)
11:29:31 - trading_pair_fetcher - An error occurred when fetching trading pairs for gate_io_paper_trade.Please check the logs (See log file for stack trace dump)
Traceback (most recent call last):
  File "/home/hummingbot/nikita/hist7512/./bin/hummingbot_quickstart.py", line 172, in <module>
    main()
  File "/home/hummingbot/nikita/hist7512/./bin/hummingbot_quickstart.py", line 168, in main
    ev_loop.run_until_complete(quick_start(args, secrets_manager))
  File "/home/hummingbot/miniconda3/envs/hummingbot/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/./bin/hummingbot_quickstart.py", line 96, in quick_start
    hb = HummingbotApplication.main_application(client_config_map=client_config_map)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/hummingbot_application.py", line 68, in main_application
    cls._main_app = HummingbotApplication(client_config_map)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/hummingbot_application.py", line 117, in __init__
    self.app = HummingbotCLI(
               ^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/ui/hummingbot_cli.py", line 63, in __init__
    self.input_field = create_input_field(completer=completer, client_config_map=client_config_map)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/ui/layout.py", line 96, in create_input_field
    history=_configure_history(client_config_map),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hummingbot/nikita/hist7512/hummingbot/client/ui/layout.py", line 295, in _configure_history
    os.makedirs(history_dir, exist_ok=True)
  File "<frozen os>", line 225, in makedirs
FileNotFoundError: [Errno 2] No such file or directory: ''

image

nikspz avatar Jun 20 '25 11:06 nikspz

@nikspz Sorry I thought I had commented on this. That is expected behavior. "default conf" is not a valid filepath. That said, I added an error message to clarify what went wrong

WarrenFerrell avatar Sep 04 '25 21:09 WarrenFerrell

As discussed with the team - this PR is archiedved, not going to implement this one

nikspz avatar Sep 11 '25 13:09 nikspz