Really slow startup with large history file, MCFLY_HISTORY_LIMIT doesn't help

[dflock]

I have this in my .bashrc:

export MCFLY_HISTORY_LIMIT=10000
eval "$(mcfly init bash)"

Doing . ~/.bashrc takes ~forever. I have no idea how long, because I did ctrl+c after a minute or so. This is a new install. I've tried changing MCFLY_HISTORY_LIMIT=1000, doing rm ~/.mcfly/history.db, etc... still takes ages/doesn't complete. Lots of disk i/o too.

I assume this is because I have a large history file. These are my current bash history related settings:

#
# History settings
#
shopt -s histappend # Append to the history file, don't overwrite it
shopt -s cmdhist # Save multi-line commands as one command
export HISTSIZE=-1 # Keeps all history
export HISTFILESIZE=-1 # Infinite file size
export HISTTIMEFORMAT='%F %T ' # Useful timestamp format
export HISTIGNORE="&:[ ]*:exit:clear"
export HISTFILE=~/.bash_eternal_history
# Record each line as it gets issued
[[ "$PROMPT_COMMAND" == *'history -a'* ]] || export PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

My ~/.bash_eternal_history file is currently 497.5 KiB, 26,339 lines. It also includes timestamps, so the last bit of it looks like this:

#1615575032
. ~/.bashrc
#1615575046
rm ~/.mcfly/history.db
#1615575049
micro ~/bash/mcfly.bash
#1615575057
micro ~/.bashrc
#1615575459
micro ~/bash/env.bash

If I do this, mcfly starts working:

$ mv ~/.bash_eternal_history ~/_.bash_eternal_history
$ rm ~/.mcfly/history.db
$ . ~/.bashrc

[cantino]

Thank you for your report! Did you see any message like McFly: Importing shell history for the first time. This may take a minute or two... in your shell?

[cantino]

I just pushed up a change, would you mind checking if that helped?

[dflock]

No, didn't get any messages, just sat there grinding the disk and not returning to the prompt.

[dflock]

OK, giving it a try. I got the McFly: Importing shell history for the first time. This may take a minute or two... message this time

I'll let it run and tell you how long it takes :)

[cantino]

I hope it's not more than a couple minutes.

[dflock]

Well... it took from 22:29:55 until 23:22:12, almost an hour.

$ ll ~/.mcfly/history.db

Permissions Size User   Group  Date Modified Name
.rw-r--r--  1.7M duncan duncan 12 Mar 23:22  /home/duncan/.mcfly/history.db 

Not played with it much yet, but it seems to work OK now.

[cantino]

I'm glad it finished, but am surprised it took so long! What kind of computer do you have?

[dflock]

OS: Xubuntu 20.04.2 LTS x86_64 
Kernel: 5.4.0-66-generic 
Uptime: 17 days, 17 hours, 3 mins 
Packages: 68690 (apt), 53 (flatpak), 1 (snap) 
Shell: bash 5.0.17 
Resolution: 1920x1080, 1920x1080, 1920x1080 
DE: Xfce 4.16 
WM: Xfwm4 
WM Theme: Arc-Dark 
Theme: Arc-Dark [GTK2/3] 
Icons: Papirus-Dark [GTK2/3] 
Terminal: kitty 
Terminal Font: Operator Mono SSm Lig Book 
CPU: Intel i7-6700K (8) @ 4.200GHz 
GPU: Intel HD Graphics 530 
Memory: 38005MiB / 64206MiB 

[dflock]

I know it's a fairly large history file, but this workstation should be able to process 26k lines of text in a few seconds, at the very most.

[cantino]

Agreed, I think this import process needs to be profiled and optimized.

[cantino]

Also, MCFLY_HISTORY_LIMIT doesn't currently affect import, so it makes sense that it didn't help. Should it?

[dflock]

Also, MCFLY_HISTORY_LIMIT doesn't currently affect import, so it makes sense that it didn't help. Should it?

It shouldn't be necessary at all, really.

[cantino]

Agreed. This should be optimized.

[SafariMonkey]

I had the same issue and adding a transaction around the insert (#146) fixed it for me. Maybe this can be considered fixed? @dflock can you test with the latest master to see if it's improved?

[michaeljoseph]

I installed today and the initial import took 12s on a 13 Mb history file.

🕙 2021-08-02 11:10:01  ❯ source ~/.bashrc 
McFly: Importing shell history for the first time. This may take a minute or two...done.

✦2 🕙 2021-08-02 11:10:13  ❯  ls -al ~/.mcfly/history.db ~/.dotfiles/bash/.bash_eternal_history
.rw-r--r-- michaeljoseph staff 13 MB Mon Aug  2 11:15:34 2021  /Users/michaeljoseph/.dotfiles/bash/.bash_eternal_history
.rw-r--r-- michaeljoseph staff 44 MB Mon Aug  2 11:15:34 2021  /Users/michaeljoseph/.mcfly/history.db

[Bad3r]

I noticed that running any command in my shell became painfully slow. After some testing, realized that moving the history.db file fixed the issue. 

the db was 13M 

eval "$(mcfly init zsh)"

export MCFLY_KEY_SCHEME=vim

export MCFLY_FUZZY=2

export MCFLY_RESULTS=2048

export MCFLY_HISTORY_LIMIT=10000

export MCFLY_INTERFACE_VIEW=BOTTOM

export MCFLY_RESULTS_SORT=LAST_RUN  

[cantino]

My DB is 18M and I haven't had any issues. What kind of computer do you have? It might be nice for us to add execution time logging to the shell scripts.