Bad UX when `.resh_history.json` is corrupted

[NicoWeio]

I am going to write a chronological list of what I did, and summarize my suggestions for improvements at the bottom.

1. I got errors like these before and after every command

resh-collect ERROR: 2022/02/03 13:55:58 resh-daemon is not running - try restarting this terminal
resh-postcollect ERROR: 2022/02/03 13:55:58 resh-daemon is not running - try restarting this terminal

2. I restarted the terminal; the errors persisted.

3. I verified that no resh-daemon was running.

4. I tried invoking resh-daemon myself:

resh-collect ERROR: 2022/02/03 13:58:46 resh-daemon is not running - try restarting this terminal
2022/02/03 13:58:46 Daemon starting... 
version: 2.8.1 commit: 11eb86fc7a91323af135221e544d86a471b1a7e0
resh-postcollect ERROR: 2022/02/03 13:58:48 resh-daemon is not running - try restarting this terminal

It gave error code 1, but no further explanation. resh-daemon --help did not give a help and probably just ignored the flag. man resh-daemon (understandably) did not find an entry.

5. The related issues I found on resh's GitHub did not talk about file corruption or a fix.

6. I ran jq . .resh_history.json, which – after some time – yielded

parse error: Invalid numeric literal at line 23914, column 1381

7. I opened the file in a text editor and removed the offending line(s).

---

You can make of my "story" whatever you want, but here is what I'd suggest as improvements:

1. Since the error messages are rather annoying anyway, why not add further information about how to proceed if restarting the terminal doesn't work, e.g. deleting the history file, checking for updates, temporarily disabling resh or reporting an Issue on GitHub.

2. resh-daemon should tell me what's wrong, or at least give me an intuitive option to do so. In case there already are such options, I expect to find them using --help or man.

3. One might consider expanding the notes on troubleshooting in the README or a separate file/wiki.

---

Backstory: Log spam completely filled my SSD, which probably caused the corruption.

[curusarn]

Thanks for the thorough description of the issue and steps leading to it. Great suggestions.

Currently, there is just one simple self-healing mechanism for .resh_history.json implemented. Resh daemon will drop unparsable lines from .resh_history.json but this is limited to dropping just one line. If there is more than one unparsable line then daemon will fail.

This was added as a fix for situations when people hard reset their computers or something similar. Most of the times Resh appends to the file so only one line at the end of the file is corrupted. Silently dropping a single corrupted line solves that.

The limit of just one line is motivated the fact that we don't want to risk silently dropping a lot of lines.

I agree that the UX is bad when larger corruption happens to the history file.

[curusarn]

In v3 (#183) there will be better message when daemon is down:

resh-postcollect ERROR: RESH daemon didn't respond - it's probably not running.
 -> Start RESH daemon manually - run: resh-daemon-start
 -> Or restart this terminal window to bring RESH daemon back up
 -> You can check logs: ~/.local/share/resh/log.json (or ~/$XDG_DATA_HOME/resh/log.json)
 -> You can create an issue at: https://github.com/curusarn/resh/issues

This should point people into docs where they will find something like this:

{"level":"error","ts":1674126194.348005,"caller":"recio/read.go:88","msg":"Error while decoding line","executable":"daemon","daemonPID":17879,"module":"histfile","error":"failed to decode json: unexpected end of JSON input","filePath":"/Users/simon/.local/share/resh/history.reshjson","line":"v1{\"cmdLine\":\"nvim ~/.local/share/resh/history.reshjson\",\"exitCode\":0,\"deviceID\":\"\",\"sessionID\":\"a1775355-4c85-4737-98cd-53866f0272d5\",\"recordID\":\"84ead232-4017-4ce2-b421-719b89ba9fbc\",\"home\":\"/Users/simon\",\"pwd\":\"/Users/simon/git/resh\",\"realPwd\":\"/Users/simon/git/resh\",\"device\":\"\",\"gitOriginRemote\":\"[email protected]:curusarn/resh.git\",\n","stacktrace":"github.com/curusarn/resh/internal/recio.(*RecIO).ReadFile\n\t/Users/simon/git/resh/internal/recio/read.go:88\ngithub.com/curusarn/resh/internal/recio.(*RecIO).ReadAndFixFile\n\t/Users/simon/git/resh/internal/recio/read.go:18\ngithub.com/curusarn/resh/internal/histfile.(*Histfile).loadHistory\n\t/Users/simon/git/resh/internal/histfile/histfile.go:101"}
{"level":"error","ts":1674126194.3480241,"caller":"recio/read.go:88","msg":"Error while decoding line","executable":"daemon","daemonPID":17879,"module":"histfile","error":"failed to decode json: unexpected end of JSON input","filePath":"/Users/simon/.local/share/resh/history.reshjson","line":"v1{\"cmdLine\":\"nvim ~/.local/share/resh/history.reshjson\",\"exitCode\":0,\"deviceID\":\"\",\"sessionID\":\"a1775355-4c85-4737-98cd-53866f0272d5\",\"recordID\":\"84ead232-4017-4ce2-b421-719b89ba9fbc\",\"home\":\"/Users/simon\",\"pwd\":\"/Users/simon/git/resh\",\"realPwd\":\"/Users/simon/git/resh\",\"device\":\"\",\"gitOriginRemote\":\"[email protected]:curusarn/resh.git\",\n","stacktrace":"github.com/curusarn/resh/internal/recio.(*RecIO).ReadFile\n\t/Users/simon/git/resh/internal/recio/read.go:88\ngithub.com/curusarn/resh/internal/recio.(*RecIO).ReadAndFixFile\n\t/Users/simon/git/resh/internal/recio/read.go:18\ngithub.com/curusarn/resh/internal/histfile.(*Histfile).loadHistory\n\t/Users/simon/git/resh/internal/histfile/histfile.go:101"}
{"level":"error","ts":1674126194.348033,"caller":"recio/read.go:88","msg":"Error while decoding line","executable":"daemon","daemonPID":17879,"module":"histfile","error":"failed to decode json: unexpected end of JSON input","filePath":"/Users/simon/.local/share/resh/history.reshjson","line":"v1{\"cmdLine\":\"nvim ~/.local/share/resh/history.reshjson\",\"exitCode\":0,\"deviceID\":\"\",\"sessionID\":\"a1775355-4c85-4737-98cd-53866f0272d5\",\"recordID\":\"84ead232-4017-4ce2-b421-719b89ba9fbc\",\"home\":\"/Users/simon\",\"pwd\":\"/Users/simon/git/resh\",\"realPwd\":\"/Users/simon/git/resh\",\"device\":\"\",\"gitOriginRemote\":\"[email protected]:curusarn/resh.git\",\n","stacktrace":"github.com/curusarn/resh/internal/recio.(*RecIO).ReadFile\n\t/Users/simon/git/resh/internal/recio/read.go:88\ngithub.com/curusarn/resh/internal/recio.(*RecIO).ReadAndFixFile\n\t/Users/simon/git/resh/internal/recio/read.go:18\ngithub.com/curusarn/resh/internal/histfile.(*Histfile).loadHistory\n\t/Users/simon/git/resh/internal/histfile/histfile.go:101"}
{"level":"error","ts":1674126194.348056,"caller":"recio/read.go:24","msg":"Encountered too many decoding errors","executable":"daemon","daemonPID":17879,"module":"histfile","corruptedRecords":6,"individualErrors":"Search 'Error while decoding line' to see individual errors","stacktrace":"github.com/curusarn/resh/internal/recio.(*RecIO).ReadAndFixFile\n\t/Users/simon/git/resh/internal/recio/read.go:24\ngithub.com/curusarn/resh/internal/histfile.(*Histfile).loadHistory\n\t/Users/simon/git/resh/internal/histfile/histfile.go:101"}
{"level":"fatal","ts":1674126194.3480692,"caller":"histfile/histfile.go:103","msg":"Failed to read file: encountered too many decoding errors, last error: failed to decode json: unexpected end of JSON input","executable":"daemon","daemonPID":17879,"module":"histfile","stacktrace":"github.com/curusarn/resh/internal/histfile.(*Histfile).loadHistory\n\t/Users/simon/git/resh/internal/histfile/histfile.go:103"}

[curusarn]

In v3 I'm also adding a help message to resh-daemon and a Troubleshooting page to github.