Timestamp support + lots of other changes

[eliask]

• code is now ok according to shellcheck
• various simplifications and enhancements to code
• hist search now searches for multiple terms with AND clauses.

[jcsalterego]

@eliask cool! A few preliminary notes:

• I generally like to break commits up instead of one fell swoop. This makes reverts easier, should we need them.
• I'm not 100% committed to shellcheck just yet; I haven't had a chance to install it locally and play around with it. If I do commit to it, I'll want to add it to the CI and maybe make mention of it in the README for contributions.
• The search functionality change is interesting, but I think I have to think through that just a little bit before changing the behavior. For example, I could see using regex (if sqlite supports it) as useful (in line with grep -E).
• I'd rather keep version bumps in master instead of feature branches.

I'll take a closer look when I can but there are some good ideas here!

[eliask]

I could break this up later if I can find the time and motivation.

One more thing in general that I didn't investigate is the timestamps from bash_history. Maybe it's configurable but for me I have all lines prefixed with their (local, UTC, who knows) timestamps like so:

$ tail -n4 .bash_history #1495730784 hist #1495730787 cat ~/.bashrc

Maybe historian could have a bi-temporal table for these where the time-of-entry-according-to-bash AND time-of-import are stored separately.

If I can get around to it, I'll probably investigate this further along with more granular commits for the changes I made here.

[jcsalterego]

There are a lot of good ideas but this has really drifted a bit from master, and there are some design choices that I'm not 100% sold on right now. I'm going to try and take a closer look at the various ideas you've implemented here.

[eliask]

Yeah, I was just dumping everything there now, though pull requests tend to update automatically.

I think I'll clean it up somewhat later :)

[eliask]

So here it is, everything is more or less split off into fine-grained commits.

• zsh and bash support are concurrently supported and imported from.
• zsh and bash history entries are more parsed robustly.
• I should still make the timestamped bash/readline history the default probably
• regarding the failing Travis CI check. I should still check that. Weird, seeing as it works locally for me...

[jcsalterego]

Thanks for all these commits! My time for working on this has been diminished to near zero but when I get a chance I'll take a close look, especially the Travis CI error.

[theY4Kman]

FWIW, I just recently migrated to this branch, and used a not-so-super-simple query to migrate. I still rock bash, and my history table was littered with rows containing #1662176785 unix timestamps in the command column; rows with valid commands would find their timestamp in the preceding row, e.g.

id	metadata_id	command	command_timestamp
1	4	xfconf-query -c xfce4-panel -p /plugins/plugin-27/size-max	1970-01-01 00:00:00
2	4	#1638675651	1970-01-01 00:00:00

Sometimes there would be multiple timestamp rows with no commands in between; I imagine these occur when I smash the Enter key at a terminal for some vertical space between commands.

So, what I did was delete (well, rename) my old history table, ran hist once to import my old .bash_history into its newly-created table, and ran this UPDATE query:

UPDATE history
SET command_timestamp = ts.timestamp
FROM (
    WITH commands AS (
        SELECT id, command
        FROM history
        WHERE NOT command REGEXP '^#\d+'
        ORDER BY id DESC
    )
    SELECT commands.id, commands.command, cast(substr(history.command, 2) AS int) AS timestamp
    FROM commands
    JOIN history ON history.id = commands.id - 1 AND history.command REGEXP '^#\d+'
) AS ts
WHERE history.id = ts.id;

Then I followed it up with a DELETE to kill the hanging timestamp command rows.

DELETE FROM history
WHERE command REGEXP '^#\d+';

---

On another note: though a hist /search was now showing the times of each command, they were being rendered as UNIX timestamps. idk about you, but I find these very hard to decipher :P I changed the search output to:

coalesce(datetime(command_timestamp, 'unixepoch', 'localtime'), command_timestamp)

I also screwed with the formatting a bit, added some colour (as I simply assume the caller is running a 256color term), and sorted by timestamp instead of id (well, along with id)

WITH color (reset, bold, grey50, grey58) AS (
    VALUES ('\033[0m', '\033[1m', '\033[38;5;8m', '\033[38;5;246m')
)
SELECT
    grey50 || '#' || grey58 || id
    || grey50 || ' @ ' || grey58 || coalesce(datetime(command_timestamp, 'unixepoch', 'localtime'), command_timestamp) || reset
    || '\n\t' || replace(command, '\n', '\n\t')
FROM history, color
WHERE 1
$(search_terms)
ORDER BY command_timestamp ASC, id DESC;

[jcsalterego]

@theY4Kman thanks for all the feedback and your notes! I have to dust off this project and incorporate some of the changes that @eliask so graciously provided years ago.

[jcsalterego]

Apologies all - I'm going to finally close this PR.