jcsalterego
Import truncates commands at a length of 23 characters on Linux
Testing this on linux in the hope that historian will be able maintain the command history intelligently across multiple terminals in bash (depends on https://github.com/jcsalterego/historian/issues/9).
However, at the moment I'm seeing a more fundamental problem - imported commands truncated to 23 characters?
Nice project!
Oof, do you have any example of import history that causes this? The delimiter as of 0.0.2 is \x01, which may be wonderful or terrible, I'm not sure yet.
Hi @jcsalterego, I've narrowed down the test case. It seems like the length of the last line in my .bash_history file determines the max length at which all other imported lines will be truncated at,
eg. A .bash_history file that won't reveal the problem
git branch here-is-a-really-long-line-that-i-have-observed-getting-truncated-1
git branch here-is-another-really-long-line-that-i-have-observed-getting-truncated-2
git fooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
eg. A .bash_history file that will reveal the problem
git branch here-is-a-really-long-line-that-i-have-observed-getting-truncated-1
git branch here-is-another-really-long-line-that-i-have-observed-getting-truncated-2
git fooo
Quite interesting, thanks @theirishpenguin. I'll have to investigate soon!
Ah, the problem isn't the import, it is displaying the output using "select" of sqlite.
If we set an explicit with as in https://github.com/jcsalterego/historian/pull/11 then the problem is reduced. That leaves a design decision as to how to display long commands when searching.
The .width docs are useful at http://www.sqlite.org/cli.html
This s/o may help too http://stackoverflow.com/questions/1751189/sqlite3-is-chopping-cutting-truncating-my-text-columns
Does it affect hist search then? Or are you using sqlite directly when searching?
@jcsalterego Affects hist search too. I chose 75 in PR as I know that adhering to a command line length of 80 is traditionally polite (actually I guess I should allow for a spacing of 2 characters between the output of the id column and the command column).
I find a width of 90 works well for me for the command column - though that might be due to my long branch names 😄 Ideally we wouldn't care about setting a fixed width - though if someone puts in a crazy 4 line curl command they may think otherwise.
Perhaps the ultimate solution would be to immediately pipe the output into something like "less" (however git log does it). Then the user is looking into a viewport of the output and can scroll left and right in the small number of cases where they have a superlong command. Lines then never need to wrap.
@theirishpenguin these are good ideas. hist log actually pipes into less (not as fancy as git which auto-detects whether there's a pager needed.) Surely there's a way to disable the formatting/truncation entirely. I'll have to investigate.
@theirishpenguin I've given it a shot and have been unable to reproduce this.
$ cat test.txt
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
$ HISTORIAN_SRC=test.txt HISTORIAN_DB=test.db ./hist import
Imported 1 item(s).
$ HISTORIAN_SRC=test.txt HISTORIAN_DB=test.db ./hist search A
1
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
(With line-wrapping you can see all 512 "A"s in this example.)
I've also tried reproducing the issue with a branch:
- https://github.com/jcsalterego/historian/tree/fix-search-length
- CI: https://travis-ci.org/jcsalterego/historian/builds/223411594
Side note: the CI notes Error: near line 8: near "NOT": syntax error but I believe that's unrelated to this issue and related to the version of sqlite3.
Thus - could you pull down that branch fix-search-length and run make test at the root level and see if you can get the test to fail? Or, if you could provide exact reproduction steps, the sooner I can reproduce the issue the sooner I can fix it. Thanks!
Cheers! I'll look into it when I get the chance. What OS did you try to reproduce this on?
On Wed, Apr 19, 2017, 6:08 AM Jerry Chen [email protected] wrote:
@theirishpenguin https://github.com/theirishpenguin I've given it a shot and have been unable to reproduce this.
$ cat test.txt AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA $ HISTORIAN_SRC=test.txt HISTORIAN_DB=test.db ./hist import Imported 1 item(s). $ HISTORIAN_SRC=test.txt HISTORIAN_DB=test.db ./hist search A 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
(With line-wrapping you can see all 512 "A"s in this example.)
I've also tried reproducing the issue with a branch:
- https://github.com/jcsalterego/historian/tree/fix-search-length
- CI: https://travis-ci.org/jcsalterego/historian/builds/223411594
Side note: the CI notes Error: near line 8: near "NOT": syntax error but I believe that's unrelated to this issue and related to the version of sqlite3.
Thus - could you pull down that branch fix-search-length and run make test at the root level and see if you can get the test to fail? Or, if you could provide exact reproduction steps, the sooner I can reproduce the issue the sooner I can fix it. Thanks!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jcsalterego/historian/issues/10#issuecomment-295094432, or mute the thread https://github.com/notifications/unsubscribe-auth/AACa5YKtQEYOaaOrk6QKrQS4O3Sb5q2tks5rxZa1gaJpZM4NAMlm .
I just tested on MacOS 10.12.4 and Ubuntu 16.04:
--J-MCKNIGHT--tpci@bastion:~/historian$ HISTORIAN_SRC=test.txt HISTORIAN_DB=test.db ./hist search A
1
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
@kudakeru thanks!
@theirishpenguin I tried on MacOS 10.12.4 (16E195). It looks like CI is using Linux testing-docker-65c7243e-25ba-42d8-abad-168ccdf60c0b 4.8.12-040812-generic #201612020431 SMP Fri Dec 2 09:33:31 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux.
@jcsalterego @kudakeru Thanks for all your efforts guys.
Here are the results of make test...
$ make test
make[1]: Entering directory `/tmp/historian/test'
uname -a
Linux me-VirtualBox 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
sqlite3 --version
-- Loading resources from /home/me/.sqliterc
3.8.2 2013-12-06 14:53:30 27392118af4c38c5203a04b8013e1afdb1cebd0d
awk --version
GNU Awk 4.0.1
test_exit_codes.sh:
htest_bare_command_returns_zero_exit_code ... PASS
htest_version_returns_zero_exit_code ... PASS
test_import.sh:
htest_bare_command_returns_zero_exit_code ... PASS
htest_import_handles_funny_characters ... htest_import_handles_funny_characters : Failed: expected 4, but got COUNT(*)
----------
4 : rows imported by simple bash_history
1c1,3
< echo `ls`
---
> command
> ----------
> echo `ls`
3,4c5,6
< foo \\bar\\baz
< foo ; bar ; baz
---
> foo \\bar\
> foo ; bar
htest_import_handles_funny_characters : Failed: expected 0, but got 1: exported commands should match
FAIL
htest_import_run_twice_will_do_nothing_the_second_time ... htest_import_run_twice_will_do_nothing_the_second_time : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history
htest_import_run_twice_will_do_nothing_the_second_time : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history should not change
FAIL
htest_import_simple ... htest_import_simple : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history
FAIL
htest_import_will_dedupe ... htest_import_will_dedupe : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history
FAIL
htest_version_returns_zero_exit_code ... PASS
test_search.sh:
htest_bare_command_returns_zero_exit_code ... PASS
htest_import_handles_funny_characters ... htest_import_handles_funny_characters : Failed: expected 4, but got COUNT(*)
----------
4 : rows imported by simple bash_history
1c1,3
< echo `ls`
---
> command
> ----------
> echo `ls`
3,4c5,6
< foo \\bar\\baz
< foo ; bar ; baz
---
> foo \\bar\
> foo ; bar
htest_import_handles_funny_characters : Failed: expected 0, but got 1: exported commands should match
FAIL
htest_import_run_twice_will_do_nothing_the_second_time ... htest_import_run_twice_will_do_nothing_the_second_time : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history
htest_import_run_twice_will_do_nothing_the_second_time : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history should not change
FAIL
htest_import_simple ... htest_import_simple : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history
FAIL
htest_import_will_dedupe ... htest_import_will_dedupe : Failed: expected 3, but got COUNT(*)
----------
3 : rows imported by simple bash_history
FAIL
htest_search_length ... PASS
htest_version_returns_zero_exit_code ... PASS
7 / 19 assertions passed
make[1]: *** [test] Error 1
make[1]: Leaving directory `/tmp/historian/test'
make: *** [test] Error 2
P.S. In case it helps...
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
@theirishpenguin Very very interesting. All this might come down to older versions of sqlite, so I'll have to get an environment up and investigate. Thanks for running make test!
Ah, that is interesting. Glad the test helped!
On Thu, Apr 20, 2017, 4:09 AM Jerry Chen [email protected] wrote:
@theirishpenguin https://github.com/theirishpenguin Very very interesting. All this might come down to older versions of sqlite, so I'll have to get an environment up and investigate. Thanks for running make test!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jcsalterego/historian/issues/10#issuecomment-295562530, or mute the thread https://github.com/notifications/unsubscribe-auth/AACa5VvREA72c9ns_bdAMQccupGxG3dbks5rxsxdgaJpZM4NAMlm .
@theirishpenguin I think it might be something in your ~/.sqliterc.
This is what I see with a stock Trusty, sqlite3, and gawk:
vagrant@trusty:~/gh/historian$ make test
make[1]: Entering directory `/gh/historian/test'
uname -a
Linux trusty 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
sqlite3 --version
3.8.2 2013-12-06 14:53:30 27392118af4c38c5203a04b8013e1afdb1cebd0d
awk --version
GNU Awk 4.0.1
Copyright (C) 1989, 1991-2012 Free Software Foundation.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.
test_exit_codes.sh:
htest_bare_command_returns_zero_exit_code ... PASS
htest_version_returns_zero_exit_code ... PASS
test_import.sh:
htest_bare_command_returns_zero_exit_code ... PASS
htest_import_handles_funny_characters ... PASS
htest_import_run_twice_will_do_nothing_the_second_time ... PASS
htest_import_simple ... PASS
htest_import_will_dedupe ... PASS
htest_version_returns_zero_exit_code ... PASS
test_search.sh:
htest_bare_command_returns_zero_exit_code ... PASS
htest_import_handles_funny_characters ... PASS
htest_import_run_twice_will_do_nothing_the_second_time ... PASS
htest_import_simple ... PASS
htest_import_will_dedupe ... PASS
htest_search_length ... PASS
htest_version_returns_zero_exit_code ... PASS
19 / 19 assertions passed
make[1]: Leaving directory `/gh/historian/test'
Hi @jcsalterego - sorry about the delay replying. Yes, I have the following settings in my ~/.sqliterc...
.headers on
.mode column
... and I have run your test without them. You are right - this is what was causing the different output. Thanks for investigating!
@theirishpenguin (Years later) Is this still an issue, or did you move on? :)
@jcsalterego (Years later, plus 6 months :slightly_smiling_face: ). Sorry, about the delay. I haven't used historian since, but I have just installed it and cannot reproduce the issue :partying_face: Works for me now!
Here is the version of sqlite3 that I am using now (on a variant of ubuntu) - 3.40.1 in case that is useful to know.
So you can close this issue. Thanks for following up and for this project.