cloc
cloc copied to clipboard
AlDanial
Timeout on QML file
I get a timeout when invoking cloc on a particular QML file. The weird thing is, if I add a comment line as follows, it works fine:
--- a/source/updater/editor/main.qml
+++ b/source/updater/editor/main.qml
@@ -797,6 +797,7 @@ ApplicationWindow
function generateJSON(onComplete)
{
+ /*abc*/
updateObject = {};
updateObject.version = version.text;
updateObject.targetVersionRegex = targetVersionRegex.text;
Wow, that's messed up! I had to bump up the timeout value to 3 seconds for it to go through:
/tmp> cloc --timeout 3 main.qml
1 text file.
1 unique file.
0 files ignored.
github.com/AlDanial/cloc v 1.89 T=2.48 s (0.4 files/s, 368.5 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
QML 1 185 20 710
-------------------------------------------------------------------------------
(this actually points out another bug: I tried giving --timeout a float and it complained). A few experiments revealed the location of the delay: the Regexp::Common module. This is easy to demonstrate by commenting out line 9087 in the current development version, ie, where the language filters for QML are defined, including the call to Regexp::Common:
9084 'QML' => [ 9085 [ 'rm_comments_in_strings', '"', '/*', '*/' ], 9086 [ 'rm_comments_in_strings', '"', '//', '' ], 9087 # [ 'call_regexp_common' , 'C++' ], 9088 [ 'remove_inline' , '//.*$' ], 9089 ],
then cloc runs quickly--with incorrect results though.
The Regexp::Common module bundled with cloc is from 2013. Do you know if your system perl has Regexp::Common installed or if uses the bundled one? It isn't straightforward to figure this out; basically have to temporarily add a debug statement as with line 78 below:
71 BEGIN {
72 if (eval "use Regexp::Common;") {
73 $HAVE_Rexexp_Common = 1;
74 } else {
75 $HAVE_Rexexp_Common = 0;
76 }
77 }
78 die "HAVE_Rexexp_Common=$HAVE_Rexexp_Common";
I should look for a newer version--but no guarantees that will fix this.
This sent me down a bit of a rabbit hole :grin:. First of all, to answer your question:
HAVE_Rexexp_Common=0 at ./cloc line 78.
Fair enough, I'll install the system version, and see if that works any better:
$ sudo apt install libregexp-common-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
libregexp-common-perl is already the newest version (2017060201-1).
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
Hmm, that's weird. I do a bit of googling and change the test for RegExpCommon:
--- a/cloc
+++ b/cloc
@@ -69,7 +69,7 @@ my $HAVE_Rexexp_Common;
# Regexp::Common isn't in the standard distribution. It will
# be installed in a temp directory if necessary.
BEGIN {
- if (eval "use Regexp::Common;") {
+ if (eval{require Regexp::Common}) {
$HAVE_Rexexp_Common = 1;
} else {
$HAVE_Rexexp_Common = 0;
I am not a perl hacker (IANAPH?) so I don't really know the ins and outs of this, but it seems to me like the test is broken? (I also note that the variable is named HAVE_ReXexp_Common, not HAVE_ReGexp_Common, but that's unimportant.) Anyway, with the test changed to the above (so using the system Regexp::Common), cloc now works on the problematic QML:
$ ./cloc ~/sources/graphia/source/updater/editor/main.qml
1 text file.
1 unique file.
Use of uninitialized value in regexp compilation at ./cloc line 12234.
0 files ignored.
github.com/AlDanial/cloc v 1.89 T=0.02 s (60.3 files/s, 55193.8 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
QML 1 185 0 730
-------------------------------------------------------------------------------
...albeit with a warning, perhaps related to the timeout? Who knows.
Thanks for doing the background research. I agree the existence test for Regexp::Common was broken; should be fixed now.
What bugs me is that even with that fix plus an updated Regexp::Common, cloc still times out on your file. I'm still puzzled as to what's happening.
Hmm yes. Having forced it to use the system library, I previously thought I had managed to stop it timing out, but now I've tried it again, it does indeed still time out. Perhaps I'd accidentally managed to test with a copy of main.qml that included the comment line.
Not sure if it's the same bug, but anyway... I get timeouts counting the lines of the dependencies for the latest release of https://git.sr.ht/~sircmpwn/aerc/refs, both using cloc from my distro's package manager and the latest from this repo:
5 errors:
Line count, exceeded timeout: ./pkg/mod/github.com/go-ini/[email protected]/ini_test.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables10.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables11.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables12.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables9.0.0.go
Regexp::Common is installed; those files are automatically generated and make heavy use of unicode, but I have no idea if that is the reason.
aerc and scdoc present no issues for me but I take it I'm not cloning the thing that produces the problem you see. What is the link to either clone or download a tarball for "aerc/refs" ?
> git clone https://git.sr.ht/~sircmpwn/aerc
> cloc aerc/
174 text files.
174 unique files.
56 files ignored.
github.com/AlDanial/cloc v 1.91 T=0.12 s (1257.5 files/s, 171729.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Go 145 2381 663 17221
make 1 19 1 102
Markdown 1 27 0 43
YAML 1 0 0 13
Bourne Shell 2 0 2 12
-------------------------------------------------------------------------------
SUM: 150 2427 666 17391
-------------------------------------------------------------------------------
> git clone https://git.sr.ht/~sircmpwn/scdoc
> cloc scdoc
30 text files.
29 unique files.
11 files ignored.
github.com/AlDanial/cloc v 1.91 T=0.02 s (1394.8 files/s, 95599.8 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C 9 51 17 990
Bourne Shell 9 81 6 312
C/C++ Header 3 16 20 53
make 1 12 0 50
Markdown 1 12 0 15
YAML 1 0 0 10
-------------------------------------------------------------------------------
SUM: 24 172 43 1430
-------------------------------------------------------------------------------
Sorry I didn't make it clear the first time... It's not aerc itself, it's the dependencies that are downloaded when you build it - "go build" will download its dependencies to $HOME/go, so try something like this in a temporary $HOME:
$ mkdir /tmp/$$;cd /tmp/$$
$ export HOME=$PWD
$ git clone https://git.sr.ht/~sircmpwn/aerc
Cloning into 'aerc'...
(...)
$ cd aerc
$ make
go build \
-ldflags "-X main.Prefix=/usr/local \
-X main.ShareDir=/usr/local/share/aerc \
-X main.Version=0.5.2.r50.g0b19b5e" \
-o aerc
go: downloading git.sr.ht/~sircmpwn/getopt v0.0.0-20190808004552-daaf1274538b
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/gdamore/tcell/v2 v2.0.0
go: downloading github.com/go-ini/ini v1.52.0
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/riywo/loginshell v0.0.0-20190610082906-2ed199a032f6
go: downloading github.com/emersion/go-message v0.13.1-0.20201112194930-f77964fe28bd
go: downloading github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21
go: downloading github.com/emersion/go-smtp v0.12.1
go: downloading github.com/miolini/datacounter v1.0.2
go: downloading github.com/pkg/errors v0.9.1
go: downloading golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
go: downloading github.com/imdario/mergo v0.3.8
go: downloading github.com/kyoh86/xdg v1.2.0
go: downloading github.com/emersion/go-imap v1.0.6-0.20201014170015-5a03a09eba6d
go: downloading github.com/emersion/go-pgpmail v0.0.0-20200303213726-db035a3a4139
go: downloading github.com/ProtonMail/crypto v0.0.0-20200420072808-71bec3603bf3
go: downloading github.com/mattn/go-runewidth v0.0.8
go: downloading github.com/creack/pty v1.1.10
go: downloading github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: downloading github.com/ddevault/go-libvterm v0.0.0-20190526194226-b7d861da3810
go: downloading golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527
go: downloading github.com/gdamore/encoding v1.0.0
go: downloading github.com/lucasb-eyer/go-colorful v1.0.3
go: downloading golang.org/x/text v0.3.4-0.20201021145329-22f1617af38e
go: downloading github.com/martinlindhe/base36 v1.1.0
go: downloading github.com/emersion/go-textwrapper v0.0.0-20200911093747-65d896831594
go: downloading github.com/mattn/go-pointer v0.0.0-20190911064623-a0a44394634f
go: downloading golang.org/x/net v0.0.0-20200301022130-244492dfa37a
go: downloading github.com/emersion/go-imap-idle v0.0.0-20190519112320-2704abd7050e
go: downloading github.com/emersion/go-imap-sortthread v1.1.1-0.20201009054724-d020d96306b3
go: downloading github.com/emersion/go-maildir v0.2.0
go: downloading github.com/fsnotify/fsnotify v1.4.7
sed -e 's:@SHAREDIR@:/usr/local/share/aerc:g' > aerc.conf < config/aerc.conf.in
scdoc < doc/aerc.1.scd > aerc.1
scdoc < doc/aerc-search.1.scd > aerc-search.1
scdoc < doc/aerc-config.5.scd > aerc-config.5
scdoc < doc/aerc-imap.5.scd > aerc-imap.5
scdoc < doc/aerc-maildir.5.scd > aerc-maildir.5
scdoc < doc/aerc-sendmail.5.scd > aerc-sendmail.5
scdoc < doc/aerc-notmuch.5.scd > aerc-notmuch.5
scdoc < doc/aerc-smtp.5.scd > aerc-smtp.5
scdoc < doc/aerc-tutorial.7.scd > aerc-tutorial.7
scdoc < doc/aerc-templates.7.scd > aerc-templates.7
scdoc < doc/aerc-stylesets.7.scd > aerc-stylesets.7
$ cd ~/go
$ exec cloc .
100 files
200 files
300 files
400 files
500 files
600 files
700 files
800 files
900 files
1000 files
1100 files
1200 files
1300 files
1400 files
1500 files
1600 files
1700 files
1800 files
1900 files
2000 files
2100 files
2200 files
2300 files
2400 files
2500 files
2600 files
2700 files
2800 files
2900 files
3000 files
3089 text files.
classified 3075 files
Duplicate file check 3075 files (2346 known unique)
Unique: 100 files
Unique: 200 files
Unique: 300 files
Unique: 400 files
Unique: 500 files
Unique: 600 files
Unique: 700 files
Unique: 800 files
Unique: 900 files
Unique: 1000 files
Unique: 1100 files
Unique: 1200 files
Unique: 1300 files
Unique: 1400 files
Unique: 1500 files
Unique: 1600 files
Unique: 1700 files
Unique: 1800 files
Unique: 1900 files
Unique: 2000 files
Unique: 2100 files
Unique: 2200 files
Unique: 2300 files
2940 unique files.
Counting: 100
Counting: 200
Counting: 300
Counting: 400
Counting: 500
Counting: 600
Counting: 700
Counting: 800
Counting: 900
Counting: 1000
Counting: 1100
Counting: 1200
Counting: 1300
Counting: 1400
Counting: 1500
Counting: 1600
Counting: 1700
Counting: 1800
Counting: 1900
Counting: 2000
Counting: 2100
Counting: 2200
Counting: 2300
Counting: 2400
Counting: 2500
Counting: 2600
Counting: 2700
Counting: 2800
Counting: 2900
567 files ignored.
7 errors:
Line count, exceeded timeout: ./pkg/mod/github.com/go-ini/[email protected]/ini_test.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/idna/tables12.00.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables10.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables11.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables12.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables13.0.0.go
Line count, exceeded timeout: ./pkg/mod/golang.org/x/[email protected]/internal/export/idna/tables9.0.0.go
github.com/AlDanial/cloc v 1.90 T=33.02 s (77.4 files/s, 38734.1 lines/s)
--------------------------------------------------------------------------------
Language files blank comment code
--------------------------------------------------------------------------------
Go 2320 58850 118445 1063411
Assembly 70 1259 1430 10426
XML 6 228 20 5802
C 18 1231 358 5657
HTML 10 441 16 2214
Markdown 48 790 0 2119
JSON 16 0 0 1788
Bourne Shell 8 127 436 857
C/C++ Header 6 175 162 605
YAML 32 57 14 396
AsciiDoc 2 95 0 254
Perl 3 53 12 197
make 7 60 4 173
INI 4 25 5 92
Bourne Again Shell 3 16 6 87
Dockerfile 3 18 20 81
Jupyter Notebook 1 0 253 74
--------------------------------------------------------------------------------
SUM: 2557 63425 121181 1094233
--------------------------------------------------------------------------------
At least I get this result on my computers (plural), maybe I just need to buy something faster :D
scdoc aerc.1 /bin/sh: 1: scdoc: not found make: *** [Makefile:52: aerc.1] Error 127
Starting to go down a rabbit hole here. Can you just post the first of the problem files,
./pkg/mod/github.com/go-ini/[email protected]/ini_test.go ?
Here's all of them just in case all.tar.gz
(fun fact: if it failed at scdoc you should already have everything in ~/go, remember to clean it up if you didn't use a temporary directory as $HOME)
Here's what I found: cloc doesn't time out on me but is super-slow
github.com/AlDanial/cloc v 1.91 T=25.05 s (0.3 files/s, 1170.1 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Go 7 228 29085 0 ------------------------------------------------------------------------------- SUM: 7 228 29085 0 -------------------------------------------------------------------------------
Counting only 1k lines/second is pretty terrible.
I took a look in one of the files, tables12.00.go, and was struck by the strings of hex characters. I copied the file to a temp location, deleted definitions for mappings and xorData, then ran cloc again. Normal speed this time; instead of 4.01 seconds for the original, took only 0.02 seconds with hex strings removed.
I'm guessing regular expressions from the Regexp::Common module freak out with these strings and do a ton of back-tracking. No work-arounds come to mind.
The attached C++ file generates a timeout too. The file was a bit bigger, I reduced it to 300 lines. cloc1.94_timeout.zip with cloc 1.64 there is no timeout.
@andreaplanet : Which platform are you running on? I have no issues on Linux:
» cloc cloc1.94_timeout.cpp
1 text file.
1 unique file.
0 files ignored.
github.com/AlDanial/cloc v 1.94 T=0.96 s (1.0 files/s, 313.1 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C++ 1 79 38 183
-------------------------------------------------------------------------------
Nonetheless I have a pretty good idea about the cause of your timeout: the three occurrences of the string "-/*+^", specifically, the /* characters. cloc is regex-based and might see these as the beginning of a conventional C comment. There are no matching */ characters so the regex gets stuck in a loop. Ref Limitation #2, this is a known problem.
However: work-around logic exists in cloc to attempt to correct for this. For what ever reason, the work-around succeeds for me but not you. For starters, try running with the --strip-str-comments switch.
I used the cloc1.94.exe downloaded from here under Windows 10. I tried now under Fedora with cloc 1.90, same result:
cloc cloc1.94_timeout.cpp
1 text file.
1 unique file.
0 files ignored.
1 error:
Line count, exceeded timeout: cloc1.94_timeout.cpp
github.com/AlDanial/cloc v 1.90 T=1.01 s (1.0 files/s, 297.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
C++ 1 79 221 0
-------------------------------------------------------------------------------
Nice the catch "-/*+^" ... lol.... But why did Version 1.64 not had the same issue? Maybe the " character should be part of the RegExp for C++ to fix that?
It's not an issue for me. Just reporting. :-)