1.7rc1 make check shtest last JSON sequence test fails

[BrianInglis]

trafficstars

Describe the bug 1.7rc1 shtest last JSON sequence test assertion fails in jq_util_input_get_position

To Reproduce $ printf '1\n' | jq -cen --seq '[inputs] == []' assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/src/util.c", line 352, function: jq_util_input_get_position Aborted (core dumped)

Expected behavior jq: error (at <stdin>:1): Unfinished abandoned text at EOF at line 2, column 0

Environment (please complete the following information):

• OS and Version: Cygwin 3.4.7-1 (Windows 10 Home 202209/2022H2)
• jq version 1.7rc1

Additional context jq 1.6 also fails the same assertion: $ printf '1\n' | jq -cen --seq '[inputs] == []' assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position Aborted (core dumped)

[emanuele6]

I get jq: error (at <stdin>:1): Unfinished abandoned text at EOF at line 2, column 0 in both jq 1.6, 1.7rc1, and the master branch, so this is probably windows specific.

[emanuele6]

Though, we run tests on the windows amd64 CI, so it is weird that the CI does not fail this assertion.

[nicowilliams]

We don't use Cygwin in the CI though.

[nicowilliams]

Maybe we should add a Cygwin build just for this.

[BrianInglis]

Cygwin is NOT Windows - it offers: "Get that Linux feeling - on Windows, used by many Unix folks and devs to make Windows mostly invisible so tolerable where mandatory. Cygwin is POSIX and as far as possible Linux (especially Fedora and somewhat RHEL, formerly Solaris) compatible, building with standard build chains with sh == dash, where '\r' is data not an EoL and such scripts won't run! ;^> GNU autotools application source builds (like yours) work flawlessly, unless there are unconditional non-POSIX Linux (or BSD) specific dependencies like #include <linux/...> or similar. There should be NO Windows artifacts visible except at the system layer, which should be dealt with by the autotools dependency checks! Your/our jq-devel (/jq-dev) package is accepted by other packages' autoconfig tests. Our packaging is Gentoo portage inspired (predating RedHat) entirely in bash using coreutils.

All other tests pass. There are sometimes test script dependencies on bashisms, although these do not appear to be an issue in shtest: $ checkbashisms -fpnx jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/[!.]test possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/base64test line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/jqtest line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/manonigtest line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/mantest line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/onigtest line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/optionaltest line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/shtest line 3 (sourced script with arguments): . "${0%/}/setup" "$@" possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/shtest line 187 (sourced script with arguments): $VALGRIND $Q $JQ -c '. as $d|path(..) as $p|$d|getpath($p)|select((type|. != "array" and . != "object") or length==0)|[$p,.]' < "$JQTESTDIR/torture/input0.json" > $d/out0 possible bashism in jq-1.7-0.rc1.x86_64/src/jq-jq-1.7rc1/tests/utf8test line 3 (sourced script with arguments): . "${0%/*}/setup" "$@"

Suggestions for source patches to generate useful diagnostic output?

[BrianInglis]

Maybe we should add a Cygwin build just for this.

There are issues with pipenv not being packaged for Cygwin, installed using pip3, getting pipenv sync to install dependencies, that I have put at the end of the queue for now, until I can talk to the Cygwin python packaging volunteers. Sometimes we get into dependency hell we volunteers don't have enough free time to address, while supporting the packages we have. So we hope someone in a company eventually needs a package enough to do the work! ;^>

[nicowilliams]

@BrianInglis we know what Cygwin is :)

This is a very weird failure. I think it denotes memory corruption because I don't see how jq_util_input_next_input_cb isn't the callback function when the assert at src/util.c:371 runs.

Running that command with valgrind on Linux shows nothing amiss.

[nicowilliams]

There are issues with pipenv not being packaged for Cygwin ...

You don't need any of that unless you want to do jq dev work on Cygwin. jq comes with pre-built jq.1.prebuilt, tests/man.test, src/parser.c, src/parser.h, src/lexer.c, src/lexer.h, etc., so you don't have to build those if you're just building from the dist tarball, or even from a Git clone.

[BrianInglis]

Good re Cygwin. Unfortunately valgrind is not packaged for Cygwin, and most package problems seem to be unreproducable in gdb, so I just add debug output to understand the situation. I can hack around later to see what I can generate, and get back to you.

Re docs - I know I don't have to build docs, as a casual user of some JSON data, but as potential adopter and maintainer of the Cygwin jq package, lacking a current maintainer, I want to be able to respond to any questions about the build that other package maintainers may raise, if they have the time to do a proper review of the CI playground build. That is also why I am raising the test failure issue with you - I need to have it fixed, or know why it fails, and be able to explain that, before I adopt the package. Which is why I want to deal with all the issues, starting at this rc stage, before your next official release. I may develop patches for Cygwin issues, and may submit those or PRs to you, and may grrracefully accept a NO or WONTFIX! ;^>

[nicowilliams]

That is also why I am raising the test failure issue with you - I need to have it fixed, or know why it fails, and be able to explain that, before I adopt the package.

Hard to say what's up. If you look you'll see that main() sets the callback that the failed assertion wants to see. There's no way it could be unset or replaced. So what could be going on?

Re docs - I know I don't have to build docs, as a casual user of some JSON data, but as potential adopter and maintainer of the Cygwin jq package, lacking a current maintainer, I want to be able to respond to any questions about the build that other package maintainers may raise, if they have the time to do a proper review of the CI playground build.

No questions should arise if you just use the pre-built manual page.

Which is why I want to deal with all the issues, starting at this rc stage, before your next official release.

+1

I may develop patches for Cygwin issues, and may submit those or PRs to you, and may grrracefully accept a NO or WONTFIX! ;^>

We've no reason to reject patches for Cygwin support. Since jq doesn't call fork(), exec*(), etc., even the subset of POSIX supported by msys will do, and there should be no problem at all on Cygwin.

I think we need you to use a debugger to see what's up with this one failure.

[BrianInglis]

Finally got a round tuit. Downloaded updated "rc1" aka rc2 and rebuilt under Cygwin: jq-cygport-64-all-test-check.log Still failed same shtest case. Created file 1 containing '1' then under gdb did: (gdb) run -cen --seq '[inputs] == []' 1 with session log jq-shtest-gdb.log does this help any, or need more output?

[nicowilliams]

Can you try applying this patch:

diff --git a/src/util.c b/src/util.c
index a5c2b61b..a134b950 100644
--- a/src/util.c
+++ b/src/util.c
@@ -349,7 +349,6 @@ jv jq_util_input_get_position(jq_state *jq) {
   jq_input_cb cb = NULL;
   void *cb_data = NULL;
   jq_get_input_cb(jq, &cb, &cb_data);
-  assert(cb == jq_util_input_next_input_cb);
   if (cb != jq_util_input_next_input_cb)
     return jv_invalid_with_msg(jv_string("Invalid jq_util_input API usage"));
   jq_util_input_state *s = (jq_util_input_state *)cb_data;

[nicowilliams]

From your log (thank you):

(gdb) p *cb
$6 = (jq_input_cb) 0x100401ac8 <jq_util_input_next_input_cb>
(gdb) p **cb
$7 = {jv (jq_state *, void *)} 0x100401ac8 <jq_util_input_next_input_cb>
(gdb) n
jq_util_input_get_position (jq=jq@entry=0xa00000270) at /usr/src/debug/jq-1.7-0.rc1/src/util.c:352
352       assert(cb == jq_util_input_next_input_cb);
(gdb) n
assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/jq/jq-1.7-0.rc1.x86_64/src/jq-1.7rc1/src/util.c", line 352, function: jq_util_input_get_position

which has me very confused. cb does == jq_util_input_next_input_cb here, and yet the assertion fails. Huh?