perl5
perl5 copied to clipboard
Perl
BBC: Math-FastGF2-0.07 triggers "stack smashing detected"
Description
With 1ef9039bcc the tests for DMALONE/Math-FastGF2/Math-FastGF2-0.07.tar.gz were starting to fail. Sample test report with v5.37.2: http://www.cpantesters.org/cpan/report/21cf2db8-0954-11ed-aef9-e4323abf57b8
Steps to Reproduce
cpan -i DMALONE/Math-FastGF2/Math-FastGF2-0.07.tar.gz
Expected behavior
Tests should succeed and module should be installed.
Observed behavior
PERL_DL_NONLAZY=1 "/tmp/basesmoker-reloperl-a8HE/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/Cauchy.t ........ ok
t/Math-FastGF2.t .. ok
*** stack smashing detected ***: terminated
t/Matrix.t ........
Failed 166/195 subtests
*** stack smashing detected ***: terminated
t/Vandermonde.t ...
All 19 subtests passed
Test Summary Report
-------------------
t/Matrix.t (Wstat: 134 (Signal: ABRT, dumped core) Tests: 29 Failed: 0)
Non-zero wait status: 134
Parse errors: Bad plan. You planned 195 tests but ran 29.
t/Vandermonde.t (Wstat: 134 (Signal: ABRT, dumped core) Tests: 19 Failed: 0)
Non-zero wait status: 134
Parse errors: No plan found in TAP output
Files=4, Tests=109, 1 wallclock secs ( 0.06 usr 0.05 sys + 0.29 cusr 0.16 csys = 0.56 CPU)
Result: FAIL
Failed 2/4 test programs. 0/109 subtests failed.
make: *** [Makefile:1065: test_dynamic] Error 255
Perl configuration
# perl -V output goes here
Summary of my perl5 (revision 5 version 37 subversion 2) configuration:
Commit id: 7f9b65d969a1c8b44028cacbb9e88596d1e5a8a3
Platform:
osname=linux
osvers=5.15.0-41-generic
archname=x86_64-linux
uname='linux k93jammy 5.15.0-41-generic #44-ubuntu smp wed jun 22 14:20:53 utc 2022 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/host/k93jammy/v5.37.2/6567 -Dmyhostname=k93jammy -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Dlibswanted=cl pthread socket inet nsl gdbm dbm malloc dl ld sun m crypt sec util c cposix posix ucb BSD gdbm_compat -Uuseithreads -Uuselongdouble -DEBUGGING=-g'
hint=recommended
useposix=true
d_sigaction=define
useithreads=undef
usemultiplicity=undef
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
Compiler:
cc='cc'
ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
optimize='-O2 -g'
cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion=''
gccversion='11.2.0'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='cc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /usr/lib64
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=/lib/x86_64-linux-gnu/libc.so.6
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.35'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector-strong'
Characteristics of this binary (from libperl):
Compile-time options:
HAS_TIMES
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
PERL_USE_DEVEL
PERL_USE_SAFE_PUTENV
USE_64_BIT_ALL
USE_64_BIT_INT
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
Built under linux
Compiled at Jul 21 2022 06:44:32
%ENV:
PERL="/tmp/basesmoker-reloperl-a8HE/bin/perl"
PERL5LIB=""
PERL5OPT=""
PERL5_CPANPLUS_IS_RUNNING="522500"
PERL5_CPAN_IS_RUNNING="522500"
PERL_CANARY_STABILITY_NOPROMPT="1"
PERL_MM_USE_DEFAULT="1"
PERL_USE_UNSAFE_INC="1"
@INC:
/home/sand/src/perl/repoperls/installed-perls/host/k93jammy/v5.37.2/6567/lib/site_perl/5.37.2/x86_64-linux
/home/sand/src/perl/repoperls/installed-perls/host/k93jammy/v5.37.2/6567/lib/site_perl/5.37.2
/home/sand/src/perl/repoperls/installed-perls/host/k93jammy/v5.37.2/6567/lib/5.37.2/x86_64-linux
/home/sand/src/perl/repoperls/installed-perls/host/k93jammy/v5.37.2/6567/lib/5.37.2
.
Description
With 1ef9039 the tests for DMALONE/Math-FastGF2/Math-FastGF2-0.07.tar.gz were starting to fail. Sample test report with v5.37.2: http://www.cpantesters.org/cpan/report/21cf2db8-0954-11ed-aef9-e4323abf57b8
Steps to Reproduce
cpan -i DMALONE/Math-FastGF2/Math-FastGF2-0.07.tar.gz
commit 1ef9039bccbfe64f47f201b6cfb7d6d23e0b08a7
Author: Karl Williamson <[email protected]>
AuthorDate: Tue Jun 21 05:45:06 2022 -0600
Commit: Karl Williamson <[email protected]>
CommitDate: Wed Jun 29 11:22:51 2022 -0600
Eval param to SvPV-foo exactly once
This changes all the API macros that retrieve a PV into a call to an
inline function so as to evaluate the parameter just once.
This is strange. I was able to reproduce this "stack smashing" on Linux at a commit more recent than the one @andk cites.
$ ./bin/perl -v | head -2 | tail -1
This is perl 5, version 37, subversion 2 (v5.37.2 (v5.37.1-82-g768686e95a)) built for x86_64-linux
$ make test
"/home/jkeenan/testing/blead/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- FastGF2.bs blib/arch/auto/Math/FastGF2/FastGF2.bs 644
make[1]: Entering directory '/home/jkeenan/.cpanm/work/1658494201.311730/Math-FastGF2-0.07/clib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/jkeenan/.cpanm/work/1658494201.311730/Math-FastGF2-0.07/clib'
make[1]: Entering directory '/home/jkeenan/.cpanm/work/1658494201.311730/Math-FastGF2-0.07/clib'
No tests defined for Math::fastgf2 extension.
make[1]: Leaving directory '/home/jkeenan/.cpanm/work/1658494201.311730/Math-FastGF2-0.07/clib'
PERL_DL_NONLAZY=1 "/home/jkeenan/testing/blead/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/Cauchy.t ........ ok
t/Math-FastGF2.t .. ok
t/Matrix.t ........ 1/195 *** stack smashing detected ***: terminated
t/Matrix.t ........ Failed 166/195 subtests
t/Vandermonde.t ... 1/? *** stack smashing detected ***: terminated
t/Vandermonde.t ... All 19 subtests passed
Test Summary Report
-------------------
t/Matrix.t (Wstat: 134 (Signal: ABRT, dumped core) Tests: 29 Failed: 0)
Non-zero wait status: 134
Parse errors: Bad plan. You planned 195 tests but ran 29.
t/Vandermonde.t (Wstat: 134 (Signal: ABRT, dumped core) Tests: 19 Failed: 0)
Non-zero wait status: 134
Parse errors: No plan found in TAP output
Files=4, Tests=109, 1 wallclock secs ( 0.03 usr 0.00 sys + 0.24 cusr 0.03 csys = 0.30 CPU)
Result: FAIL
Failed 2/4 test programs. 0/109 subtests failed.
make: *** [Makefile:1060: test_dynamic] Error 255
However, I was not able to reproduce these failures on a recently built perl on FreeBSD.
$ ./bin/perl -v | head -2 | tail -1; uname -mrs
This is perl 5, version 37, subversion 2 (v5.37.2 (v5.37.1-184-g5888abcc02)) built for amd64-freebsd-thread-multi
FreeBSD 12.3-RELEASE-p1 amd64
$ make test
...
CPAN::Reporter: make result is 'pass', No errors.
DMALONE/Math-FastGF2/Math-FastGF2-0.07.tar.gz
/usr/bin/make -- OK
Running make test for DMALONE/Math-FastGF2/Math-FastGF2-0.07.tar.gz
"/usr/home/jkeenan/testing/blead/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- FastGF2.bs blib/arch/auto/Math/FastGF2/FastGF2.bs 644
No tests defined for Math::fastgf2 extension.
PERL_DL_NONLAZY=1 "/usr/home/jkeenan/testing/blead/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/Cauchy.t ........ ok
t/Math-FastGF2.t .. ok
t/Matrix.t ........ ok
t/Vandermonde.t ... ok
All tests successful.
Files=4, Tests=1111, 1 wallclock secs ( 0.09 usr 0.01 sys + 0.41 cusr 0.05 csys = 0.55 CPU)
Result: PASS
(/usr/bin/make test exited with 0)
CPAN::Reporter: Test result is 'pass', 'make test' no errors.
CPAN::Reporter: preparing a CPAN Testers report for Math-FastGF2-0.07
CPAN::Reporter: sending test report with 'pass' via Metabase
Could this problem be platform-specific?
@khwilliamson, can you take a look?
Thank you very much. Jim Keenan
Could this problem be platform-specific?
FWIW, I see the problem on Windows (MSWin32-x64-multi-thread), perl-5.37.2
It manifests itself as a couple of pop-ups telling me that perl.exe has stopped working. The test summary reports:
Test Summary Report
-------------------
t/Matrix.t (Wstat: 1280 (exited 5) Tests: 29 Failed: 0)
Non-zero exit status: 5
Parse errors: Bad plan. You planned 195 tests but ran 29.
t/Vandermonde.t (Wstat: 1280 (exited 5) Tests: 19 Failed: 0)
Non-zero exit status: 5
Parse errors: No plan found in TAP output
Files=4, Tests=109, 6 wallclock secs ( 0.03 usr + 0.05 sys = 0.08 CPU)
Result: FAIL
All tests pass when I build the Math-FastGF2-0.07 with perl-5.36.0.
There's a few warnings issued during make:
In file included from C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/perl.h:4477,
from FastGF2.xs:3:
perlsubs.c: In function 'mat_get_raw_values_c':
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
perlsubs.c:393:14: note: in expansion of macro 'SvPV'
393 | to_start=SvPV(Str,len) + width - 1;
| ^~~~
In file included from C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/perl.h:7543:
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
perlsubs.c: In function 'mat_getvals_str':
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
perlsubs.c:434:14: note: in expansion of macro 'SvPV'
434 | to_start=SvPV(Str,len) + width - 1;
| ^~~~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
perlsubs.c: In function 'mat_set_raw_values_c':
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
perlsubs.c:464:16: note: in expansion of macro 'SvPV'
464 | from_start=SvPV(Str,len) + width - 1;
| ^~~~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
perlsubs.c:472:16: note: in expansion of macro 'SvPV'
472 | from_start=SvPV(Str,len);
| ^~~~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
perlsubs.c: In function 'mat_setvals_str':
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
perlsubs.c:502:16: note: in expansion of macro 'SvPV'
502 | from_start=SvPV(Str,len) + width - 1;
| ^~~~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
perlsubs.c:511:16: note: in expansion of macro 'SvPV'
511 | from_start=SvPV(Str,len);
| ^~~~
C:\perl-5.37.2-1210\lib\MSWin32-x64-multi-thread\CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
In file included from FastGF2.xs:10:
perlsubs.c: In function 'mat_offset_to_rowcol':
perlsubs.c:530:17: warning: assignment to 'int' from 'SV *' {aka 'struct sv *'} makes integer from pointer without a cast [-Wint-conversion]
530 | *row = *col = &PL_sv_undef;
| ^
FastGF2.c: In function 'XS_Math__FastGF2_gf2_pow':
FastGF2.c:253:18: warning: implicit declaration of function 'gf2_pow' [-Wimplicit-function-declaration]
253 | RETVAL = gf2_pow(width, a, b);
| ^~~~~~~
That copy'n'paste is from the 5.37.2 build. I think it's pretty much the same warnings (though not identical) on the 5.36.0 build. UPDATE: The warnings regarding sv_inline.h are not present in the 5.36.0 build.
Cheers, Rob
I find (on Windows) that the problem goes away if, in perlsubs.c (which is part of the Math-FastGF2-0.07 source distro), I replace each occurrence of SvPV(Str,len) with SvPV(Str,PL_na).
Here's the actual patch I used:
--- perlsubs.c_orig 2022-07-24 11:42:59 +1000
+++ perlsubs.c 2022-07-24 14:14:14 +1000
@@ -390,7 +390,7 @@
if ( (width > 1) && byteorder &&
(native_byteorder != byteorder) ) {
- to_start=SvPV(Str,len) + width - 1;
+ to_start=SvPV(Str,PL_na) + width - 1;
for (i=width ; i-- ; --to_start, ++from_start) {
from=from_start; to=to_start;
for (j=words; j--; to += width, from += width) {
@@ -431,7 +431,7 @@
if ( (width > 1) && byteorder &&
(native_byteorder != byteorder) ) {
- to_start=SvPV(Str,len) + width - 1;
+ to_start=SvPV(Str,PL_na) + width - 1;
for (i=width ; i-- ; --to_start, ++from_start) {
from=from_start; to=to_start;
for (j=words; j--; to += width, from += width) {
@@ -461,7 +461,7 @@
if ( (width > 1) && byteorder &&
(native_byteorder != byteorder) ) {
- from_start=SvPV(Str,len) + width - 1;
+ from_start=SvPV(Str,PL_na) + width - 1;
for (i=width ; i-- ; --from_start, ++to_start) {
from=from_start; to=to_start;
for (j=words; j--; to += width, from += width) {
@@ -469,7 +469,7 @@
}
}
} else {
- from_start=SvPV(Str,len);
+ from_start=SvPV(Str,PL_na);
memcpy(to_start, from_start, len);
}
return;
@@ -499,7 +499,7 @@
if ( (width > 1) && byteorder &&
(native_byteorder != byteorder) ) {
int words;
- from_start=SvPV(Str,len) + width - 1;
+ from_start=SvPV(Str,PL_na) + width - 1;
words = len / self->width;
for (i=width ; i-- ; --from_start, ++to_start) {
from=from_start; to=to_start;
@@ -508,7 +508,7 @@
}
}
} else {
- from_start=SvPV(Str,len);
+ from_start=SvPV(Str,PL_na);
memcpy(to_start, from_start, len);
}
return;
This enables the module to build and test successfully on both perl-5.36.0 and perl-5.37.2 - and it also removes the vast majority of the warnings that were being emitted. The only warnings that remain are:
perlsubs.c: In function 'mat_offset_to_rowcol':
perlsubs.c:530:17: warning: assignment to 'int' from 'SV *' {aka 'struct sv *'}makes integer from pointer without a cast [-Wint-conversion]
530 | *row = *col = &PL_sv_undef;
| ^
FastGF2.c: In function 'XS_Math__FastGF2_gf2_pow':
FastGF2.c:253:18: warning: implicit declaration of function 'gf2_pow' [-Wimplici-function-declaration]
253 | RETVAL = gf2_pow(width, a, b);
| ^~~~~~~
Does that patch look like a correct fix ?
(Whenever I want to assign to a char* I always pre-allocate the memory using Newx or similar. Is that not needed here ?)
Cheers, Rob
SvPV (and similar) are documented that 'len' is to be STRLEN. This module declares it to be int. That was harmless before the blamed commit, but with it, it results in undefined behavior.
I can think of several options. If there is only a little bit of code that calls one of these with an improper len, then they can be fixed. If there are a bunch (which I and #irc thing is likely), then something needs to change. We can determine this by adding an assert. I think this should be the first step.
The commit could be reverted which would reintroduce the long-standing behavior of 'sv' being evaluated more than once, which the commit was attempting to fix.
The commit would be changed so that if the system had GCC brace groups, it would work properly. And if the system didn't, the per-thread variables PL_Sv and PL_na could be used, with some loss of efficiency, to achieve the single evaluation goal.
Opinions welcome
Does that patch look like a correct fix ?
SvPV_nolen() would be better since it avoids the assignment to the len parameter entirely.
(Whenever I want to assign to a
char*I always pre-allocate the memory usingNewxor similar. Is that not needed here ?)
The memory is allocated by newSVpv(), this isn't the best way to do this, but it works.
Also affected: MBAILEY/Digest-OAT-0.04.tar.gz Sample fail report: http://www.cpantesters.org/cpan/report/566f2156-0c40-11ed-8ab5-218bd7b38173
Also affected: MBAILEY/Digest-OAT-0.04.tar.gz Sample fail report: http://www.cpantesters.org/cpan/report/566f2156-0c40-11ed-8ab5-218bd7b38173
As was the case in my comment of 3 days ago, I was able to reproduce this test failure on Linux but not on FreeBSD. So there is an OS-specific aspect to this problem.
On 7/25/22 12:22, James E Keenan wrote:
Also affected: MBAILEY/Digest-OAT-0.04.tar.gz Sample fail report: http://www.cpantesters.org/cpan/report/566f2156-0c40-11ed-8ab5-218bd7b38173 <http://www.cpantesters.org/cpan/report/566f2156-0c40-11ed-8ab5-218bd7b38173>As was the case in my comment https://github.com/Perl/perl5/issues/19983#issuecomment-1192553488 of 3 days ago, I was able to reproduce this test failure on Linux but not on FreeBSD. So there is an OS-specific aspect to this problem.
It's a hardare/configuration issue. If the size of int and the size of STRLEN, the problem won't manifest itself (in the GF2 case; I haven't looked at the new one)
On 7/25/22 12:22, James E Keenan wrote: Also affected: MBAILEY/Digest-OAT-0.04.tar.gz Sample fail report: http://www.cpantesters.org/cpan/report/566f2156-0c40-11ed-8ab5-218bd7b38173 http://www.cpantesters.org/cpan/report/566f2156-0c40-11ed-8ab5-218bd7b38173 As was the case in my comment <#19983 (comment)> of 3 days ago, I was able to reproduce this test failure on Linux but not on FreeBSD. So there is an OS-specific aspect to this problem. It's a hardare/configuration issue. If the size of int and the size of STRLEN, the problem won't manifest itself (in the GF2 case; I haven't looked at the new one) …
Additional data: On FreeBSD-12, where I normally compile with clang10, I compiled a perl at blead f9757a510d28a063aaf6e6503efaeb09604b2c96 using gcc10 instead. Math-FastGF2 installed successfully (albeit with build-time warnings), but Digest-OAT failed.
[perlmonger: Digest-OAT-0.04-1] $ thisperl -V:gccversion
gccversion='10.3.0';
[perlmonger: Digest-OAT-0.04-1] $ thisprove -vb t/Digest-OAT.t
t/Digest-OAT.t ..
1..5
ok 1 - use Digest::OAT;
Failed 4/5 subtests
Test Summary Report
-------------------
t/Digest-OAT.t (Wstat: 139 (Signal: SEGV, dumped core) Tests: 1 Failed: 0)
Non-zero wait status: 139
Parse errors: Bad plan. You planned 5 tests but ran 1.
Files=1, Tests=1, 0 wallclock secs ( 0.04 usr 0.01 sys + 0.08 cusr 0.02 csys = 0.14 CPU)
Result: FAIL
So this maybe an "OS + C-compiler" issue.
So this maybe an "OS + C-compiler" issue.
The result of the bug (which the C compiler warns about) in Math-FastGF2 is undefined behaviour - this means the result can be anything, the code might execute as the author expected, or it might smash the stack, or it might whistle a tune.
It's a long standing bug in Math-GF2 which the compiler has been warning about all along.
You listed the test result of Digest::OAT - do you see type mismatch warnings when building it?
I see the same issue - there's just one instance of an int named "len" being provided as the second arg to SvPV().
The difference is that, with Digest::OAT, "len" is actually used later in the XSub, so we can't use SVPV_nolen() instead.
Changing the declaration of int to STRLEN fixes it for me.
UPDATE: Here's the actual patch I used:
--- OAT.xs_orig 2022-07-26 05:23:31 +1000
+++ OAT.xs 2022-07-26 11:16:39 +1000
@@ -11,7 +11,7 @@
unsigned oat ( SV* key ) {
/* Perl Stuff */
- int len = 0;
+ STRLEN len;
unsigned char* p = SvPV( key, len );
/* End Perl Stuff */
Cheers, Rob
You listed the test result of Digest::OAT - do you see type mismatch warnings when building it?
Re-doing the work done in the cpan .build directory:
[perlmonger: Digest-OAT-0.04-1] $ thisperl Makefile.PL
Checking if your kit is complete...
Warning: the following files are missing in your kit:
OAT.bs
OAT.c
Please inform the author.
Generating a Unix-style Makefile
Writing Makefile for Digest::OAT
Writing MYMETA.yml and MYMETA.json
[perlmonger: Digest-OAT-0.04-1] $ make
cp lib/Digest/OAT.pm blib/lib/Digest/OAT.pm
AutoSplitting blib/lib/Digest/OAT.pm (blib/lib/auto/Digest/OAT)
Running Mkbootstrap for OAT ()
chmod 644 "OAT.bs"
"/usr/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- OAT.bs blib/arch/auto/Digest/OAT/OAT.bs 644
"/usr/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/bin/perl" "/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/ExtUtils/xsubpp" -typemap '/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/ExtUtils/typemap' OAT.xs > OAT.xsc
Please specify prototyping behavior for OAT.xs (see perlxs manual)
mv OAT.xsc OAT.c
gcc10 -c -I. -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -O2 -pipe -fstack-protector -fno-strict-aliasing -DVERSION=\"0.04\" -DXS_VERSION=\"0.04\" -DPIC -fPIC "-I/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE" OAT.c
In file included from /home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE/perl.h:4477,
from OAT.xs:3:
OAT.xs: In function 'oat':
/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
1905 | Perl_SvPV_helper(aTHX_ sv, &len, flags, SvPVnormal_type_, \
/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE/sv.h:1925:37: note: in expansion of macro 'SvPV_flags'
1925 | #define SvPV(sv, len) SvPV_flags(sv, len, SV_GMAGIC)
| ^~~~~~~~~~
OAT.xs:15:24: note: in expansion of macro 'SvPV'
15 | unsigned char* p = SvPV( key, len );
| ^~~~
In file included from /home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE/perl.h:7543,
from OAT.xs:3:
/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE/sv_inline.h:879:33: note: expected 'STRLEN * const' {aka 'long unsigned int * const'} but argument is of type 'int *'
879 | STRLEN * const lp,
| ~~~~~~~~~~~~~~~^~
rm -f blib/arch/auto/Digest/OAT/OAT.so
gcc10 -shared -L/usr/local/lib -fstack-protector-strong OAT.o -o blib/arch/auto/Digest/OAT/OAT.so
chmod 755 blib/arch/auto/Digest/OAT/OAT.so
[perlmonger: Digest-OAT-0.04-1] $ make test
"/usr/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- OAT.bs blib/arch/auto/Digest/OAT/OAT.bs 644
PERL_DL_NONLAZY=1 "/usr/home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/Digest-OAT.t .. Failed 4/5 subtests
Test Summary Report
-------------------
t/Digest-OAT.t (Wstat: 139 (Signal: SEGV, dumped core) Tests: 1 Failed: 0)
Non-zero wait status: 139
Parse errors: Bad plan. You planned 5 tests but ran 1.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.06 cusr 0.03 csys = 0.12 CPU)
Result: FAIL
Failed 1/1 test programs. 0/1 subtests failed.
*** Error code 255
Stop.
make: stopped in /usr/home/jkeenan/.cpan/build/Digest-OAT-0.04-1
[perlmonger: Digest-OAT-0.04-1] $ thisprove -vb t/Digest-OAT.t
t/Digest-OAT.t ..
1..5
ok 1 - use Digest::OAT;
Failed 4/5 subtests
Test Summary Report
-------------------
t/Digest-OAT.t (Wstat: 139 (Signal: SEGV, dumped core) Tests: 1 Failed: 0)
Non-zero wait status: 139
Parse errors: Bad plan. You planned 5 tests but ran 1.
Files=1, Tests=1, 0 wallclock secs ( 0.05 usr 0.01 sys + 0.09 cusr 0.02 csys = 0.16 CPU)
Result: FAIL
Also affected: SERG/Lingua-RU-Translit-0.02.tar.gz Fail report: http://www.cpantesters.org/cpan/report/01fd58c6-0d68-11ed-a170-054335d591d0
Also affected: SERG/Lingua-RU-Translit-0.02.tar.gz Fail report: http://www.cpantesters.org/cpan/report/01fd58c6-0d68-11ed-a170-054335d591d0
This behaves similar to Digest::OAT, i.e., on FreeBSD-12 it installs successfully against a blead perl compiled with clang10 (albeit with a lot of build-time warnings), but segfaults when that perl is compiled with gcc10.
OAT.xs: In function 'oat': /home/jkeenan/testing/f9757a510d28a063aaf6e6503efaeb09604b2c96/lib/perl5/5.37.3/amd64-freebsd-thread-multi/CORE/sv.h:1905:31: warning: passing argument 3 of 'Perl_SvPV_helper' from incompatible pointer type [-Wincompatible-pointer-types]
That's a yes. From OAT.xs:
int len = 0;
unsigned char* p = SvPV( key, len );
Also affected: SERG/Lingua-RU-Translit-0.02.tar.gz
From Translit.xs:
int i, j, nrus, rslots, flen;
...
from=SvPV(string, flen);
Same bug.
Also affected: AUDREYT/Locale-Hebrew-1.05.tar.gz Fail report: http://www.cpantesters.org/cpan/report/c61c517e-0e28-11ed-af95-e2a2dbbe1b49
Also affected: IWOODHEAD/Bloom16-0.01.tar.gz Fail report: http://www.cpantesters.org/cpan/report/15858c26-0f73-11ed-b6a6-d422d4038d6b
Given the amount of CPAN breakage that has been attributed to https://github.com/Perl/perl5/commit/1ef9039bccbfe64f47f201b6cfb7d6d23e0b08a7, I think it's time for us to consider reverting this commit.
@khwilliamson, can you investigate?
On Linux, I am still getting test failures in several of the distros cited in this ticket. From my ./.cpanreporter/reports-sent.db:
test FAIL Digest-OAT-0.04 (perl-5.37.11) x86_64-linux 5.10.0-18-amd64
test FAIL Math-FastGF2-0.07 (perl-5.37.11) x86_64-linux 5.10.0-18-amd64
test FAIL Lingua-RU-Translit-0.02 (perl-5.37.11) x86_64-linux 5.10.0-18-amd64
test FAIL Locale-Hebrew-1.05 (perl-5.37.11) x86_64-linux 5.10.0-18-amd64
PL UNKNOWN Bloom16-0.01 (perl-5.37.11) x86_64-linux 5.10.0-18-amd64
(Bloom16 has prerequisite problems that prevented me from getting far enough to see whether it was failing due to the problems raised in this ticket.)
I can't reproduce this issue on macOS. Has it been fixed?
It's a bug in the downstream modules, which C compilers warn about in older perls (and does for each of these).
I doubt some of the modules will be fixed, since they haven't had anything like a recent release:
Digest::OAT: last release 2009 Lingua::RU::Translit 2002 Locale::Hebrew 2010 Bloom16 2003
(Math::FastGF2 2019, so it might get a release)
I've added comments to the ticket for each of these (and created a ticket for Bloom16) describing the cause of the problem, quoting at least one example of the problem in that module and the warning from the compiler testing under my system perl 5.32.1.
I'm not sure I think this is a blocker. This is sort of the low key breakage we expect once in a while when Perl changes its code, assuming people are doing things as specified.
@khwilliamson Thoughts?
See https://github.com/Perl/perl5/pull/20037#issuecomment-1447444692
That PR could be merged, but there is unease about doing so. As @tonycoz pointed out, the module authors ignored compiler warnings.