Intermittent STDERR test failures

[ilmari]

On my employer's Jenkins build nodes (CentOS 5), I'm seeing intermittend failures of STDERR-related tests. Some examples:

#   Failed test 'tee_merged|perl|both|multiline - tee STDOUT (STDOUT)'
#   at t/lib/Cases.pm line 240.
#                   'STDERR:First line
# Second line
# '
#     doesn't match '(?^:STDOUT\:First\ line\
# Second\ line\
# )'
# Looks like you failed 1 test of 873.
t/14-stderr-tied.t ......... 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/873 subtests 

#   Failed test 'tee_merged|perl|both|unicode - tee STDOUT (STDOUT)'
#   at t/lib/Cases.pm line 240.
#                   'STDERR:Hi! ☺
# '
#     doesn't match '(?^u:STDOUT\:Hi\!\ \☺\
# )'
# Looks like you failed 1 test of 873.
t/11-stderr-string.t ....... 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/873 subtests 

The t/00-report-prereqs.t output for both runs was:

# 
# Versions for all modules listed in MYMETA.json (including optional ones):
# 
# === Configure Requires ===
# 
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker 6.17 6.66
# 
# === Build Requires ===
# 
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker  any 6.66
# 
# === Test Requires ===
# 
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker  any 6.66
#     File::Spec           any 3.40
#     IO::File             any 1.16
#     Test::More          0.62 0.98
#     lib                  any 0.63
# 
# === Test Recommends ===
# 
#     Module         Want     Have
#     ---------- -------- --------
#     CPAN::Meta 2.120900 2.120921
#     Inline::C      0.50  missing
# 
# === Runtime Requires ===
# 
#     Module       Want Have
#     ------------ ---- ----
#     Carp          any 1.29
#     Exporter      any 5.68
#     File::Spec    any 3.40
#     File::Temp    any 0.23
#     IO::Handle    any 1.34
#     Scalar::Util  any 1.27
#     strict        any 1.07
#     warnings      any 1.18

I haven't been able to reproduce it outside Jenkins yet, so I can't give any more detail information.

[dhorne]

I get the same running under Perl 5:18 on Mac OS X 10.10

[dhorne]

Hmm, but not when I run the install again. Note - in the first instance I was installing perlbrew from the CPAN shell. I then tried installing just Capture::Tiny and it worked

[jandubois]

FWIW, I've also just got this on an automated build with 5.18 on Solaris:

# 
# Versions for all modules listed in MYMETA.json (including optional ones):
# 
# === Configure Requires ===
# 
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker 6.17 6.66
# 
# === Build Requires ===
# 
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker  any 6.66
# 
# === Test Requires ===
# 
#     Module              Want     Have
#     ------------------- ---- --------
#     ExtUtils::MakeMaker  any     6.66
#     File::Spec           any     3.40
#     IO::File             any     1.16
#     Test::More          0.62 1.001003
#     lib                  any     0.63
# 
# === Test Recommends ===
# 
#     Module         Want     Have
#     ---------- -------- --------
#     CPAN::Meta 2.120900 2.132830
#     Inline::C      0.50  missing
# 
# === Runtime Requires ===
# 
#     Module       Want Have
#     ------------ ---- ----
#     Carp          any 1.29
#     Exporter      any 5.68
#     File::Spec    any 3.40
#     File::Temp    any 0.23
#     IO::Handle    any 1.34
#     Scalar::Util  any 1.27
#     strict        any 1.07
#     warnings      any 1.18
# 
t/00-report-prereqs.t ...... ok
t/01-Capture-Tiny.t ........ ok
t/02-capture.t ............. ok
t/03-tee.t ................. ok
t/06-stdout-closed.t ....... ok
t/07-stderr-closed.t ....... ok
t/08-stdin-closed.t ........ ok
t/09-preserve-exit-code.t .. ok

#   Failed test 'tee_merged|perl|both|unicode - tee STDOUT (STDERR)'
#   at t/lib/Cases.pm line 241.
#                   'STDOUT:Hi! ☺
# '
#     doesn't match '(?^u:STDERR\:Hi\!\ \☺\
# )'
# Looks like you failed 1 test of 873.
t/10-stdout-string.t ....... 

It isn't deterministic, as the tests passed fine on an identical build the day before (well, building against an older OpenSSL, but otherwise the same config and same machine).