Random test failure: takes_rows_of_nu_value_strings_and_pipes_it_to_stdin_of_external

[sophiajt]

Describe the bug The takes_rows_of_nu_value_strings_and_pipes_it_to_stdin_of_external will occasionally fail, only to succeed if you run the test suite again. We should dig into what is causing the occasional failure.

To Reproduce Steps to reproduce the behavior:

1. Run the test suite
2. If you're "lucky", you'll see the test fail

Expected behavior All tests should pass, with no random failures

Configuration (please complete the following information):

• OS: Doesn't seem to matter, though I haven't confirmed
• Version: master
• Optional features (if any)

Add any other context about the problem here.

[amtoine]

hi there :wave:

i think this bug is back again :open_mouth:

i've seen it randomly pop up when running cargo test --workspace lately :thinking:

[amtoine]

i think this issue should probably be opened again :relieved:

[dmatos2012]

It also happened in one of the PRs, whereby closing and opening the issue fixed it recently

[amtoine]

It also happened in one of the PRs, whereby closing and opening the issue fixed it recently

same in #8189

[fdncred]

Our CI does weird things. I'm not sure why. I restart CI's all the time because of odd failures and they usually work the 2nd time but sometimes it takes more. It's getting to be a real problem. I'd guess we're approaching 75% of CIs fail for some odd reason and restarting fixes them. We'd love to have someone find out what's going on and fix it.

[amtoine]

@fdncred

Our CI does weird things. I'm not sure why. I restart CI's all the time because of odd failures and they usually work the 2nd time but sometimes it takes more. It's getting to be a real problem. I'd guess we're approaching 75% of CIs fail for some odd reason and restarting fixes them. We'd love to have someone find out what's going on and fix it.

it's not only the CI over on GitHub :open_mouth:

personnaly, i did see

failures:

---- shell::pipeline::commands::internal::takes_rows_of_nu_value_strings_and_pipes_it_to_stdin_of_external stdout ----
=== stderr

thread 'shell::pipeline::commands::internal::takes_rows_of_nu_value_strings_and_pipes_it_to_stdin_of_external' panicked at 'assertion failed: `(left == right)`
  left: `""`,
 right: `"Ecuado"`', tests/shell/pipeline/commands/internal.rs:31:9


failures:
    shell::pipeline::commands::internal::takes_rows_of_nu_value_strings_and_pipes_it_to_stdin_of_external

test result: FAILED. 384 passed; 1 failed; 18 ignored; 0 measured; 0 filtered out; finished in 26.67s

error: test failed, to rerun pass `-p nu --test main`

randomly when running cargo test --workspace locally :thinking:

[fdncred]

hmmm, I wonder why I haven't seen that 🤔

Oh, wait! I wonder if it's because the test is calling the os's echo command. I thought that was a big no-no because each systems echo is a little different, also, at least on Windows, my echo command is from the git-for-windows install. Seems like we should be using a testbin for that maybe?

I think nu --testbin cococo is the echo replacement. It would be good to update that test to that to see if it fixes the problem. Like this I think

        r#"
            open nu_times.csv
            | get origin
            | each { |it| nu --testbin cococo $it | nu --testbin chop }
            | get 2
            "#

[amtoine]

hmmm, I wonder why I haven't seen that thinking

the thing is that i can not reproduce this reliably... the error appears sometimes, disappear after a cargo clean sometime, ...

Oh, wait! I wonder if it's because the test is calling the os's echo command. I thought that was a big no-no because each systems echo is a little different, also, at least on Windows, my echo command is from the git-for-windows install. Seems like we should be using a testbin for that maybe?

mm maybe yeah but again it's strange that the error appears that randomly.

[amtoine]

I think nu --testbin cococo is the echo replacement. It would be good to update that test to that to see if it fixes the problem. Like this I think

        r#"
            open nu_times.csv
            | get origin
            | each { |it| nu --testbin cococo $it | nu --testbin chop }
            | get 2
            "#

in replacement for

        r#"
            open nu_times.csv
            | get origin
            | each { |it| ^echo $it | nu --testbin chop }
            | get 2
            "#

in tests/shell/pipeline/commands/internal.rs#L22, right?

i do not understand the source of nushell and especially these tests to be helpful...

[fdncred]

in tests/shell/pipeline/commands/internal.rs#L22, right?

Yup. That's what I'm saying.

[KoviRobi]

I wonder if there is some helpful error on the stderr, maybe the following change could print it:

diff --git a/tests/shell/pipeline/commands/internal.rs b/tests/shell/pipeline/commands/internal.rs
index 82fe1bb02..4b6d3f3d0 100644
--- a/tests/shell/pipeline/commands/internal.rs
+++ b/tests/shell/pipeline/commands/internal.rs
@@ -28,7 +28,7 @@ fn takes_rows_of_nu_value_strings_and_pipes_it_to_stdin_of_external() {
         ));

         // chop will remove the last escaped double quote from \"Estados Unidos\"
-        assert_eq!(actual.out, "Ecuado");
+        assert_eq!(actual.out, "Ecuado", "stderr: {}", actual.err);
     })
 }

[amtoine]

Yup. That's what I'm saying.

gotcha :ok_hand:

[sophiajt]

Closing as I haven't seen this issue in many months.

Please feel free to file a new issue if it comes back.