cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

sql/pgwire: TestAuthenticationAndHBARules failed

Open cockroach-teamcity opened this issue 1 year ago • 6 comments

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 3314a3e81d78361ba4ac45fc52a84efdfb3466ed:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:9:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence2560213530
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.11s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.02s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #126758 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1.24-rc]
  • #126239 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2]
  • #126063 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.7-rc]
/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-40580

cockroach-teamcity avatar Jul 26 '24 17:07 cockroach-teamcity

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ df2118610b0ea8cc174f19717129882e24c517b3:

        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence2438763659
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.21s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2917/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.01s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (23.91s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #128765 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.9-rc]
  • #128094 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-master]
  • #126758 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1.24-rc no-test-failure-activity]
  • #126239 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 14 '24 20:08 cockroach-teamcity

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 52fbd4aee17b0fe61dfb3badd2401fd3b1430bf2:

        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:45:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:51:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba/root (0.02s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (22.63s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/empty_hba
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba258861039
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:16:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/812/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:20:
        set_hba [0 args]
        <no input to command>
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # host  all root all cert-password # CockroachDB mandatory rule
        # loopback all all all trust       # built-in CockroachDB default
        # host     all all all cert-password # built-in CockroachDB default
        # local    all all     password      # built-in CockroachDB default
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        host     all      root all     cert-password
        loopback all      all  all     trust
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/empty_hba
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba258861039
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba (1.24s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #129181 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.10-rc]
  • #128765 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.9-rc]
  • #128094 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-master]
  • #126239 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 28 '24 20:08 cockroach-teamcity

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 2381c25c1d4b743b692b772c6777805bb06920ea:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba1152177290
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:16:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:20:
        set_hba [0 args]
        <no input to command>
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # host  all root all cert-password # CockroachDB mandatory rule
        # loopback all all all trust       # built-in CockroachDB default
        # host     all all all cert-password # built-in CockroachDB default
        # local    all all     password      # built-in CockroachDB default
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        host     all      root all     cert-password
        loopback all      all  all     trust
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/empty_hba
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba1152177290
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba (1.29s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/empty_hba/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:39:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:45:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2927/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:51:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba/root (0.02s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-sql-foundations branch-release-23.1.26-rc release-blocker]
  • #129181 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.10-rc]
  • #128765 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.9-rc]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Sep 10 '24 16:09 cockroach-teamcity

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ f5651549b20f1ac1d3e700469b1db64c8d6b5169:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2726/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:9:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2726/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence1004874535
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (6.40s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2726/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2726/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2726/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.03s)

Parameters:

  • TAGS=bazel,gss,deadlock
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #131532 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-master]
  • #131110 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.2]
  • #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.1.26-rc]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 02 '24 20:10 cockroach-teamcity

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 7002082b08ad63d1c4f8596b1ece693361268c75:

        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2934/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence880269100
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.13s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2934/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2934/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2934/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.01s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (23.09s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #131978 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.1.28-rc]
  • #131532 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-master]
  • #131110 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.2]
  • #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.1.26-rc]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 13 '24 20:10 cockroach-teamcity

[stressing here] https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_StressBazel/17381660?hideProblemsFromDependencies=false&hideTestsFromDependencies=false

souravcrl avatar Oct 19 '24 17:10 souravcrl

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 4c0c983648a468f8f9196956662175193ae645ab:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3447/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:9:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3447/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence2562826838
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.26s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3447/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:34:
        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3447/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:40:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3447/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:46:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.01s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #133360 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-24.3]
  • #132769 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.2.13-rc]
  • #131978 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.1.28-rc]
  • #131532 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-master]
  • #131110 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.2]
  • #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.1.26-rc]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 27 '24 16:10 cockroach-teamcity

@rafiss the commit for 23.1 4c0c983648a468f8f9196956662175193ae645ab had the test build PR https://github.com/cockroachdb/cockroach/commit/292e587e3011e582497ec78d4b61c1e399cce52a but the test build run logs did not have any of these added log messages: https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_StressBazel/17469114:id/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure%3Dfalse_hba_default_equivalence2562826838/pgwiretest-dev.96f48ffe8e96.roach.2024-10-27T16_07_54Z.081701.log Does this confirm that server did not explicitly send 'N' and the libpq client wrongly inferred not 'S' as 'N'? Is there something else we can do on top of this to validate?

souravcrl avatar Oct 28 '24 09:10 souravcrl

That is surprising. Could we try adding even more logs, like this? https://github.com/cockroachdb/cockroach/pull/133688

I wonder if this means the server never received the connection at all.

rafiss avatar Oct 29 '24 16:10 rafiss

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.1 @ 01f5bff3e9a800e91636444ff69b557d65153aaa:

        connect [1 args]
        <no input to command>
        ----
        ok defaultdb
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:45:
        connect_unix [1 args]
        <no input to command>
        ----
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:51:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba/root (0.01s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (23.37s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/empty_hba
    test_log_scope.go:156: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba2806366762
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:16:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/empty_hba:20:
        set_hba [0 args]
        <no input to command>
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # host  all root all cert-password # CockroachDB mandatory rule
        # loopback all all all trust       # built-in CockroachDB default
        # host     all all all cert-password # built-in CockroachDB default
        # local    all all     password      # built-in CockroachDB default
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        host     all      root all     cert-password
        loopback all      all  all     trust
        host     all      all  all     cert-password
        local    all      all          password
=== CONT  TestAuthenticationAndHBARules/insecure=false/empty_hba
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_empty_hba2806366762
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/empty_hba (1.21s)

Parameters:

  • TAGS=bazel,gss
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #133360 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-24.3]
  • #132769 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.2.13-rc]
  • #131978 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-product-security branch-release-23.1.28-rc]
  • #131532 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-master]
  • #131110 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.2]
  • #130253 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-product-security branch-release-23.1.26-rc]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Nov 12 '24 21:11 cockroach-teamcity

The latest failure has additional debug logs present.

The last debug log we see is:

I241112 21:42:24.157507 8266 sql/pgwire/pre_serve.go:459 ⋮ [T1,n1,client=‹@›] 246  client did not request SSL version=196608 AcceptSQLWithoutTLS=false and connType=‹local›

This corresponds to the test at line 45: https://github.com/cockroachdb/cockroach/blob/01f5bff3e9a800e91636444ff69b557d65153aaa/pkg/sql/pgwire/testdata/auth/empty_hba#L43-L47

However, the failure is this test at line 53: https://github.com/cockroachdb/cockroach/blob/01f5bff3e9a800e91636444ff69b557d65153aaa/pkg/sql/pgwire/testdata/auth/empty_hba#L49-L55

The latest logging patch was written so that we show a log for every possible error return path of the maybeUpgradeToSecureConn function. So that means we can conclude that the connection attempt never even reached CRDB. @souravcrl I believe that means we should go with the solution you proposed in #132729.

Thanks for your patience as we attempted to get to the bottom of this mystery. Another direction we can try is to switch away from the lib/pq driver (which is not maintained) and use pgx instead for this test.

rafiss avatar Nov 12 '24 22:11 rafiss

Hey thanks for verifying this @rafiss . There are other places we use lib/pq library. Is there a need to remove from also as under stress the error handling logic is wrong?

souravcrl avatar Nov 14 '24 07:11 souravcrl

Unfortunately, replacing the lib/pq library everywhere in our code/tests would be an enormous project. For the past couple years we have been changing specific use cases to using pgx, but there hasn't been a need to prioritize a project to make the change across the entire codebase. Instead, the approach we use is to tactically update specific tests/packages to using pgx only when there is some need to. (For example, if there are flakes, or if a test needs a driver feature that is only in pgx.)

rafiss avatar Nov 14 '24 08:11 rafiss

Based on the specified backports for linked PR #135086, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1, branch-release-24.2, branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] avatar Nov 14 '24 14:11 blathers-crl[bot]