FerretDB icon indicating copy to clipboard operation
FerretDB copied to clipboard

Add test demonstrating parallel subtests problem

Open AlekSi opened this issue 1 year ago • 2 comments

Description

I found it while working on #2330 / #2333.

Run it with env GOFLAGS='-run=TestQueryProjection' task test-integration-pg. Sometimes it passes, and sometime it fails with:

Confusing output
-> env GOFLAGS='-run=TestQueryProjection' task test-integration-pg
task: [test-integration-pg] go test -count=1 -timeout=30m -race=true -tags=ferretdb_debug,ferretdb_tigris,ferretdb_hana -shuffle=on -coverpkg=../... -coverprofile=integration-pg.txt . -target-backend=ferretdb-pg -target-tls -postgresql-url=postgres://[email protected]:5432/ferretdb -compat-url='mongodb://username:[email protected]:47018/?tls=true&tlsCertificateKeyFile=../build/certs/client.pem&tlsCaFile=../build/certs/rootCA-cert.pem'

2023-04-11T11:04:55.528+0400	INFO	setup/startup.go:68	Target system: ferretdb-pg (built-in).
2023-04-11T11:04:55.529+0400	INFO	debug	debug/debug.go:95	Starting debug server on http://127.0.0.1:60849/
2023-04-11T11:04:55.714+0400	INFO	setup/startup.go:79	Compat system: MongoDB (mongodb://username:[email protected]:47018/?tls=true&tlsCertificateKeyFile=../build/certs/client.pem&tlsCaFile=../build/certs/rootCA-cert.pem).
-test.shuffle 1681196695714473000
FAIL
	github.com/FerretDB/FerretDB/build/version	coverage: 40.0% of statements in ../...
	github.com/FerretDB/FerretDB/ferretdb	coverage: 13.9% of statements in ../...
	github.com/FerretDB/FerretDB/integration	coverage: 27.0% of statements in ../...
	github.com/FerretDB/FerretDB/integration/setup	coverage: 74.1% of statements in ../...
	github.com/FerretDB/FerretDB/integration/shareddata	coverage: 59.6% of statements in ../...
	github.com/FerretDB/FerretDB/internal/bson	coverage: 60.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/clientconn	coverage: 60.3% of statements in ../...
	github.com/FerretDB/FerretDB/internal/clientconn/conninfo	coverage: 85.7% of statements in ../...
	github.com/FerretDB/FerretDB/internal/clientconn/connmetrics	coverage: 4.7% of statements in ../...
	github.com/FerretDB/FerretDB/internal/clientconn/cursor	coverage: 6.1% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers	coverage: [no statements]
	github.com/FerretDB/FerretDB/internal/handlers/common	coverage: 9.1% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/common/aggregations	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/commonerrors	coverage: 32.4% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/dummy	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/hana	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/hana/hanadb	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/pg	coverage: 23.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/pg/pgdb	coverage: 55.7% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/pg/pjson	coverage: 56.5% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/proxy	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/registry	coverage: 50.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/tigris	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/tigris/tigrisdb	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/handlers/tigris/tjson	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/types	coverage: 21.3% of statements in ../...
	github.com/FerretDB/FerretDB/internal/types/fjson	coverage: 60.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/ctxutil	coverage: 69.2% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/debug	coverage: 80.8% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/debugbuild	coverage: 66.7% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/hex	coverage: 0.0% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/iterator	coverage: 62.8% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/lazyerrors	coverage: 81.8% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/logging	coverage: 43.2% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/must	coverage: 57.1% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/resource	coverage: 72.7% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/state	coverage: 35.9% of statements in ../...
	github.com/FerretDB/FerretDB/internal/util/testutil	coverage: 35.7% of statements in ../...
	github.com/FerretDB/FerretDB/internal/wire	coverage: 49.9% of statements in ../...
FAIL	github.com/FerretDB/FerretDB/integration	3.359s
FAIL
task: Failed to run task "test-integration-pg": exit status 1

Adding -v shows errors like:

    setup.go:246:
        	Error Trace:	/Users/aleksi/Code/FerretDB/FerretDB/integration/setup/setup.go:246
        	            				/opt/homebrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1150
        	            				/opt/homebrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1328
        	            				/opt/homebrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1545
        	Error:      	Received unexpected error:
        	            	connection(127.0.0.1:60964[-10]) socket was unexpectedly closed: EOF
        	Test:       	TestQueryProjection

So we probably use cleanup incorrectly.

Readiness checklist

  • [ ] I added/updated unit tests.
  • [ ] I added/updated integration/compatibility tests.
  • [ ] I added/updated comments and checked rendering.
  • [ ] I made spot refactorings.
  • [ ] I updated user documentation.
  • [ ] I ran task all, and it passed.
  • [ ] I ensured that PR title is good enough for the changelog.
  • [ ] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
  • [ ] I marked all done items in this checklist.

AlekSi avatar Apr 11 '23 06:04 AlekSi

Codecov Report

Merging #2393 (e9a3c90) into main (f2027f1) will increase coverage by 0.11%. The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2393      +/-   ##
==========================================
+ Coverage   64.02%   64.14%   +0.11%     
==========================================
  Files         391      391              
  Lines       18987    18987              
==========================================
+ Hits        12157    12179      +22     
+ Misses       5925     5906      -19     
+ Partials      905      902       -3     

see 8 files with indirect coverage changes

Flag Coverage Δ
integration 57.09% <ø> (+0.16%) :arrow_up:
mongodb 4.79% <ø> (ø)
pg 57.01% <ø> (+0.16%) :arrow_up:
unit 25.88% <ø> (-0.03%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

codecov[bot] avatar Apr 11 '23 07:04 codecov[bot]

@AlekSi this pull request has merge conflicts.

mergify[bot] avatar Apr 19 '23 10:04 mergify[bot]

Merged into #2564

AlekSi avatar May 04 '23 19:05 AlekSi