buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard

Published 20 hours ago • verified publisher icon moby

WCOW/tests: revisit skipped integration tests to run on Windows where applicable

Open profnandaa opened this issue 1 year ago • 4 comments

I'm opening this issue as a tracker for the explicitly skipped integration tests on Windows. Some should be skipped by design but others need to be enabled on Windows too.

Here is the list (might keep growing as new tests come in):

Script 
  $ grep -r -B 1 ".SkipOnPlatform(" | grep -v "^--" | grep -v "integration.SkipOn" |  sed 's/-/ /' | sed 's/^/- [ ] /'

/client:

  • [ ] client/build_test.go testClientGatewayEmptyImageExec
  • [ ] client/client_test.go testCacheExportCacheKeyLoop
  • [ ] client/client_test.go testBridgeNetworkingDNSNoRootless
  • [ ] client/client_test.go testExportedImageLabels
  • [ ] client/client_test.go testExportBusyboxLocal
  • [ ] client/client_test.go testHostnameLookup
  • [ ] client/client_test.go testHostnameSpecifying
  • [ ] client/client_test.go testStdinClosed
  • [ ] client/client_test.go testSSHMount
  • [ ] client/client_test.go testExtraHosts
  • [ ] client/client_test.go testShmSize
  • [ ] client/client_test.go testUlimit
  • [ ] client/client_test.go testNetworkMode
  • [ ] client/client_test.go testSecurityMode
  • [ ] client/client_test.go testSecurityModeSysfs
  • [ ] client/client_test.go testSecurityModeErrors
  • [ ] client/client_test.go testSecretMounts
  • [ ] client/client_test.go testSecretEnv
  • [ ] client/client_test.go testBuildHTTPSource
  • [ ] client/client_test.go testSourceDateEpochImageExporter
  • [ ] client/client_test.go testBuildExportWithForeignLayer
  • [ ] client/client_test.go testBuildExportZstd
  • [ ] client/client_test.go testPullZstdImage
  • [ ] client/client_test.go testZstdLocalCacheExport
  • [ ] client/client_test.go testCacheExportIgnoreError
  • [ ] client/client_test.go testBasicS3CacheImportExport
  • [ ] client/client_test.go testBasicAzblobCacheImportExport
  • [ ] client/client_test.go testSnapshotWithMultipleBlobs
  • [ ] client/client_test.go testExportLocalNoPlatformSplit
  • [ ] client/client_test.go testMoveParentDir
  • [ ] client/client_test.go testSchema1Image
  • [ ] client/client_test.go testMountWithNoSource
  • [ ] client/client_test.go testReadonlyRootFS
  • [ ] client/client_test.go testRmSymlink
  • [ ] client/client_test.go testProxyEnv
  • [ ] client/client_test.go testParallelLocalBuilds
  • [ ] client/client_test.go testMultipleCacheExports
  • [ ] client/client_test.go testMountStubsDirectory
  • [ ] client/client_test.go testMountStubsTimestamp
  • [ ] client/client_test.go testLLBMountPerformance

/cmd/buildctl:

  • [ ] cmd/buildctl/buildctl_test.go testUsage
  • [ ] cmd/buildctl/build_test.go testBuildWithLocalFiles
  • [ ] cmd/buildctl/build_test.go testBuildLocalExporter
  • [ ] cmd/buildctl/build_test.go testBuildContainerdExporter
  • [ ] cmd/buildctl/build_test.go testBuildMetadataFile
  • [ ] cmd/buildctl/diskusage_test.go testDiskUsage
  • [ ] cmd/buildctl/prune_test.go testPrune

/frontend:

  • [ ] frontend/dockerfile/dockerfile_addchecksum_test.go testAddChecksum
  • [ ] frontend/dockerfile/dockerfile_addgit_test.go testAddGit
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testCopyHeredoc
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testCopyHeredocSpecialSymbols
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testRunBasicHeredoc
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testRunFakeHeredoc
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testRunShebangHeredoc
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testRunComplexHeredoc
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testHeredocIndent
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testHeredocVarSubstitution
  • [ ] frontend/dockerfile/dockerfile_heredoc_test.go testOnBuildHeredoc
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountContext
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountTmpfs
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountInvalid
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountRWCache
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testCacheMountUser
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testCacheMountDefaultID
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountEnvVar
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountArg
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountEnvAcrossStages
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountMetaArg
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountFromError
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountTmpfsSize
  • [ ] frontend/dockerfile/dockerfile_mount_test.go testMountDuplicate
  • [ ] frontend/dockerfile/dockerfile_outline_test.go testOutlineArgs
  • [ ] frontend/dockerfile/dockerfile_outline_test.go testOutlineSecrets
  • [ ] frontend/dockerfile/dockerfile_outline_test.go testOutlineDescribeDefinition
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testProvenanceAttestation
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testGitProvenanceAttestation
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testMultiPlatformProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testClientFrontendProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testClientLLBProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testSecretSSHProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testOCILayoutProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testNilProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testDuplicatePlatformProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testDockerIgnoreMissingProvenance
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testFrontendDeduplicateSources
  • [ ] frontend/dockerfile/dockerfile_provenance_test.go testDuplicateLayersProvenance
  • [ ] frontend/dockerfile/dockerfile_runnetwork_test.go testRunGlobalNetwork
  • [ ] frontend/dockerfile/dockerfile_secrets_test.go testSecretFileParams
  • [ ] frontend/dockerfile/dockerfile_secrets_test.go testSecretRequiredWithoutValue
  • [ ] frontend/dockerfile/dockerfile_ssh_test.go testSSHSocketParams
  • [ ] frontend/dockerfile/dockerfile_ssh_test.go testSSHFileDescriptorsClosed
  • [ ] frontend/dockerfile/dockerfile_targets_test.go testTargetsList
  • [ ] frontend/dockerfile/dockerfile_targets_test.go testTargetsDescribeDefinition
  • [ ] frontend/dockerfile/dockerfile_test.go testDefaultEnvWithArgs
  • [x] frontend/dockerfile/dockerfile_test.go testEnvEmptyFormatting
  • [x] frontend/dockerfile/dockerfile_test.go testDockerignoreOverride
  • [x] frontend/dockerfile/dockerfile_test.go testEmptyDestDir
  • [ ] frontend/dockerfile/dockerfile_test.go testExportCacheLoop
  • [ ] frontend/dockerfile/dockerfile_test.go testTarExporter
  • [ ] frontend/dockerfile/dockerfile_test.go testWorkdirCreatesDir
  • [ ] frontend/dockerfile/dockerfile_test.go testCacheReleased
  • [ ] frontend/dockerfile/dockerfile_test.go testSymlinkedDockerfile
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyChownExistingDir
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyWildcardCache
  • [ ] frontend/dockerfile/dockerfile_test.go testEmptyWildcard
  • [ ] frontend/dockerfile/dockerfile_test.go testWorkdirUser
  • [ ] frontend/dockerfile/dockerfile_test.go testWorkdirCopyIgnoreRelative
  • [ ] frontend/dockerfile/dockerfile_test.go testWorkdirExists
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyChownCreateDest
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyThroughSymlinkContext
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyThroughSymlinkMultiStage
  • [ ] frontend/dockerfile/dockerfile_test.go testCopySocket
  • [ ] frontend/dockerfile/dockerfile_test.go testIgnoreEntrypoint
  • [ ] frontend/dockerfile/dockerfile_test.go testQuotedMetaArgs
  • [ ] frontend/dockerfile/dockerfile_test.go testMultiArgs
  • [ ] frontend/dockerfile/dockerfile_test.go testDefaultShellAndPath
  • [ ] frontend/dockerfile/dockerfile_test.go testExportMultiPlatform
  • [ ] frontend/dockerfile/dockerfile_test.go testContextChangeDirToFile
  • [ ] frontend/dockerfile/dockerfile_test.go testNoSnapshotLeak
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyFollowAllSymlinks
  • [ ] frontend/dockerfile/dockerfile_test.go testCopySymlinks
  • [ ] frontend/dockerfile/dockerfile_test.go testHTTPDockerfile
  • [ ] frontend/dockerfile/dockerfile_test.go testCmdShell
  • [ ] frontend/dockerfile/dockerfile_test.go testPullScratch
  • [ ] frontend/dockerfile/dockerfile_test.go testGlobalArg
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileDirs
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileInvalidCommand
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileInvalidInstruction
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileADDFromURL
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileAddArchive
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileAddArchiveWildcard
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileAddChownExpand
  • [ ] frontend/dockerfile/dockerfile_test.go testSymlinkDestination
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileScratchConfig
  • [ ] frontend/dockerfile/dockerfile_test.go testExposeExpansion
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerignore
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerignoreInvalid
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileLowercase
  • [ ] frontend/dockerfile/dockerfile_test.go testExportedHistory
  • [ ] frontend/dockerfile/dockerfile_test.go testUser
  • [ ] frontend/dockerfile/dockerfile_test.go testUserAdditionalGids
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyChmod
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyOverrideFiles
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyVarSubstitution
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyWildcards
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyRelative
  • [ ] frontend/dockerfile/dockerfile_test.go testAddURLChmod
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileFromGit
  • [ ] frontend/dockerfile/dockerfile_test.go testDockerfileFromHTTP
  • [ ] frontend/dockerfile/dockerfile_test.go testMultiStageImplicitFrom
  • [ ] frontend/dockerfile/dockerfile_test.go testMultiStageCaseInsensitive
  • [ ] frontend/dockerfile/dockerfile_test.go testLabels
  • [ ] frontend/dockerfile/dockerfile_test.go testWildcardRenameCache
  • [ ] frontend/dockerfile/dockerfile_test.go testOnBuildCleared
  • [ ] frontend/dockerfile/dockerfile_test.go testCacheMultiPlatformImportExport
  • [ ] frontend/dockerfile/dockerfile_test.go testImageManifestCacheImportExport
  • [ ] frontend/dockerfile/dockerfile_test.go testCacheImportExport
  • [ ] frontend/dockerfile/dockerfile_test.go testReproducibleIDs
  • [ ] frontend/dockerfile/dockerfile_test.go testImportExportReproducibleIDs
  • [ ] frontend/dockerfile/dockerfile_test.go testNoCache
  • [ ] frontend/dockerfile/dockerfile_test.go testPlatformArgsImplicit
  • [ ] frontend/dockerfile/dockerfile_test.go testPlatformArgsExplicit
  • [ ] frontend/dockerfile/dockerfile_test.go testBuiltinArgs
  • [ ] frontend/dockerfile/dockerfile_test.go testTarContext
  • [ ] frontend/dockerfile/dockerfile_test.go testTarContextExternalDockerfile
  • [ ] frontend/dockerfile/dockerfile_test.go testFrontendUseForwardedSolveResults
  • [ ] frontend/dockerfile/dockerfile_test.go testFrontendInputs
  • [ ] frontend/dockerfile/dockerfile_test.go testFrontendSubrequests
  • [ ] frontend/dockerfile/dockerfile_test.go testDockefileCheckHostname
  • [ ] frontend/dockerfile/dockerfile_test.go testShmSize
  • [ ] frontend/dockerfile/dockerfile_test.go testUlimit
  • [ ] frontend/dockerfile/dockerfile_test.go testCgroupParent
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedImageContext
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedImageContextPlatform
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedImageContextTimestamps
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedImageContextScratch
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedLocalContext
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedOCILayoutContext
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedOCILayoutContextExport
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedInputContext
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedMultiplatformInputContext
  • [ ] frontend/dockerfile/dockerfile_test.go testNamedFilteredContext
  • [ ] frontend/dockerfile/dockerfile_test.go testSourceDateEpochWithoutExporter
  • [ ] frontend/dockerfile/dockerfile_test.go testSBOMScannerImage
  • [ ] frontend/dockerfile/dockerfile_test.go testSBOMScannerArgs
  • [ ] frontend/dockerfile/dockerfile_test.go testMultiPlatformWarnings
  • [ ] frontend/dockerfile/dockerfile_test.go testReproSourceDateEpoch
  • [ ] frontend/dockerfile/dockerfile_test.go testCopyUnicodePath
  • [ ] frontend/dockerfile/errors_test.go testErrorsSourceMap
  • [ ] frontend/frontend_test.go testRefReadFile **
  • [ ] frontend/frontend_test.go testRefReadDir **
  • [ ] frontend/frontend_test.go testRefStatFile **
  • [ ] frontend/frontend_test.go testRefEvaluate **

/solver:

  • [ ] solver/jobs_test.go testParallelism

Previously Skipped Tests:

  • [ ] cache/contenthash/checksum_test.go createRef
  • [ ] cache/manager_test.go TestSnapshotExtract
  • [ ] cache/manager_test.go TestExtractOnMutable
  • [ ] cache/manager_test.go TestMergeOp
  • [ ] cache/manager_test.go TestDiffOp
  • [ ] source/git/source_test.go testRepeatedFetch
  • [ ] source/git/source_test.go testFetchBySHA
  • [ ] source/git/source_test.go testFetchByTag
  • [ ] source/git/source_test.go testMultipleRepos
  • [ ] source/git/source_test.go TestCredentialRedaction
  • [ ] source/git/source_test.go testSubdir
  • [x] source/http/source_test.go TestHTTPSource
  • [x] source/http/source_test.go TestHTTPChecksum
  • [x] source/http/source_test.go TestHTTPDefaultName

Misc. Tasks

  • [ ] cross comp container/distribution binaries to avoid drifting like freebsd one and most of this logic should be moved to https://github.com/moby/buildkit/blob/master/.github/workflows/.test.yml where we are self-contained and run integration tests through a WCOW container. ref: https://github.com/moby/buildkit/pull/4479#pullrequestreview-1781259322
  • [ ] adding the --parallel option for gotestsum, re: https://github.com/moby/buildkit/pull/5027#issuecomment-2242962544
  • [ ] support multiple architecture for code image mirror logic in util/testutil/integration/run_windows.go

profnandaa avatar Dec 14 '23 07:12 profnandaa

/cc. @gabriel-samfira

profnandaa avatar Dec 14 '23 07:12 profnandaa

@profnandaa Look at some of these cases as well https://github.com/moby/buildkit/blame/master/cache/manager_test.go#L577 in cache and source packages. Can some of these skips be removed now?

tonistiigi avatar Dec 14 '23 07:12 tonistiigi

@tonistiigi -- sure, let me rake through the previously skipped cases too, I'd missed that.

profnandaa avatar Dec 14 '23 07:12 profnandaa

Found 13 tests that were previously skipped, I've added them at the end of the list.

profnandaa avatar Dec 14 '23 09:12 profnandaa