balena-engine icon indicating copy to clipboard operation
balena-engine copied to clipboard

Regressions in integration tests since the 20.10 upgrade

Open lmbarros opened this issue 2 years ago • 4 comments

Several of "CLI integration tests" are failing since we upgraded to 20.10 (this PR ~~this PR, and specifically this commit~~).

We must investigate and fix these failures. I looked at just a few of them, which are related with building images and seem like legit issues we should look into.

Here's a list of failures I get on my laptop when running TIMEOUT=240m make TEST_INTEGRATION_DIR=./integration-cli/ test-integration:

FAIL: TestDockerSuite/TestBuildAddFileNotFound
FAIL: TestDockerSuite/TestBuildAddRemoteFileMTime
FAIL: TestDockerSuite/TestBuildBuildTimeArg
FAIL: TestDockerSuite/TestBuildBuildTimeArgDefaultOverride
FAIL: TestDockerSuite/TestBuildBuildTimeArgDefinitionWithNoEnvInjection
FAIL: TestDockerSuite/TestBuildBuildTimeArgEnv
FAIL: TestDockerSuite/TestBuildBuildTimeArgHistory
FAIL: TestDockerSuite/TestBuildBuildTimeArgOverrideArgDefinedBeforeEnv
FAIL: TestDockerSuite/TestBuildBuildTimeArgOverrideEnvDefinedBeforeArg
FAIL: TestDockerSuite/TestBuildBuildTimeArgUnconsumedArg
FAIL: TestDockerSuite/TestBuildCacheFrom
FAIL: TestDockerSuite/TestBuildCacheFromEqualDiffIDsLength
FAIL: TestDockerSuite/TestBuildClearCmd
FAIL: TestDockerSuite/TestBuildContainerWithCgroupParent
FAIL: TestDockerSuite/TestBuildCopyWildcardCache
FAIL: TestDockerSuite/TestBuildEmptyScratch
FAIL: TestDockerSuite/TestBuildFails
FAIL: TestDockerSuite/TestBuildFromGitWithContext
FAIL: TestDockerSuite/TestBuildFromGitWithF
FAIL: TestDockerSuite/TestBuildFromMixedcaseDockerfile
FAIL: TestDockerSuite/TestBuildFromStdinWithF
FAIL: TestDockerSuite/TestBuildFromURLWithF
FAIL: TestDockerSuite/TestBuildIntermediateTarget
FAIL: TestDockerSuite/TestBuildJSONEmptyRun
FAIL: TestDockerSuite/TestBuildMultiStageArg
FAIL: TestDockerSuite/TestBuildMultiStageCache
FAIL: TestDockerSuite/TestBuildMultiStageCopyFromErrors
FAIL: TestDockerSuite/TestBuildMultiStageCopyFromSyntax
FAIL: TestDockerSuite/TestBuildMultiStageGlobalArg
FAIL: TestDockerSuite/TestBuildMultiStageResetScratch
FAIL: TestDockerSuite/TestBuildMultiStageUnusedArg
FAIL: TestDockerSuite/TestBuildNetContainer
FAIL: TestDockerSuite/TestBuildNetNone
FAIL: TestDockerSuite/TestBuildNoDupOutput
FAIL: TestDockerSuite/TestBuildNotVerboseFailure
FAIL: TestDockerSuite/TestBuildNotVerboseFailureWithNonExistImage
FAIL: TestDockerSuite/TestBuildOnBuildLimitedInheritance
FAIL: TestDockerSuite/TestBuildOnBuildOutput
FAIL: TestDockerSuite/TestBuildRUNErrMsg
FAIL: TestDockerSuite/TestBuildRUNoneJSON
FAIL: TestDockerSuite/TestBuildResourceConstraintsAreUsed
FAIL: TestDockerSuite/TestBuildRm
FAIL: TestDockerSuite/TestBuildSameDockerfileWithAndWithoutCache
FAIL: TestDockerSuite/TestBuildShellInherited
FAIL: TestDockerSuite/TestBuildShellUpdatesConfig
FAIL: TestDockerSuite/TestBuildSpacesWithQuotes
FAIL: TestDockerSuite/TestBuildStartsFromOne
FAIL: TestDockerSuite/TestBuildStderr
FAIL: TestDockerSuite/TestBuildStepsWithProgress
FAIL: TestDockerSuite/TestBuildVerboseOut
FAIL: TestDockerSuite/TestBuildVolumeFileExistsinContainer
FAIL: TestDockerSuite/TestBuildWithFailure
FAIL: TestDockerSuite/TestBuildWithInaccessibleFilesInContext
FAIL: TestDockerSuite/TestBuildWithTabs
FAIL: TestDockerSuite/TestBuildWorkdirCmd
FAIL: TestDockerSuite/TestConflictContainerNetworkAndLinks
FAIL: TestDockerSuite/TestConflictContainerNetworkHostAndLinks
FAIL: TestDockerSuite/TestEventsFilterType
FAIL: TestDockerSuite/TestExecStateCleanup
FAIL: TestDockerSuite/TestImagesFilterSinceAndBefore
FAIL: TestDockerSuite/TestImagesOrderedByCreationDate
FAIL: TestDockerSuite/TestLinksPingLinkedContainers
FAIL: TestDockerSuite/TestPortHostBinding
FAIL: TestDockerSuite/TestPortList
FAIL: TestDockerSuite/TestPsListContainersFilterAncestorImage
FAIL: TestDockerSuite/TestRmiParentImageFail
FAIL: TestDockerSuite/TestRmiTag
FAIL: TestDockerSuite/TestRmiTagWithExistingContainers
FAIL: TestDockerSuite/TestRmiUntagHistoryLayer
FAIL: TestDockerSuite/TestRunTLSVerify
FAIL: TestDockerSuite/TestRunTmpfsMountsOverrideImageVolumes
FAIL: TestDockerSuite/TestRunWithNetAliasOnDefaultNetworks
FAIL: TestDockerSuite/TestUserDefinedNetworkConnectDisconnectAlias
FAIL: TestDockerSuite/TestVolumeCLINoArgs
FAIL: TestDockerRegistrySuite/TestBuildByDigest
FAIL: TestDockerRegistrySuite/TestBuildMultiStageImplicitPull
FAIL: TestDockerRegistrySuite/TestPsListContainersFilterAncestorImageByDigest
FAIL: TestDockerRegistrySuite/TestPushMultipleTags
FAIL: TestDockerSchema1RegistrySuite/TestPullIDStability
FAIL: TestDockerSchema1RegistrySuite/TestPushMultipleTags
FAIL: TestDockerRegistryAuthHtpasswdSuite/TestBuildFromAuthenticatedRegistry
FAIL: TestDockerRegistryAuthHtpasswdSuite/TestBuildWithExternalAuth
FAIL: TestDockerPluginSuite/TestPluginIDPrefix

lmbarros avatar Nov 01 '22 14:11 lmbarros