cnti-testcatalog
[BUG] prometheus_traffic SHA Mixup
Describe the bug When determining if prometheus is installed, the testsuite first checks if any prometheus is found with the correct sha. The issue with this is, when retagging and pushing that image the sha changes, This causes it to skip prometheus even if the exact image and version is found.
To Reproduce
- docker pull quay.io/prometheus/prometheus:v2.35.0
- docker inspect --format='{{index .RepoDigests 0}}' quay.io/prometheus/prometheus:v2.35.0
- docker tag quay.io/prometheus/prometheus:v2.35.0 localhost:32000/prometheus:v2.35.0
- docker push localhost:32000/prometheus:v2.35.0
- docker inspect --format='{{index .RepoDigests 0}}' localhost:32000/prometheus
From here you can see the pulled sha is 2acfab1966f0dbecc6afbead13eca7f47062cfe8726bb9db25e39e0c0b88e9c3
but after tag and push it becomes 4b86ad59abc67fa19a6e1618e936f3fd0f6ae13f49260da55a03eeca763a0fb5
Expected behavior Look at the installed Prometheus
Screenshots
LOGS
I, [2022-07-14 15:08:59 -04:00 #3632585] INFO -- cnf-testsuite: remote_sha_list: [{"name" => "main", "manifest_digest" => "sha256:ebdc21b3b2c506e65cacc3e9572d6405b97d0aca46393534e0b91108c26f5fb0"}, {"name" => "latest", "manifest_digest" => "sha256:56e7f18e05dd567f96c05046519760b356f52450c33f6e0055a110a493a41dc4"}, {"name" => "v2.37.0", "manifest_digest" => "sha256:56e7f18e05dd567f96c05046519760b356f52450c33f6e0055a110a493a41dc4"}, {"name" => "v2.37.0-rc.1", "manifest_digest" => "sha256:1c85614c947fad778bcf50b91ac9c095eced50b661dcd849a5c326959b7d7148"}, {"name" => "v2.37.0-rc.0", "manifest_digest" => "sha256:8153bb82486bfd67ffdd9c4b08de8442fc435061a163610e3f7ae499811ae965"}, {"name" => "v2.36.2", "manifest_digest" => "sha256:df0cd5887887ec393c1934c36c1977b69ef3693611932c3ddeae8b7a412059b9"}, {"name" => "v2.36.1", "manifest_digest" => "sha256:f4c5fa1018a5c15f81250b8122f36f90146d5c2b323f9ec1f41d1b36dc0e7cb9"}, {"name" => "v2.36.0", "manifest_digest" => "sha256:18dfa7c0cabaedbef241ca7cecf2ad56c5a86b10c4f583088f30c37bcef312f4"}, {"name" => "v2.36.0-rc0", "manifest_digest" => "sha256:82c17b13415109c8ce71f202d4a43f605e22077529c273c670d7af3f9ea05e9c"}, {"name" => "v2.35.0", "manifest_digest" => "sha256:2acfab1966f0dbecc6afbead13eca7f47062cfe8726bb9db25e39e0c0b88e9c3"}, {"name" => "v2.35.0-rc1", "manifest_digest" => "sha256:90c8a5e317ca535ff3f96f3d8e169bb26b0259100ac3e32946b9d785e0c6e756"}, {"name" => "v2.35.0-rc0", "manifest_digest" => "sha256:6d771ecbc65574bbd07fd9ba1fd2554bd4a1e41cf50eff04a0f93e63eb5754d0"}, {"name" => "v2.34.0", "manifest_digest" => "sha256:b37103e03399e90c9b7b1b2940894d3634915cf9df4aa2e5402bd85b4377808c"}, {"name" => "v2.34.0-rc.2", "manifest_digest" => "sha256:d9297ead7c4e8885f26be256f97b301877c52516e1784b05453f6e7895109a28"}, {"name" => "v2.34.0-rc.1", "manifest_digest" => "sha256:bb0253da4a40e37f7cd71022fe2a8eaba8ef70c2346535bde41e1e0067ff347b"}, {"name" => "v2.33.5", "manifest_digest" => "sha256:91100b06e86d724d5757e8f5a8920d1e22981067a5acf7eb2f47c3ae62be4ed2"}, {"name" => "v2.34.0-rc.0", "manifest_digest" => "sha256:be100707974bf9104a46e434f47574d224e24588da54845e1bca4836ebaf607f"}, {"name" => "v2.33.4", "manifest_digest" => "sha256:fb7e3a27469dd7e15f3b181cf510954db04b855722be94448dc23494005c433c"}, {"name" => "v2.33.3", "manifest_digest" => "sha256:20c90b9a99b12b4349150e347811cc44dccdb05c291d385320be63dc12cce73b"}, {"name" => "v2.33.2", "manifest_digest" => "sha256:bb5b8b92dee2b6ea0468c3d3ec31ce76bb644ed900e5ae74738fa8522f60d3e0"}, {"name" => "v2.33.1", "manifest_digest" => "sha256:3a75763d209af6ef82aca8fb202a76092a690d25f24d2def7f4eff64e79b7ed9"}, {"name" => "v2.33.0", "manifest_digest" => "sha256:3df0bcb08390d85c2b13d11801a2e53bdbdf3d2ddf8f8e5db740a279a7a85508"}, {"name" => "v2.33.0-rc.1", "manifest_digest" => "sha256:7d538b587655e1466e1c1985fb883a8352225ec5e611163ffe1734004fde98c5"}, {"name" => "v2.33.0-rc.0", "manifest_digest" => "sha256:9ef2593e6e6fa76d3e8a5a35527e91467bb76f2c3322dafc0b0c1d35861aade9"}, {"name" => "v2.32.1", "manifest_digest" => "sha256:cb9817249c346d6cfadebe383ed3b3cd4c540f623db40c4ca00da2ada45259bb"}, {"name" => "v2.32.0", "manifest_digest" => "sha256:68aa603f9d797a8423e766b625cab4202bda7d9be8fc44d4e904dcea7f142177"}, {"name" => "v2.31.2", "manifest_digest" => "sha256:05128da5ddc760fef7760b93da9883611426658b0dd707ee8d7886923487b472"}, {"name" => "v2.30.4", "manifest_digest" => "sha256:2b1306256b6e48ecc2a160b65401f2fc86d6085dc4677350d0739dc7c1fb3c44"}, {"name" => "v2.32.0-rc.1", "manifest_digest" => "sha256:40632d74c1f24b35d61e1c145975a2d4caaa7ad3159c819969f9044e531f061f"}, {"name" => "v2.32.0-rc.0", "manifest_digest" => "sha256:17f94684a6399d303785b42e4393190e6008655ad1e8ff305b77bec9cafbacb9"}, {"name" => "v2.32.0-beta.0", "manifest_digest" => "sha256:312c146b3e0b4f0fe8e3b6c6cfb2ae3926ee603dee72971e644a76526ca5cb96"}, {"name" => "v2.31.1", "manifest_digest" => "sha256:a8779cfe553e0331e9046268e26c539fa39ecf90d59836d828163e65e8f4fa35"}, {"name" => "v2.31.0", "manifest_digest" => "sha256:bad0c4a275e7cf46635fd215676a8d0066f73f577bb11e7de9a9a452f0b5fd97"}, {"name" => "v2.31.0-rc.1", "manifest_digest" => "sha256:b95a7115f778acd90b7c67418293b81f1c246cab19e1074276fff56f5d8d71a9"}, {"name" => "v2.31.0-rc.0", "manifest_digest" => "sha256:5fc11c67874fd9af83d4d10e871fd2e6d174afc13bea5cc25042ea95d2b82c22"}, {"name" => "v2.30.3", "manifest_digest" => "sha256:e9620d250b16ffe0eb9e3eac7dd76151848424c17d577632ae9ca61d1328687e"}, {"name" => "v2.30.2", "manifest_digest" => "sha256:c3f58c653e91a0a9c1c88e217c545b27a706a72e796fb4c463b2a4677940f12a"}, {"name" => "v2.30.1", "manifest_digest" => "sha256:88eee6112f8dff542332c7bfd4a4e8b0312949954e939c8298f084515a48097c"}, {"name" => "v2.30.0", "manifest_digest" => "sha256:58fb03aafb142d58cb03965a63a79e9233180d95ee23ca2a76f081b38d286954"}, {"name" => "v2.30.0-rc.0", "manifest_digest" => "sha256:acb276f6cc1a80377e3e10e04dcaf566d78ed0cba82e0f21f4300d3599f57e3b"}, {"name" => "v2.29.2", "manifest_digest" => "sha256:0205de008ba4fcd6d054d7375fbabcbc5239ee6919a287d89b6a5dc244a1b43e"}, {"name" => "v2.29.1", "manifest_digest" => "sha256:ccc801f38fdac43f0ed3e1b0220777e976828d6558f8ef3baad9028e0d1797ae"}, {"name" => "v2.29.0", "manifest_digest" => "sha256:8de955e47c601b024e05af1f81bc55f2ebfdb4775d49b0acad2575c7ceebb3aa"}, {"name" => "v2.29.0-rc.2", "manifest_digest" => "sha256:64f18d3fd257d0821c51872d283dfda17fb8e60d4221bdd687bbf13c501ada78"}, {"name" => "v2.29.0-rc.1", "manifest_digest" => "sha256:bc590221977a083d98ddedfe8fa4f85f39ab73ca6a7e87080997731cd205b243"}, {"name" => "v2.29.0-rc.0", "manifest_digest" => "sha256:4d9b7d636b67bea5047139fcc8365cb8e2429ab988705f8372f2cfbc2100f905"}, {"name" => "v2.28.1", "manifest_digest" => "sha256:5c030438c1e4c86bdc7428f24ee1ad18476eefdfa8a7f76a8ccc9b74f1970d81"}, {"name" => "v2.28.0", "manifest_digest" => "sha256:d5db4b724a53ec7135df90bdb908e00439522539156df2697d969c5931c673a4"}, {"name" => "v2.28.0-rc.0", "manifest_digest" => "sha256:7e2060cac38ca90a10346a431239106290c3d52c55957e82e2277f44c52ce50a"}, {"name" => "v2.27.1", "manifest_digest" => "sha256:5accb68b56ba452e449a5e552411acaeabbbe0f087acf19a1157ce3dd10a8bed"}, {"name" => "v2.26.1", "manifest_digest" => "sha256:e41817a13100572ef99858213eb6b7fa10d6e722a36e5019fc2dc75603dda842"}, {"name" => "v2.27.0", "manifest_digest" => "sha256:d1a9a86b9a3e60a9ea3cde141bdc936847456acc497e0affe7e288234383efa5"}, {"name" => "v2.27.0-rc.0", "manifest_digest" => "sha256:0dbb289d1f763e4b2d9db8adb3b4a35e36ba7049875fd73bec24cb08a619a285"}, {"name" => "v2.26.0", "manifest_digest" => "sha256:38d40a760569b1c5aec4a36e8a7f11e86299e9191b9233672a5d41296d8fa74e"}, {"name" => "v2.26.0-rc.0", "manifest_digest" => "sha256:d32e7f464223fad38273f93a2176038debcb7bc6ca0a35028bee3d527cb0253d"}, {"name" => "v2.25.2", "manifest_digest" => "sha256:c24ab4ef6187de7e79c4578e6fccb22cb73b8da662987328a828a65410fe6ae9"}, {"name" => "v2.25.1", "manifest_digest" => "sha256:1b4ed7c0e11efb8f68bbc913909d7650a46cb783d84cf541ee5e054925d6173f"}, {"name" => "master", "manifest_digest" => "sha256:981e8cce3b654c9d5bd3784771cb01d4ab54ebf4092c81dc4ecb3756e162c06a"}, {"name" => "v2.25.0", "manifest_digest" => "sha256:fd8b3c4c7ced91cbe96aa8a8dd4d02aa5aff7aefdaf0e579486127745c758c27"}, {"name" => "v2.25.0-rc.0", "manifest_digest" => "sha256:efcaa8d05fa7ccb7dd70a711a7d3efb26f0f821e485b9de9077be709dc4f425c"}, {"name" => "v2.24.1", "manifest_digest" => "sha256:ea420f6cd98e4e43e264a7a2e6e27e8328f47aa32d937e0c6e8e3b30fdefe6eb"}, {"name" => "v2.24.0", "manifest_digest" => "sha256:943c7c57115a449353e0158dcba4eaab2e56de07b7d552b5145cb6c0d1cbab19"}, {"name" => "v2.24.0-rc.0", "manifest_digest" => "sha256:73004531374bc42a62cb834cc54d7a5ed6cd4cb0ce5639c3d8137d30053bd4fa"}, {"name" => "v2.23.0", "manifest_digest" => "sha256:0eac377a90d361be9da35b469def699bcd5bb26eab8a6e9068516a9910717d58"}, {"name" => "v2.23.0-rc.0", "manifest_digest" => "sha256:aa582c8622300edb1ea14d2a8a40f3d9924cdae9334767f04448875ec15fbf67"}, {"name" => "v2.22.2", "manifest_digest" => "sha256:f7ffebdd428ba005e2c65ec1c7debdffdd39c43eb6bd0ddfa64ea7aba2a3fd14"}, {"name" => "v2.22.1", "manifest_digest" => "sha256:b899dbd1b9017b9a379f76ce5b40eead01a62762c4f2057eacef945c3c22d210"}, {"name" => "v2.22.0", "manifest_digest" => "sha256:60190123eb28250f9e013df55b7d58e04e476011911219f5cedac3c73a8b74e6"}, {"name" => "v2.22.0-rc.0", "manifest_digest" => "sha256:96c521a7b51735ee2f675a99711362d291fedcdbfda778f1f236ae9c19f1c7ee"}, {"name" => "v2.21.0", "manifest_digest" => "sha256:d43417c260e516508eed1f1d59c10c49d96bbea93eafb4955b0df3aea5908971"}, {"name" => "v2.21.0-rc.1", "manifest_digest" => "sha256:f6920b4f808d4b8b6d41516c26927f8e4a87a2970f5dadb3adc0ab5e4b06d490"}, {"name" => "v2.21.0-rc.0", "manifest_digest" => "sha256:b5981fc04e2c34fc32551b02210a78c84af5c3bc4da652ade52d5f56c36d50ca"}, {"name" => "v2.20.1", "manifest_digest" => "sha256:788260ebd13613456c168d2eed8290f119f2b6301af2507ff65908d979c66c17"}, {"name" => "v2.19.3", "manifest_digest" => "sha256:8580e76d36a27da031f7164bb3284ad6415ab0a64ae566998d42cbc5a52fa1b0"}, {"name" => "v2.20.0", "manifest_digest" => "sha256:d4ba4dd1a9ebb90916d0bfed3c204adcb118ed24546bf8dd2e6b30fc0fd2009e"}, {"name" => "v2.20.0-rc.1", "manifest_digest" => "sha256:5fa4456107ab10fae167a4b32839754cbbafdb3c24f61210d095f7f824b77800"}, {"name" => "v2.20.0-rc.0", "manifest_digest" => "sha256:ce87608048337b2f00e1969f4ba322dd40271447c14afe8c30bd1238e6d534e9"}, {"name" => "v2.19.2", "manifest_digest" => "sha256:752b7b142e06e5accb97088d9131c5d1495e45ee2eddcd80282b4de6516df697"}, {"name" => "v2.19.1", "manifest_digest" => "sha256:efe62fa8804e9fd2612a945b70c630cc27e21b5fb8233ccc8be4cfbe06d26b04"}, {"name" => "v2.19.0", "manifest_digest" => "sha256:bfad037f95e5e34d595502aa02cac6467b7eadc4b08a601d150844003051fb1b"}, {"name" => "v2.18.2", "manifest_digest" => "sha256:4d3303d1eb424e345cf48969bb7575d4d58472ad783ac41ea07fba92686f7ef5"}, {"name" => "v2.19.0-rc.0", "manifest_digest" => "sha256:401687a9c2107b93b71f88f6bc404b1b44c529a1ec6ffe3a605c788ed9666e5c"}, {"name" => "v2.18.1", "manifest_digest" => "sha256:5880ec936055fad18ccee798d2a63f64ed85bd28e8e0af17c6923a090b686c3d"}, {"name" => "v2.18.0", "manifest_digest" => "sha256:e0737e9c2a5bfe8957a0f340124c9c9d4ec78372902ee56f670d55a3ea73cf4b"}, {"name" => "v2.18.0-rc.1", "manifest_digest" => "sha256:916d17bc116684f80d440cc761ffa688fe9bd6a98df86c8ded74d598c1c4aa6f"}, {"name" => "v2.18.0-rc.0", "manifest_digest" => "sha256:2b98de77cfaad5674f503cd0b235254f4cdf88f90adfe2b7558d2b4ac22509a2"}, {"name" => "v2.17.2", "manifest_digest" => "sha256:42d2395cd7192cac9c787b517c4c276a184e5512da793cce9db8ee626f2cbde1"}, {"name" => "v2.17.1", "manifest_digest" => "sha256:0d05276441fc3ec287025db43190873fd0088d1b16a45a0012a59f4cbd6e5be9"}, {"name" => "v2.17.0", "manifest_digest" => "sha256:2a3fd0d34a6a09a664cf72013a29670a5fa4f9b24e7683fafce89fa4c63d305a"}, {"name" => "v2.17.0-rc.4", "manifest_digest" => "sha256:1236825d0347e9d27ab12d9643a1ee6abcb801dda050293d18ee3f7a5fdc4633"}, {"name" => "v2.17.0-rc.3", "manifest_digest" => "sha256:908fc6eb39a21128100abd3aaf8b8565121e1107e0144ac728cd4131fb40ae13"}, {"name" => "v2.17.0-rc.1", "manifest_digest" => "sha256:07a6a2d6aadf95a6b3b452cd643b6ec4ea66124e6807e58516c445a99ffd448e"}, {"name" => "v2.17.0-rc.0", "manifest_digest" => "sha256:054acb89b61d0fc7a7eb7ee1c1da5ba51db9c1f672d2316a8526f6a92c7fbe8e"}, {"name" => "v2.16.0", "manifest_digest" => "sha256:e4ca62c0d62f3e886e684806dfe9d4e0cda60d54986898173c1083856cfda0f4"}, {"name" => "v2.16.0-rc.1", "manifest_digest" => "sha256:21480647ec721eebdc897caffef95bf34f8a1b0babfbd3f956e316e55075b1a1"}, {"name" => "v2.16.0-rc.0", "manifest_digest" => "sha256:722f45fe02bd0c6d74e0d9f8e1fb58356a5ae459464d5130c9792a659473973a"}, {"name" => "v2.15.2", "manifest_digest" => "sha256:914525123cf76a15a6aaeac069fcb445ce8fb125113d1bc5b15854bc1e8b6353"}, {"name" => "v2.15.1", "manifest_digest" => "sha256:169b743ceb4452266915272f9c3409d36972e41cb52f3f28644e6c0609fc54e6"}, {"name" => "v2.15.0", "manifest_digest" => "sha256:894a9e56e389d37db06bdbac8a1024500dd3a31a3fe7c4a8a1651e184fa8c261"}, {"name" => "v2.15.0-rc.0", "manifest_digest" => "sha256:06be903663baa97698e70d6a29d2375b99828c211c582f2fdb6c93c3285b044f"}] I, [2022-07-14 15:08:59 -04:00 #3632585] INFO -- cnf-testsuite: local_digests: ["docker.io/conformance/cluster-tools@sha256:e015da5f2771d1529af6987f244edb5347ffe393e2fc26eae44affd395cf7bc2", "sha256:bfe3a36ebd2528b454be6aebece806db5b40407b833e2af9617bf39afaff8c16", "sha256:0369cf4303ffdb467dc219990960a9baa8512a54b0ad9283eaf55bd6c0adb934", "sha256:6fb66cd78abfe9e0735a9a751f2586b7984e0d279e87fa8dd175781de6595627", "docker.io/library/import-2022-05-19@sha256:c37ad6940632d8bcbf52df4ce5e8724343b154b0c6a1def65d6f1e2137bdc5d7", "docker.io/library/import-2022-05-19@sha256:8c0e8f18ad72ccc5a2dbb7c0035fccca03a940344c89da11240107ddadc58e41", "docker.io/library/import-2022-05-19@sha256:7962cd2a34ae337f56bc841a5c0cf896ec116a721ca983bbefc302c32cd5c1bb", "docker.io/library/import-2022-05-19@sha256:a0da6981c92f5499921ab1bf71a1faed8366bdffb65816ca9fdd31a2a8d92d2f", "sha256:4c1e997385b8fb4ad4d1d3c7e5af7ff3f882e94d07cf5b78de9e889bc60830e6", "localhost:32000/grafana/promtail@sha256:575daa3096c5cb6576c0ae5801b92109d4852537a0ca1b414ba044098e51b2bc", "localhost:32000/prometheus-operator@sha256:00a499185c862b85351c4a74630bbe95dc544db3afb106573ceb1d7a9fab58ce", "localhost:32000/kube-state-metrics@sha256:b6913dc5904467291b59880dfd306c9079b7d0900452c558f4cd46628cb764c5", "localhost:32000/prometheus-config-reloader@sha256:04785e8dd1bf560c82e541add88ccb78394925e633be4e4dc0f2b35598949171", "localhost:32000/prometheus@sha256:4b86ad59abc67fa19a6e1618e936f3fd0f6ae13f49260da55a03eeca763a0fb5", "localhost:32000/node-exporter@sha256:d5b2a2e2bb07a4a5a7c4bd9e54641cab63e1d2627622dbde17efc04849d3d30d"]
{ "mine": [ "localhost:32000/prometheus@sha256:4b86ad59abc67fa19a6e1618e936f3fd0f6ae13f49260da55a03eeca763a0fb5", ], "remote": [ { "name" => "v2.35.0", "manifest_digest" => "sha256:2acfab1966f0dbecc6afbead13eca7f47062cfe8726bb9db25e39e0c0b88e9c3" } ] }
TIL that re-tagging an image changes the contents of the image, so the SHA of the image does change even if the action is simple as re-tagging and pushing to a local registry.
The solution I could think of here is to provide a configuration in the cnf-testsuite.yml to allow using a custom registry URL to verify image SHAs for third-party tools the testsuite checks for.
Will post more details after exploring.
Sounds good, also not sure but is there a way to check if Prometheus is running without checking image SHAs? For example calling an endpoint that should be up if Prometheus is running? (this could be configured in the cnf-testsuite)
I intended the custom registry option to support anyone using an alternate/mirror registry for any third-party tool we use in the testsuite. But I found out yesterday that registry APIs are non-standard (For example: DockerHub's API is different from GitLab's. So fetching SHAs would have to be different for each API)
Looks like implementing a prometheus url config is the simplest way to go. Might have to do this other third-party tools if required later.
Update: Resolution for this is being handled as a part of Issue-1581 / Issue-1580.
This is pending https://github.com/cncf/cnf-testsuite/issues/1591 and https://github.com/cncf/cnf-testsuite/issues/1581 which are currently "In Progress"
AC and Review will be in https://github.com/cncf/cnf-testsuite/issues/1581, no AC required for this one, marking as Reviewer Approved.