Deleting s3 object NotFound errors

[r4f4]

/kind bug

What steps did you take and what happened: If the bootstrap data s3 object is deleted, further attempts to delete it will cause "NotFound" errors.

What did you expect to happen: NotFound errors are ignored since the object was already deleted. I confirmed in the AWS console that the openshift-bootstrap-data-rdossant-installer-03-jjf6b bucket contained no objects.

Anything else you would like to add: Here are the relevant lines from the log output:

I0312 18:00:13.479037     109 s3.go:220] "Deleting S3 object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-1" reconcileID="575c2ec1-a75a-4451-bbd5-dc80f62df49d" machine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" cluster="openshift-cluster-api-guests/rdossant-installer-03-jjf6b" bucket="openshift-bootstrap-data-rdossant-installer-03-jjf6b" key="control-plane/rdossant-installer-03-jjf6b-master-1"
I0312 18:00:13.602972     109 s3.go:220] "Deleting S3 object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-2" reconcileID="9cda22be-5acd-4670-840f-8a6708437385" machine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" cluster="openshift-cluster-api-guests/rdossant-installer-03-jjf6b" bucket="openshift-bootstrap-data-rdossant-installer-03-jjf6b" key="control-plane/rdossant-installer-03-jjf6b-master-2"
I0312 18:00:13.608919     109 s3.go:220] "Deleting S3 object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="1ed0ad52-ffc1-4b62-97e4-876f8e8c3242" machine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" cluster="openshift-cluster-api-guests/rdossant-installer-03-jjf6b" bucket="openshift-bootstrap-data-rdossant-installer-03-jjf6b" key="control-plane/rdossant-installer-03-jjf6b-master-0"

E0312 18:04:25.282967     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYY3QSWKBBDZ7R8, host id: 2f3HawFbPheaptP9E+WRbu3fhEXTMwyZQ1DBPGBG7qlg74ssQR0XISM4OSlxvrn59GeFREtN4hp9C+S5LgQD2g==
 >
E0312 18:04:25.284197     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYY3QSWKBBDZ7R8, host id: 2f3HawFbPheaptP9E+WRbu3fhEXTMwyZQ1DBPGBG7qlg74ssQR0XISM4OSlxvrn59GeFREtN4hp9C+S5LgQD2g==
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="7fac94a1-772a-4c7b-a631-5ef7fc015d5b"
E0312 18:04:25.286152     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYPFY0EQBM42VYH, host id: nJZakAhLrbZ1xrSNX3tyk0IKmMgFjsjMSs/D9nzci90GfRNNfUnvwZTbcaUBQYiuSlY5+aysCuwejWpvi8FmGusbQCK1Qtjr9pjqDQfxzY4=
 >
E0312 18:04:25.287353     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYPFY0EQBM42VYH, host id: nJZakAhLrbZ1xrSNX3tyk0IKmMgFjsjMSs/D9nzci90GfRNNfUnvwZTbcaUBQYiuSlY5+aysCuwejWpvi8FmGusbQCK1Qtjr9pjqDQfxzY4=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-2" reconcileID="b6c792ad-5519-48d5-a994-18dda76d8a93"
E0312 18:04:25.291383     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYGWSJDR35Q4GWX, host id: Qnltg++ia3VapXjtENZOQIwfAxbxfwVLPlC0DwcRBx+L60h52ENiNqMOkvuNwJyYnPxbo/CaawzMT11oIKGO9g==
 >
E0312 18:04:25.292132     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYGWSJDR35Q4GWX, host id: Qnltg++ia3VapXjtENZOQIwfAxbxfwVLPlC0DwcRBx+L60h52ENiNqMOkvuNwJyYnPxbo/CaawzMT11oIKGO9g==
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-1" reconcileID="92e1f8ed-b31f-4f75-9083-59aad15efe79"
E0312 18:04:25.679859     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYSBZGYPC7SNJEX, host id: EplmtNQ+RxmbU88z+4App6YEVvniJpyCeMiMZuUegJIMqZgbkA1lmCjHntSLDm4eA857OdhtHsn+zD6AX7uelGIsogzN2ZziiAZXZrbIIEg=
 >
E0312 18:04:25.680663     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYSBZGYPC7SNJEX, host id: EplmtNQ+RxmbU88z+4App6YEVvniJpyCeMiMZuUegJIMqZgbkA1lmCjHntSLDm4eA857OdhtHsn+zD6AX7uelGIsogzN2ZziiAZXZrbIIEg=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="9e436c67-aca0-409c-9179-0ce4cccce9ad"
E0312 18:04:25.766805     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYVX5DZCE3WREHG, host id: ttRZOKC/DPaAN55tH4SHv5NwPaOtKFxgYtB5w6wC/xWGz5xWtHmrb+QMqestuHJvJDjpxvz1dkOY3IUaDLYPdA==
 >
E0312 18:04:25.767862     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYVX5DZCE3WREHG, host id: ttRZOKC/DPaAN55tH4SHv5NwPaOtKFxgYtB5w6wC/xWGz5xWtHmrb+QMqestuHJvJDjpxvz1dkOY3IUaDLYPdA==
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-2" reconcileID="ba3546c1-410d-4074-b915-508cebcf6462"
E0312 18:04:25.900297     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYR8FXVXJFPZEBF, host id: /IDCA+L+h9/vU7m777mKwyzP1evlKKxB2l3YUe0ZZNS1MBKdVGxDFKrDIHxkYcm9OX0cyRpbUD5+kaGr/Oit3rLFlBdyPdNrG1UsqqO2/X8=
 >
E0312 18:04:25.901660     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYR8FXVXJFPZEBF, host id: /IDCA+L+h9/vU7m777mKwyzP1evlKKxB2l3YUe0ZZNS1MBKdVGxDFKrDIHxkYcm9OX0cyRpbUD5+kaGr/Oit3rLFlBdyPdNrG1UsqqO2/X8=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-1" reconcileID="cabe571a-b320-4ecd-8da4-d3335aee54db"
E0312 18:04:26.081225     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYN8R43NCKYZ7J9, host id: vQw1gpiTnvUzv50VLpKRa8njFma68Ie0LzHB5AXGGlGq/X7kymyA9M17dGgi+hcrvAc4dZ04J9lZrAFDPHbyj+SzzsUOhpneJW8Sr0CafBc=
 >
E0312 18:04:26.082195     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYN8R43NCKYZ7J9, host id: vQw1gpiTnvUzv50VLpKRa8njFma68Ie0LzHB5AXGGlGq/X7kymyA9M17dGgi+hcrvAc4dZ04J9lZrAFDPHbyj+SzzsUOhpneJW8Sr0CafBc=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="c45efec6-6528-4e11-9c81-da27718d1686"
E0312 18:04:26.175397     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: JZAE8SEJZQM2RY2T, host id: EHq4CKwSzqwiQpcOTvRahwsbX+hVY1UinAAQ7vsvDrlpDKEEtxcjtq4F4qa8hzeZECG3OGp+n6FQjLkSw2PXAA==
 >
[...]
E0312 18:17:23.150499     109 awsmachine_controller.go:307] "unable to delete machine" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: EVM8Q64ZV5T5K19G, host id: N6AOedamPQIzaVRnZDz+vg2ueYleJ0e6A0TytrjreNIEj6YUzzzBEEPw2f4WowD+RReANFI0Vac=
 >
E0312 18:17:23.152811     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: EVM8Q64ZV5T5K19G, host id: N6AOedamPQIzaVRnZDz+vg2ueYleJ0e6A0TytrjreNIEj6YUzzzBEEPw2f4WowD+RReANFI0Vac=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-bootstrap" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-bootstrap" reconcileID="ddaee7ad-fe36-4aaa-b3d4-8399065e1ee8"

S3 config used: https://github.com/openshift/installer/blob/master/pkg/asset/manifests/aws/cluster.go#L149-L152

Environment:

• Cluster-api-provider-aws version: main @ 5aa9d22533db1f24da9c4b9e667ee162a2cb52e1
• Kubernetes version: (use kubectl version): v1.29.1+5a4819c
• OS (e.g. from /etc/os-release): RHCOS