Updating a RouteTable resource in-place results in sync issues

[mattzech]

Describe the bug We are trying to add more routes to a RouteTable, but the ec2-controller is unable to reconcile these changes. Cannot find route, but the route exists as active in the status list

  - lastTransitionTime: "2024-04-10T20:56:12Z"
    status: "True"
    type: ACK.ReferencesResolved
  - message: "InvalidRoute.NotFound: no route with destination-cidr-block 8.8.8.8/32
      in route table rtb-03892bf39fc88\n\tstatus code: 400, request id: 18bb8e8c-cc4c-40c9-80aa-e29852122416"
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2024-04-10T20:56:15Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: "InvalidRoute.NotFound: no route with destination-cidr-block 8.8.8.8/32
      in route table rtb-03892bf39fc88\n\tstatus code: 400, request id: 18bb8e8c-cc4c-40c9-80aa-e29852122416"
    status: Unknown
    type: ACK.ResourceSynced
  ownerID: "824203098861"
  routeStatuses:
  - destinationCIDRBlock: 8.8.8.8/32
    origin: CreateRoute
    state: active
    transitGatewayID: tgw-0997b36cf235fg31

I am trying to figure out what to do to get rid of this error message. Trying to Create a Route results in 400 error because the Route does not already exist? Why is this being checked and is there anywhere else I should look for more info? Seeing CreateRoute events in CloudTrail every few seconds and need this stable to proceed with building out our infrastructure.

Thanks!

Steps to reproduce

• Create a route object
• Add several routes with target TGW
• Observe error

Expected outcome Seamless update of routes. All that is being done is adding more routes to the list

Environment

• Kubernetes version 1.27
• Using EKS (yes/no), if so version? No
• AWS service targeted (S3, RDS, etc.) EC2