Cleanup with remoteManifests causes unexpected behavior

[ctrahey]

While it is documented that cleanup is enabled by default, I did not expect resources that existed in my cluster before running skaffold to be removed. Perhaps it should be considered: If the render step uses remoteManifests, should cleanup default to false? Is there some other way to avoid a stock behavior that is destructive to resources not created by skaffold?

Steps to reproduce:

1. Have a k8s cluster with some resource (deployment) already in it
2. Setup skaffold.yaml with a manifests section referring to the existing resource
3. run skaffold dev (in my case there was some error - so it didn't even fully stabilize before cleaning up!)
4. Observe your pre-existing resource is deleted.

manifests:
  remoteManifests:
    - manifest: some-namespace:deployment/some-deployment-name
      kubeContext: default

Build [someregistry.io/somerepository] succeeded
WARN[0062] k8s/*.yaml did not match any file             subtask=-1 task=DevLoop
Tags used in deployment:
 - someregistry.io/somerepository -> someregistry.io/somerepository:0873d67e-dirty@sha256:d..1
Starting deploy...
 - deployment.apps/some-deployment-name configured
Waiting for deployments to stabilize...
 - some-namespace:deployment/some-deployment-name: container some-ctr-name is waiting to start: someregistry.io/somerepository:0873d67e-dirty@sha256:d..1 can't be pulled
(repeated 2x)
Cleaning up...
 - deployment.apps "some-deployment-name" deleted
1/1 deployment(s) failed

So in my case an image pulling issue took place, but the issue is in the default cleanup behavior, which is perhaps unexpected when the manifests are not coming skaffold's control.

Commentary on use case: We don't use Skaffold for actual deployments, but want to use it for some triaging and rapid testing during development with remote clusters. So the cluster had resources deployed via Helm, and the idea is to use skaffold to "take over" the deployment object you're developing against. Works great! However, since the deployment itself is deployed with a very different tool (for other reasons, the helm chart cannot be used as the skaffold render step in this case - it is managed somewhere else with complex logic for it's values, etc).

I'll be okay knowing that I need --cleanup=false but this might be worth a consideration.

[ctrahey]

Perhaps the ideal solution is to include another key in the remoteManifests configuration for a cleanup setting.