pulumi-command icon indicating copy to clipboard operation
pulumi-command copied to clipboard

CopyFile: support assets, archives, and recursive copying

Open thomas11 opened this issue 1 year ago • 0 comments

This PR allows the CopyFile resource to recursively copy directories as well, similar to scp -r.

This PR enhances the CopyFile resource in a few ways.

  • It can now copy directories as well, recursively with their contents.
  • It now takes the standard Pulumi asserts and archives as input, allowing for seamless interop with other resources.
  • It now checks whether the specified file or directory have changed (in content, not timestamp) and copies only if they did.

In light of these changes, I've renamed CopyFile to just Copy.

Resolves #23 Resolves #33 Resolves #42

TODO

  • [ ] The preview operation doesn't tell the user whether the resource will copy or not, rendering it rather useless. Probably need to add an output property.
  • [x] ~~The current implementation fails as soon as a file or directory already exists on the remote, like the previous CopyFile.~~
  • [x] Before GA, should we rename CopyFile to Copy or RemoteCopy or insert your proposal here?
  • [ ] The integration test copies a bunch of EC2 setup code with TestEc2RemoteTs, which we should share instead. What would be a good way to do that, given the code is in the TS tests, not in the Go driver?

Design questions

  • [ ] Should we expose knobs such as "clear remote directory before copying" (to avoid leftover files from previous copies) or should we even do that by default? Impacts behavior and should be resolved before GA.
  • [ ] Should we expose a flag to allow the user to fail the copy if the remote exists, rather than silently overwriting?

Implementation notes

  • I've looked around for open source Go packages implementing scp of folders, but to my surprise, I couldn't find one with an acceptable license that 1) wasn't ancient and 2) was cross-platform.
  • The current implementation copies files sequentially and is therefore probably slow for large trees. If we replaced the implementation, I don't think the shape of the resource would change, so this shouldn't be a GA blocker.
  • The size.ts file in both steps of the integration test is useless because it always has the same value. However, I found that without a TS file present in the additional step, it would not be run. Haven't debugged further.

thomas11 avatar Apr 30 '24 11:04 thomas11