terraform-provider-iterative
terraform-provider-iterative copied to clipboard
Optimize cloud-init user data script
On public cloud machines, the startup script needs to be compatible with any modern and sensible GNU+Linux image, and that requires some optimizations and workarounds. For example, we should try to avoid depending on package managers and software other than systemd
.
Installing rclone
Whilst rclone
binary distributions are self–contained Go executables, they come bundled in zip
files, and extracting them in a portable way is hard.
- [x] Short term: #286
- [ ] Mid term: integrate this functionality in the agent by using the
rclone
module.
Watching files
In order to avoid installing a file watcher,[^1] we've replaced “smart” data+log synchronization by polling, considerably increasing transfer costs.
- [x] Short term: #322 <- #321
- ~at very least, we should use
md5sum
andsleep
to push logs only when (1) there are new log lines and (2) a significant amount of time has passed since the last synchronization~
- ~at very least, we should use
- [ ] Mid term: integrate this functionality in the agent by using the
fsnotify
module.
Issues
- https://github.com/iterative/terraform-provider-iterative/issues/321
Other fixes
- [x] #318
- [x] #313
[^1]: Installing software usually requires resorting to distribution–specific package managers.
I haven't yet experimented enough so this might not be applicable, but are you open to having an alternative/selectable options for the log capture?
The ideas I have rattling around might be past the scope of what task is trying to provide.
Supporting arbitrary log backends with the same interface doesn't look like a trivial task. Still, you can always install and run software like the CloudWatch Agent as part of the task script.