dobi icon indicating copy to clipboard operation
dobi copied to clipboard
verified publisher icon dnephin

Autoclean bugs & design

Open cescoferraro opened this issue 9 years ago • 2 comments

  • [ ] Autoclean command should not require a variable.
  • [x] Autoclean command should force remove a container to avoid things like You cannot remove a running container d8c9d51c44f7323114923f0c0dcaa85b8f14b08e5f49450a35aa63b1d11c3a6d. Stop the container before attempting removal or use -f

If autoclean is meant to be project specific, maybe there is room for another command. Maybe dobi format that should

  • [ ] not be project specific
  • [ ] stop/remove all running containers created by dobi?
  • [ ] remove exited containers.
cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ docker ps -a | grep cescoferraro-
d8c9d51c44f7        cescoferraro/mongo:latest   "/run.sh"                 8 hours ago         Up 8 hours                27017/tcp, 28017/tcp   dobi-cescoferraro-serve
d43f786a2c7c        cescoferraro/api:tools      "/usr/bin/reflex -r \\"   41 hours ago        Exited (0) 14 hours ago                          api-cescoferraro-watch
850158394fbc        cescoferraro/power:watch    "docker/script/watch"     42 hours ago        Exited (0) 14 hours ago                          power-cescoferraro-watch
6e77373d6526        cescoferraro/iot:base       "ionic serve --all"       2 days ago          Exited (0) 14 hours ago                          iot-cescoferraro-watch


cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ dobi autoclean
[job:rm watch]  Removed
[job:rm test-unit]  Removed
[job:rm test-examples]  Removed
[job:rm test-examples]  Removed
[job:rm test-unit]  Removed
[alias:rm test] Done
[WARN] [mount:rm source] .:/go/src/github.com/dnephin/dobi Bind mounts are not removable
[job:rm shell]  Removed
[WARN] container=dobi-cescoferraro-serve [job:rm serve]  Failed to remove container: API error (409): You cannot remove a running container d8c9d51c44f7323114923f0c0dcaa85b8f14b08e5f49450a35aa63b1d11c3a6d. Stop the container before attempting removal or use -f

[job:rm serve]  Removed
[WARN] [image:rm releaser] dobi-release failed to remove "dobi-release:dobi-cescoferraro": no such image
[image:rm releaser] dobi-release Removed
[job:rm release-version]  Removed
[ERROR] A value is required for variable "env.DOBI_VERSION"


cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ VERSION=22 go run main.go autoclean
[job:rm watch] {[]} Removed
[job:rm test-unit] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-unit] {[]} Removed
[alias:rm test] Done
[WARN] [mount:rm source] .:/go/src/github.com/dnephin/dobi Bind mounts are not removable
[job:rm shell] {[]} Removed
[WARN] [job:rm serve] {[]} container=dobi-cescoferraro-serve Failed to remove container: API error (409): You cannot remove a running container d8c9d51c44f7323114923f0c0dcaa85b8f14b08e5f49450a35aa63b1d11c3a6d. Stop the container before attempting removal or use -f

[job:rm serve] {[]} Removed
[WARN] [image:rm releaser] dobi-release failed to remove "dobi-release:dobi-cescoferraro": no such image
[image:rm releaser] dobi-release Removed
[job:rm release-version] {[]} Removed
[ERROR] A value is required for variable "env.DOBI_VERSION"
exit status 1


cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ DOBI_VERSION=234 VERSION=22 go run main.go autoclean
[job:rm watch] {[]} Removed
[job:rm test-unit] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-unit] {[]} Removed
[alias:rm test] Done
[WARN] [mount:rm source] .:/go/src/github.com/dnephin/dobi Bind mounts are not removable
[job:rm shell] {[]} Removed
[WARN] [job:rm serve] {[]} container=dobi-cescoferraro-serve Failed to remove container: API error (409): You cannot remove a running container d8c9d51c44f7323114923f0c0dcaa85b8f14b08e5f49450a35aa63b1d11c3a6d. Stop the container before attempting removal or use -f

[job:rm serve] {[]} Removed
[WARN] [image:rm releaser] dobi-release failed to remove "dobi-release:dobi-cescoferraro": no such image
[image:rm releaser] dobi-release Removed
[job:rm release-version] {[]} Removed
[WARN] [image:rm dist-img] dnephin/dobi failed to remove "dnephin/dobi:234": no such image
[image:rm dist-img] dnephin/dobi Removed
[job:rm release-version] {[]} Removed
[ERROR] A value is required for variable "env.GITHUB_TOKEN"
exit status 1


cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ GITHUB_TOKEN=2 DOBI_VERSION=234 VERSION=22 go run main.go autoclean
[job:rm watch] {[]} Removed
[job:rm test-unit] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-unit] {[]} Removed
[alias:rm test] Done
[WARN] [mount:rm source] .:/go/src/github.com/dnephin/dobi Bind mounts are not removable
[job:rm shell] {[]} Removed
[WARN] [job:rm serve] {[]} container=dobi-cescoferraro-serve Failed to remove container: API error (409): You cannot remove a running container d8c9d51c44f7323114923f0c0dcaa85b8f14b08e5f49450a35aa63b1d11c3a6d. Stop the container before attempting removal or use -f

[job:rm serve] {[]} Removed
[WARN] [image:rm releaser] dobi-release failed to remove "dobi-release:dobi-cescoferraro": no such image
[image:rm releaser] dobi-release Removed
[job:rm release-version] {[]} Removed
[WARN] [image:rm dist-img] dnephin/dobi failed to remove "dnephin/dobi:234": no such image
[image:rm dist-img] dnephin/dobi Removed
[job:rm release-version] {[]} Removed
[ERROR] A value is required for variable "env.CIRCLE_TAG"
exit status 1


cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ GITHUB_TOKEN=2 DOBI_VERSION=234 VERSION=22 CIRCLE_TAG=22 go run main.go autoclean
[job:rm watch] {[]} Removed
[job:rm test-unit] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-unit] {[]} Removed
[alias:rm test] Done
[WARN] [mount:rm source] .:/go/src/github.com/dnephin/dobi Bind mounts are not removable
[job:rm shell] {[]} Removed
[WARN] [job:rm serve] {[]} container=dobi-cescoferraro-serve Failed to remove container: API error (409): You cannot remove a running container d8c9d51c44f7323114923f0c0dcaa85b8f14b08e5f49450a35aa63b1d11c3a6d. Stop the container before attempting removal or use -f

[job:rm serve] {[]} Removed
[WARN] [image:rm releaser] dobi-release failed to remove "dobi-release:dobi-cescoferraro": no such image
[image:rm releaser] dobi-release Removed
[job:rm release-version] {[]} Removed
[WARN] [image:rm dist-img] dnephin/dobi failed to remove "dnephin/dobi:234": no such image
[image:rm dist-img] dnephin/dobi Removed
[job:rm release-version] {[]} Removed
[job:rm github-release] {[]} Removed
[job:rm binary] {[./dist/bin/]} Removed
[job:rm deps] {[vendor/]} Removed
[alias:rm release] Done
[WARN] [image:rm refactorer] dnephin/go-refactor failed to remove "dnephin/go-refactor:latest": no such image
[image:rm refactorer] dnephin/go-refactor Removed
[job:rm refactor-shell] {[]} Removed
[WARN] [mount:rm projectdir] .:/home/cescoferraro/code/go/src/github.com/dnephin/dobi Bind mounts are not removable
[image:rm mongo] cescoferraro/mongo Removed
[job:rm mocks] {[]} Removed
[image:rm linter] dobi-linter Removed
[job:rm lint] {[]} Removed
[job:rm github-release] {[]} Removed
[WARN] [image:rm example-tester] dobi-test-examples failed to remove "dobi-test-examples:dobi-cescoferraro": no such image
[image:rm example-tester] dobi-test-examples Removed
[job:rm docs-watch] {[]} Removed
[job:rm docs-shell] {[]} Removed
[WARN] [image:rm docs-img] dobi-docs-dev failed to remove "dobi-docs-dev:dobi-cescoferraro": no such image
[image:rm docs-img] dobi-docs-dev Removed
[job:rm docs-build] {[./docs/build/html]} Removed
[job:rm docs] {[]} Removed
[WARN] [image:rm dist-img] dnephin/dobi failed to remove "dnephin/dobi:234": no such image
[image:rm dist-img] dnephin/dobi Removed
[WARN] [mount:rm dist] ./dist/bin/:/go/bin/ Bind mounts are not removable
[job:rm deps] {[vendor/]} Removed
[job:rm binary] {[./dist/bin/]} Removed
[job:rm docs-build] {[./docs/build/html]} Removed
[job:rm test-unit] {[]} Removed
[job:rm lint] {[]} Removed
[alias:rm ci-circle-main] Done
[job:rm test-examples] {[]} Removed
[alias:rm ci-circle-examples] Done
[image:rm builder] dobi-dev Removed
[job:rm binary] {[./dist/bin/]} Removed
[alias:rm binary-linux] Done
[job:rm binary] {[./dist/bin/]} Removed
[job:rm binary] {[./dist/bin/]} Removed
[job:rm docs-build] {[./docs/build/html]} Removed
[job:rm test-examples] {[]} Removed
[job:rm test-unit] {[]} Removed
[alias:rm test] Done
[job:rm lint] {[]} Removed
[alias:rm all] Done


cescoferraro@desktop: ~/code/go/src/github.com/dnephin/dobi on feature/serve [?]
$ docker ps -a | grep cescoferraro-
d8c9d51c44f7        c8cd8c23fe0e                "/run.sh"                 8 hours ago         Up 8 hours                27017/tcp, 28017/tcp   dobi-cescoferraro-serve
d43f786a2c7c        cescoferraro/api:tools      "/usr/bin/reflex -r \\"   41 hours ago        Exited (0) 15 hours ago                          api-cescoferraro-watch
850158394fbc        cescoferraro/power:watch    "docker/script/watch"     42 hours ago        Exited (0) 15 hours ago                          power-cescoferraro-watch
6e77373d6526        cescoferraro/iot:base       "ionic serve --all"       2 days ago          Exited (0) 15 hours ago                          iot-cescoferraro-watch

cescoferraro avatar Nov 21 '16 11:11 cescoferraro

Autoclean command should not require a variable.

I think this is still necessary, because the variable could be used in a tag, and without it you wont know which image to remove. With the new capture support it should be possible to encode the variables in the config, most of the time.

You cannot remove a running container

I wonder if the order is wrong. It should be ordered in such a way that all containers are stopped before images are removed.

dnephin avatar Nov 21 '16 16:11 dnephin

I think this is still necessary, because the variable could be used in a tag, and without it you wont know which image to remove.

I think the that all tags should be removed. Maybe we should not remove the image if Pull. isSet, but otherwise people expect autoclean to clean everything.

I wonder if the order is wrong. It should be ordered in such a way that all containers are stopped before images are removed.

we could just append stop before rm here. https://github.com/dnephin/dobi/blob/master/cmd/autoclean.go#L44 but not all resources have a stop action. We must sort our resources better than alphabetically, so we can apply custom logic to each resource, like:

func removeTasks(conf *config.Config) []string {
	resources := conf.FilterNot(&config.JobConfig{})
	tasks := []string{}
	for i := len(resources) - 1; i >= 0; i-- {
		tasks = append(tasks, resources[i]+":rm")
	}
	resources = conf.FilterBy(&config.JobConfig{})
	for i := len(resources) - 1; i >= 0; i-- {
		tasks = append(tasks, resources[i]+":stop")
		tasks = append(tasks, resources[i]+":rm")
	}
	return tasks
}


// FilterBy returns a list of resource names of the same time as the given parameter
func (c *Config) FilterBy(thing interface{}) []string {
	filter := reflect.TypeOf(thing)
	names := []string{}
	for name := range c.Resources {
		if filter == reflect.TypeOf(c.Resources[name]) {
			names = append(names, name)
		}
	}
	sort.Strings(names)
	return names
}

But then I realised there is no stop action for job task.

cescoferraro avatar Nov 22 '16 03:11 cescoferraro