workflow
workflow copied to clipboard
teamhephy
Support for Deis deployment pipelines
From @deis-admin on January 19, 2017 23:32
From @kamilchm on July 14, 2014 19:1
Does someone use deis for building deployment pipelines? Heroku have pipelines in labs https://devcenter.heroku.com/articles/labs-pipelines. How can I implement equivalent of it using deis?
Copied from original issue: deis/deis#1318
Copied from original issue: deis/workflow#711
From @deis-admin on January 19, 2017 23:32
From @developerinlondon on July 14, 2014 22:48
+1 to that!
From @deis-admin on January 19, 2017 23:32
From @ntquyen on July 14, 2014 23:42
+1!
From @deis-admin on January 19, 2017 23:32
From @krancour on July 29, 2014 15:31
+1, but also probably not part of an MVP, IMO.
From @deis-admin on January 19, 2017 23:32
From @gabrtv on July 29, 2014 19:41
@kamilchm thanks for pointing us to Heroku pipelines! We're going to study it closely and see if we should include it in Deis. As @krancour pointed out, we can add that feature after our stable release.
Today I would recommend using multiple applications and promoting between them manually via git push or deis pull. This approach has proven sufficient for most CI/CD pipelines we've seen -- and can be implemented easily with something like Jenkins.
From @deis-admin on January 19, 2017 23:32
From @kamilchm on July 30, 2014 6:26
I watched deis pull work and it's a good move in that direction. I can build my deployment pipeline on top of it now but I'm looking forward for pipelines as a concept built into Deis.
I really appreciate where you have come with Deis as PaaS solution. Keep up the good work!
From @deis-admin on January 19, 2017 23:32
From @carmstrong on July 30, 2014 6:58
@kamilchm We really appreciate the kind words!
From @deis-admin on January 19, 2017 23:32
From @suquant on January 14, 2015 11:5
+1
From @deis-admin on January 19, 2017 23:32
From @wotek on February 26, 2015 12:49
+1
From @deis-admin on January 19, 2017 23:32
From @bjwschaap on May 22, 2015 17:31
+1
From @deis-admin on January 19, 2017 23:32
From @rvadim on June 9, 2015 3:35
+1
From @deis-admin on January 19, 2017 23:32
From @scottrobertson on June 15, 2015 19:40
+1
From @deis-admin on January 19, 2017 23:32
From @jjungnickel on June 16, 2015 8:1
+1
From @deis-admin on January 19, 2017 23:32
From @JacopoDaeli on January 21, 2016 0:58
+1
From @deis-admin on January 19, 2017 23:32
From @nicolasbelanger on June 30, 2016 20:56
+1
From @deis-admin on January 19, 2017 23:32
From @phspagiari on July 4, 2016 18:13
+1
From @intellix on March 17, 2017 15:44
I've started working on a UI for Deis Workflow which has the ability to create pipelines and view them: https://github.com/xcaliber-tech/deis-admin
Pipeline view: https://www.dropbox.com/s/dy4p6ehcxfwm236/Screenshot%202017-03-02%2015.25.16.png?dl=0
Still a few items required to finish it though
From @onbjerg on April 19, 2017 15:11
That's great! Will it also support GitHub's auto-deploy feature like Heroku? It would be cool if Deis could create a staging or review app for pull requests 🥇
From @intellix on April 19, 2017 16:20
@onbjerg I started working on a Webhook server which tells Deis to rebuild and create these feature branches with adapters as middleware for the different providers: Github and Bitbucket as Pull Requests come in. I've been working around the clock on other projects though so haven't been able to finalise it :(
Just pushed what I had so far. Not tested at all as it was still actively being played with, but I'm hoping people will get it working and PR :) https://github.com/xcaliber-tech/deis-webhooks
deis/workflow#711
I haven't used Heroku's pipelines feature, and I don't really want to copy another build pipeline, also not knowing what it looks like or anything else to compare it to besides either Jenkins or TravisCI, both of those I'm not sure about, so what config format should we use for the build pipeline feature?
Right now I feel like the incumbent is the Jenkinsfile, with the groovy syntax of Jenkinsfiles.
The jenkins-jobs repo is really something else!
@kingdonb it's nothing to do with Jenkins, CI or anything like that. From the heroku docs: "Pipelines allow you to define how your code should be promoted from one environment to the next. For example, you can push code to staging, have it built into a slug and later promote the staging slug to production".
Its essentially promoting a build from staging to production without rebuilding etc.
Sorry if I am misunderstanding your comment.
No you're explaining along the line of thought I was hoping to provoke...
I really like the idea of adding a formal pipeline concept to Deis but I don't know how the pipeline config should be instantiated. Follow?
I think we probably won't start with Jenkinsfile or .travis.yml, but I'm sure some kind of config artifact will be needed to describe the pipeline steps and their operation, and I don't know if it should be "hephyrc" ...
I'm thinking we will probably end up creating a new config format standard
My expert sense says don't create a new standard if you can avoid it, but Jenkinsfile is not for us...
We don't have environments now, but that seems like the first formal concept to add to the "deis app" to make it pipeline-capable
Then the second requirement is a function that must evaluate to "true" in order to promote from one environment to the other. I can imagine a separate function being invoked for dev -> test than the one you would use for test -> prod but the idea that it should return true in order to promote the environment build seems like an obvious thing in 2018 that I think we should codify somehow.
Someone said it would be cool if Deis supported FaaS natively somehow. Kubeless, Serverless, I think both are very interesting. Brigade seems like the most obvious place to land our CI since it came from Deis team and is event-driven so seems like a natural to connect with git hooks or any other pipeline build triggers.
@scottrobertson What runtimes would you use in a pipeline? J/W
@scottrobertson What runtimes would you use in a pipeline? J/W
I mostly run Ruby apps, but also PHP and Python.
I am primarily a Rubyist too!
Heroku-style buildpacks seem to handle pretty well on Deis in Ruby
It's been a long time since i used Deis, but it worked very well for Ruby when i did, especially Rails apps.
it's nothing to do with Jenkins, CI or anything like that
@kingdonb, I think you might still be misunderstanding @scottrobertson's point. Or I am inferring that because you went on talking about CI.
I was around when this feature was first requested. It had actually been discussed internally long before this issue was even opened. It was well understood to be something equivalent to Heroku pipelines and that feature, as @scottrobertson was trying to explain, does not have anything to do with CI. And although it does have something to do with deployment, it has nothing to do with CD-- continuous deployment.
Taking it back to 12factor, recall that a release is defined as build + config. The "pipeline" feature that was requested here was the ability to (manually) promote the build (slug) from one "environment" (e.g. staging) to another (e.g. "production"; where the config is different), all without rebuilding.
Contrast to how you could do the above with Workflow today if you were willing to make some small compromised. In both cases, create separate apps for each "environment":
Option 1: Don't use Builder. Have your own Jenkins/Travis/Circle/whatever-based CI process that cranks out Docker images as deployable artifacts. use deis pull workflow only. When you're happy with the Docker image you've deployed to my-app-staging, deis pull the same image to my-app-production.
$ deis apps:create my-app-staging --no-remote
$ deis apps:create my-app-prod --no-remote
$ deis pull <image> -a my-app-staging
### Happy with staging?
$ deis pull <same image> -a my-app-production
Options 2: Use git push workflow and when you're happy with what's in my-app-staging, git push to a different remote that is connected to the my-app-staging app.
$ deis apps:create my-app-staging --remote staging
$ deis apps:create my-app-prod --remote prod
$ git push staging master
### Happy with staging?
$ git push prod master
The problem with option 1, is it's sort of "advanced usage." (Or that's what I am told. Personally, this is how I had felt Workflow should always be used, but the git push workflow is very valuable to some people.)
The problem with option 2 is that the application is actually rebuilt when promoted. You lose the guarantee that the code running in prod is the same as what you were happy with in staging. (I for one, could not live with that.)
The feature request had been to combine the best of the two options above so you could continue to use that popular git push workflow to do the initial build and then move the deployable artifact (a "slug" if you are using the git push workflow) into another environment without rebuilding.
(You are absolutely right that doing this [probably] requires a missing notion of "environment" to be implemented, such that a single application has multiple environments. Don't be surprised by the sheer volume of effort that probably represents. It's a big change to the model.)
My best advice is to avoid further conflating this with CI and CD. Look at CI and CD as higher level than Workflow and the deis CLI. Think about how the like of Jenkins/Travis/Circle, etc. could be used to orchestrate deployments to Workflow instead of trying to build CI/CD into Workflow.
Also...
Someone said it would be cool if Deis supported FaaS natively somehow. Kubeless, Serverless, I think both are very interesting.
Brigade seems like the most obvious place to land our CI since it came from Deis team and is event-driven so seems like a natural to connect with git hooks or any other pipeline build triggers.
Not sure how these two things are connected.
Just as I've cautioned not to let CI/CD cloud the (Heroku-style) pipeline waters, don't let FaaS cloud those waters either. FaaS is compelling, but don't hitch all these different horses together.
Thank you @krancour. Sorry i could not explain properly, things have been a bit crazy!