strider
strider copied to clipboard
Strider-CD
Strider 2.0 Brainstorming
Just putting this out there, want to list out some of the "must have"s and "nice to have"s for 2.0.
Must Haves
- [ ] Add support for organizations (groups)
- [ ] On-boarding (setup) UI / Post setup #681
- [ ] Multiple build environments
- [ ] Simultaneous builds
- [x] Upgrade to latest Express (4 atm)
- [ ] Use express.Router() throughout to decouple routes
- [ ] Upgrade to latest Bootstrap (3 atm) / Cleanup responsive UI
- [ ] Remove deprecated APIs
- [ ] Update plugin API, no globals
- [x] Bring back strider.json as a build config #674
- [ ] Make into an SPA
- [ ] Upgrade to consistent (one) view templating e.g. Handlebars #689
Nice To Haves
- [ ] Refresh the website
- [ ] Abstract database calls
- [ ] Features behind flags / Faster release cycles
Feel free to disagree, and contribute! Discussion welcome!
One that I wont put up there, just because it's too controversial, is to rewrite the client-side with Ember.js. Especially with how angular is going and the spaghetti code that we have (even after browserify), it would really help bring in new contributors from the Ember community, since they'll be able to jump in very easily.
Here's a good question:
Would it be easier to start from scratch, with what we learned, and to write it from the ground up (feels like that would be better) or to refactor and hope that we don't introduce many new bugs?
Since we have some fundamental changes that have been requested, and plenty of code that is just not up to par.
Would certainly love to help where I can with UI/webapp tasks and any testing assistance.
So I've got to ask about this Angular vs. Ember thing:
- What are some of the difficulties you are experiencing with UI development?
- What would Ember accomplish that Angular can't?
- Is the spaghetti code a result of Angular's design or the Strider application design/modularity?
I'd support Ember over Angular. Especially in a large app like this, Ember's conventions get you a lot of benefits.
I think that build configuration should be stored along with projects instead of the DB. So I'm suggesting to make Strider build git projects having strider.yml file without making any further configurations
Please expound on this feature request. Are you saying they can have strider config that lets strider know how to build it? So minimal configuration needed for setup, only adding the repo? On Dec 9, 2014 6:01 PM, "Vitaly Sukhomlinov" [email protected] wrote:
I think that build configuration should be stored along with projects instead of the DB. So I'm suggesting to make Strider build git projects having strider.yml file without making any further configurations
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-66375100.
Correct, I'm seeing a value in doing that if you're planning to add access for organizations.
On the topic of a config file, there was recently a little discussion on this here #592 .
According to @niallo 's comment there is already support for a strider.json or ini file but it is completely undocumented.
@bitwit
What are some of the difficulties you are experiencing with UI development?
Javascript in HTML, which basically is the same thing that we as the web development community have been trying to minimize since the days of the dominance of PHP. Not only is this hard to debug, read, and keep dry, but it's also hard to extend and expand.
What would Ember accomplish that Angular can't?
Ember is great for large applications that need to scale and grow. Making it an SPA is perfect for Strider, since it'll have a slightly longer load in the beginning, and then be lightning fast. Ember has nice conventions and really the best router, which is an absolute necessity for web apps. Ember Data, is amazing, and will totally fit the way strider is organized, providing nice abstractions. This will allow for making the backend a REST API that can serve multiple front ends. Also ember-cli and the build/addon system, which is super!
Is the spaghetti code a result of Angular's design or the Strider application design/modularity?
Both, since Angular doesn't portray any conventions that everyone should follow, it lends itself to being used poorly.
@knownasilya Great answer! That gives some interesting food for thought on why Ember is possibly a better MVW framework for open source projects -> Conventions! It's also probably the best anti-Angular argument I've heard lately among all the FUD that seems to be trending lately :)
I'll admit I've seen some lack of convention in Strider's plugin controller implementations. When working on the plugin template, I had to review a lot of plugins to try and extract what seemed like the ideal standard to build upon. The dependence on the plugin controller's $scope already having object's like config attached via a parent $scope also seemed a bit counterintuitive.
A few debatable pros and cons of switching:
big pro - Conventions. Great for getting all open source contributors working in better alignment. pro - Other abstractions that are part of Ember that are not in core Angular e.g. Ember Data con - I believe Angular is a more widely known framework, which is also useful for open source. Currently Strider is a pure MEAN stack application. MEEN is phonetically the same though :D con - Guaranteed there would be no backward compatibility with 1.x plugins
Could probably flesh this out more
On another subject - I think community support for Strider needs improvement too, i.e. a better place than IRC to tell people where to direct questions. Possibly creating a forum or using Stack Overflow with a #strider tag to follow. I think in IRC sometimes no one is really around and questions get lost.
There was a great discussion recently about improving the docs and linking directly to them from within the webapp ui - I think this is a great direction too
What about using gitter? https://gitter.im/ along with an SO bot for IRC Also we could use discourse as a forum, so something like discuss.strider.com
And yes, the wiki docs are a good idea!
Stack overflow would be great. It's not something I've ever gotten into as a contributor but there is a urge amout of valuable content there - agree that irc doesn't seem to be very useful.
On Tuesday, December 9, 2014, Ilya Radchenko [email protected] wrote:
What about using gitter? https://gitter.im/
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-66400661.
Niall O'Higgins W: http://niallohiggins.com E: [email protected] T: @niallohiggins
There was a great discussion recently about improving the docs and linking directly to them from within the webapp ui - I think this is a great direction too ~bitwit
Right, @shaunc made some great suggestions, see here https://github.com/Strider-CD/strider/issues/208#issuecomment-64938623
Regarding client-side JS frameworks... I like the React stuff. I don't know anything about Ember. I like Meteor too. I also like Angular. cries
:D I would order things React > Ember > Angular fwiw
On Wed Dec 10 2014 at 12:21:16 PM Keyvan Fatehi [email protected] wrote:
There was a great discussion recently about improving the docs and linking directly to them from within the webapp ui - I think this is a great direction too
~bitwit
Right, @shaunc https://github.com/shaunc made some great suggestions, see here #208 (comment)
https://github.com/Strider-CD/strider/issues/208#issuecomment-64938623
Regarding client-side JS frameworks... I like the React stuff. I don't know anything about Ember. I like Meteor too. I also like Angular. cries
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-66507321.
React is only the V in MV*, so there's plenty missing, and it's just too low-level with an extra language (jsx) to learn. They do have nice ideas, and many of those have already landed in Ember behind feature flags.
React and Angular are nice for custom brewed frameworks, like within a company, but not so nice for an opensource app where you're trying to get everyone on the same page.
Meteor is a whole different story, but also has very loose conventions, and plenty of ways to get lost. Plus you have to buy into the whole stack, package management, server, client, db.
I think was kidding with meteor w/ regards to Strider (although I use Meteor currently for rapid prototyping... kinda like how I still sometimes use ruby on rails) -- but I'm not kidding about the way it (and React) manage to give you two-way data binding without encouraging obtrusive javascript and cryptic new html tags like the way angular does it. You must admit, though, that angular is powerful and our plugin development has been made super easy as a result of how angular slots right into it. @bitwit makes some good points regarding that as well. In the case of plugins the solution is more docs for sure, not changing toolchains, not more coding.
That said,
I see React being just V as an advantage, not a disadvantage, and fitting of the super-modular style of strider. Maybe Ember is great but this has never been shown to me and I don't know how great the cost of transition is, I don't know how big it is and how modular it is. If it's a kitchen-sink framework (like Rails is when compared to Sinatra or Rack), then I'd wager the cost is high. I'd wager it's not very fitting of our modular style and that we'd be excluding more people from the project.
I'd wager that the cost of sticking a jsx transform in our browserify and rewriting some (new) views in a React style is minimal in comparison to a switch to Ember or putting effort towards removal of existing Angular stuff. I know for a fact React will allow us to step-by-step move from Angular to React, over time. Can the same be said about Ember? Hell, even backbone seems safer / more realistic. I am not saying let's switch to React! I'm saying this: I'd rather focus on fixing, maintaining, documenting, cleaning up the current stuff than changing toolsets or technologies unless it really makes sense to do so.
The one main pain point I heard was routing. The only client-side routing I know about in Strider was written by me really quickly because I got sick of clicking the tab I was working on ... we can totally write that properly, modularly, or find one that is well written and independent of these giant frameworks (angular, ember, etc). e.g. I think this might be good? https://github.com/RubaXa/Pilot
As far as single-page apps, I think they're overrated and suck when added as an afterthought. Strider works well as a multi-page app and should be sticking to this (and very minimal javascript).
That said,
I am totally into the idea of creating a separate Strider Ember/SPA repository that consumes Strider as an API endpoint. In my opinion, this takes Strider in a good direction.
Well said @keyvanfatehi,
At the core, I think Strider's attractiveness is its ease of adoption. Being MEAN stack has lent really well to that. Having been someone who tried CI before and never understood where the benefit was given the time required to get it up and running, Strider was this amazing breath of fresh air that made it so simple. There was little argument NOT to do CI anymore. So I think that when it comes to Strider direction and decision making the bottomline question should always be "Does this adhere to Strider's core competency - simplicity and approachability?" That principle should be applicable both to interface as well as open source contribution considerations.
I'm busy porting my application from ember to ember-cli. I like Ember but it definitely requires that you "drink the cool-aid", so can see why it might not be the best choice here.
I would think that an important consideration would be trying to decouple as much as possible, whatever the choice. I think its a great idea to split the client (or clients?) into another repo, and make interactions between components REST based as far as possible.... Maybe there should be a spec for how a client -- eg -- accepts a plugin.
Getting back to the templating discussion, the usage of angular templating on top of swig is a big issue since they are using the same control structures. We should definitely pick either single page app approach or the clean express way. Or replace everything with something else haha.
+1 for "single-page-app" in separated project (or projects) as that way you could connect various different clients or types of clients, all talking to a clear REST interface.
And one more thing, I'd avoid using Bootstrap and Angular in plugins. That will extremely complicate upgrade of these libraries from both plugin maintainer and project owner perspective in case of any major API change (like Angular 2.0). It might be a part of an initiative to redesign plugins API
@vsukhomlinov that is a good point. We would have to write an API that plugin authors would have to use, but at the same time keep it as flexible as it is now, and to prevent ugly plugins. This is complex enough that it would need it's own brainstorming thread :smile:
ugly plugins = bad spaghetti code, bringing in libraries that are too fat, etc..
Lots of cool ideas here, I definitely like the direction of a Single Page/REST API style design. A few thoughts I had over the weekend that relate to this and beyond:
- I think 2.0 should consider more 'types' of plugins; particularly Project, and Webapp plugin levels. Some plugins such as the strider-github-status plugin are probably better suited at a higher level than on a branch by branch basis. The status plugin even behaves like it is on all branches when you add it to master. Pretty much everything is a 'job' plugin unless you are making your own provider (not likely) and I think that is narrowing the possibility and flexibility of plugins. Some plugins should be powerful enough to have both project level settings as well as show up in the drag-drop interface on a branch by branch basis. e.g. a scheduler plugin that is mostly configured at a high level, but registers individuals branches using the current interface. Obviously you don't want to make this too complicated for users, but I'm sure others who have written strider plugins have experienced making a job plugin that is mostly webapp logic.
- If moving toward a REST API where the front end framework is not necessarily known, then plugins would need to store their display mechanisms as part of their API response. i.e. If my plugin needs a timepicker and I can't be certain that Bootstrap UI is being used, then I need to indicate that the configuration value should display as a timepicker. Currently plugins have too much UI dependency to go pure API. As @knownasilya suggested, this probably is it's own discussion entirely.
- I see @keyvanfatehi is already looking into supporting
.strider.jsonasap. I think supporting a JSON format for configuration makes a lot of sense since it most closely reflects how config data is stored in MongoDB. Also glad to see the idea of making the JSON easy to export from any project is being considered :)
If my plugin needs a timepicker and I can't be certain that Bootstrap UI is being used
I think we can allow the plugins to depend on styling, because we don't want the plugins to clash, or the authors to bring in bootstrap just for their plugin (overkill). The only issue is backwards compatibility, but that can be solved with a minimum/maximum semver version, e.g.
"strider": {
"compatible": ">= 1.5"
}
On Dec 15, 2014, at 2:18 PM, Kyle Newsome [email protected] wrote: I think 2.0 should consider more 'types' of plugins; particularly Project, and Webapp plugin levels. Some plugins such as the strider-github-status plugin are probably better suited at a higher level than on a branch by branch basis. The status plugin even behaves like it is on all branches when you add it to master. Pretty much everything is a 'job' plugin unless you are making your own provider (not likely) and I think that is narrowing the possibility and flexibility of plugins. Some plugins should be powerful enough to have both project level settings as well as show up in the drag-drop interface on a branch by branch basis. e.g. a scheduler plugin that is mostly configured at a high level, but registers individuals branches using the current interface. Obviously you don't want to make this too complicated for users, but I'm sure others who have written strider plugins have experienced making a job plugin that is mostly webapp logic.
We are designing a plugin interface for our own application. We decided to go with a two-level structure: and “app” which contains several “plugins”. Perhaps that would be applicable here. A plugin means you are connecting to a particular interface. Sometimes you want to connect to more than one interface at once; other times you explicitly don’t want to provide one interface while plugging into another.
With this structure, the “app” repo would contain a “strider.json” in the root that specifies where the app plugs in and any other configuration to use those plugs.
I love the idea of creating a REST API, it allows everyone to write platform agnostic applications (mobile applications, plugins, custom interfaces etc) and makes the jobs of committers easier and faster so we don't have to deal with view management at all. The main drawback is it privatises the logic and can cause lock-in for plugin authors and committers.
I'd love to help out where possible, for me I'd love to have a mobile app with some form of notifications of build status' done in something like Xamarin to keep the codebase consistent.
On the plugin types, I like the idea of having Strider plugins and project plugins. As @bitwit rightly points out, the git plugin should really be a strider plugin as opposed to a branch per branch plugin, it would simplify the interface no-end.
I agree with @keyvanfatehi on such a large change to Strider RE Ember/Angular/React. In a purely pragmatic universe I could say that something like React would really help but it only has single direction binding and if a one page app is really want the people want I think 2 way binding is a must but maybe single direction would suffice with sockets but that then means there would have to be a separate backend for connection management and communicating with any http based API. Definitely food for thought.
RE the differences between multipage and single page apps, I think this is a perceived speed thing and directly relates to the UX of a project, personally right now Strider is miles ahead of competitors purely because it is a single page app and responds very very quickly. I think it's part of it's hook in the CI community, fast to set up, easy to get running and very fast running even on a small server. I'm running it on a single core 4GB system with only 10GB space.
A bit crazy suggestion, but hear me out. Plugins we have are not portable and can't be used in any other projects. They complicate UI upgrades as well as API changes. Why can't we pull core plugins into the main project and focus on the core instead of making super generic plugin APIs?
It depends which plugins you mean, but most of the reasoning is to have a very strong and well-tested module system.
If the default Strider distribution is heavily plugin-based, then by necessity the plugin system is going to be powerful and good.
Yes, there is overhead to this approach, but in the case of Strider, where the goal is to explicitly not have a monolith and to be extremely pluggable, I think it is a model which has served us well.
My experience with CI/CD tools is that so much customization is be needed by different organizations for their pipeline that the CI/CD system must be built with this in mind.
It is also philosophically similar to node.js itself: Small core and large module ecosystem.
On Friday, December 19, 2014, Vitaly Sukhomlinov [email protected] wrote:
A bit crazy suggestion, but hear me out. Plugins we have are not portable and can't be used in any other projects. They complicate UI upgrades as well as API changes. Why can't we pull core plugins into the main project and focus on the core instead of making super generic plugin APIs?
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-67658472.
Niall O'Higgins W: http://niallohiggins.com E: [email protected] T: @niallohiggins
I agree with @niallo, having a powerful plugin system not only gives the core a smaller footprint it allows people to install the modules of their choice (some people still use SVN for some reason) so won’t have a use for seemingly innocent core plugins like Git/Github/etc
I'm not talking about plugins in general. Strider uses a dozen of plugins, how many of them are core part of the application so it won't work without them? I'm suggesting to integrate the core so we all can focus on refactoring the apis, REST, decide on UI framework instead of making everything backward compatible.
Vitaly, In your comment you mention backwards compatibility and refactoring APIs as a motivation for integrating the core.
I am in favour of continuously improving Strider, refactoring APIs and increasing consistency, but I don't see the connection between these things and an integrated core.
I'm sure you have deeper reasons - but please expand on your position and argument. From where I sit, these things are orthogonal.
Here are the Strider plugins which come by default:
strider-bitbucket strider-build-badge strider-cli strider-custom strider-email-notifier strider-env strider-extension-loader strider-git strider-github strider-gitlab strider-heroku strider-mailer strider-metadata strider-node strider-python strider-ruby strider-simple-runner strider-slack strider-ssh-deploy
With the exception of strider-extension-loader, none of them are really "core" to the application. You could argue that strider-extension-loader should be a part of the core repo but I went with the node.js tradition of "lots of small modules". Maybe "strider-cli" should be a part of core too, but I'm not sure.
Strider is specifically about a small core, and lots of plugins - each one that does a small things well - the same philosophy as node.js and UNIX. This is a fundamentally different approach/philosophy to a monolithic application.
There is nice a symmetry to this design, in that Strider itself is "just a bunch of Strider plugins" (which are actually just npm modules with some extra metadata!)
I suspect many of the contributors are attracted to the project specifically because they appreciate this design.
Now, I agree there is overhead to this (managing many repos and modules, versioning etc, API friction) that doesn't exist in a monolithic application. However, software is about tradeoffs. The tradeoff Strider took was to be highly modular.
I am strongly against expanding the scope of Strider core without very good reasons.
On Fri, Dec 19, 2014 at 9:59 AM, Vitaly Sukhomlinov < [email protected]> wrote:
I'm not talking about plugins in general. Strider uses a dozen of plugins, how many of them are core part of the application so it won't work without them? I'm suggesting to integrate the core so we all can focus on refactoring the apis, REST, decide on UI framework instead of making everything backward compatible.
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-67673156.
Niall O'Higgins W: http://niallohiggins.com E: [email protected] T: @niallohiggins
I am totally into the idea of creating a separate Strider Ember/SPA repository that consumes Strider as an API endpoint. In my opinion, this takes Strider in a good direction.
+1 for "single-page-app" in separated project (or projects) as that way you could connect various different clients or types of clients, all talking to a clear REST interface.
I also think the cleanest way to incrementally migrate to a new frontend is with a separate project talking to REST backend.
I primarily use Spine for SPAs but out of Ember, React, Backbone, etc, Angular is last on my list. Too much magic, logic in the view, and not modular enough. However, with Strider's very modular plugin architecture, there should be a data-binding standard to have a sane way to manage backend updates. Ember's component focus could work well here.
@doublerebel I've already started on a new client written in Ember. Would love for anyone to join in, I can teach about Ember if you're new as well.
I'm also cleaning up the backed for strider, to make it a proper API.
I would just like to open a discussion on using bootstrap. Other frameworks, like getuikit.com are both smaller, cleaner in code, and more feature complete (Compare the gridsystem of uikit and bootstrap), and they come with several advanced form elements (date/time- picker was mentioned).
I've yet to see a good alternative for Bootstrap that is so well supported and handles mobile so well. Btw, we are sing an old version, and the plan is to upgrade. I would be open to using something else I I saw something worth while that actual compared.
I would argue that UIkit (it follows 'mobile first') handles mobile better then Bootstrap. The grid system is a good example of that. And as for support, it's developed by a company with long experience with making themes for joomla and Wordpress.
The uikit grid system works like this:
<ul class="uk-grid uk-grid-width-1-1 uk-grid-width-medium-1-2 uk-grid-width-large-1-4" data-uk-grid-margin data-uk-grid-match="{target: '> li > .uk-panel'}">
<li>
<div class="uk-panel">lorem ipsum</div>
</li>
<li>
<div class="uk-panel">lorem ipsum</div>
</li>
<li class="uk-width-medium-2-2 uk-width-large-2-4">
<div class="uk-panel">lorem ipsum</div>
</li>
<li>
<div class="uk-panel">lorem ipsum</div>
</li>
<li>
<div class="uk-panel">lorem ipsum let's make an extra high panel here.</div>
</li>
<li>
<div class="uk-panel">lorem ipsum</div>
</li>
<li>
<div class="uk-panel">lorem ipsum</div>
</li>
<li>
<div class="uk-panel">lorem ipsum</div>
</li>
...
</ul>
On desktop: a 4 column layout where each row is height matched individually. second row will automatically get a margin top for gutter.
On tablet: a 2 column layout that is automatically wrapped. In the above example row 3 will be higher then the others due to the larger uk-panel. each row after the first will get a margin top for gutter.
On mobile: A single column where each item get a margin top gutter, and no height matching.
The third item will be a single row on tablets, and half a row on desktops.
In bootstrap you have to wrap each row manually, and then 'fix' the other resolutions....
Another example is their tab module that will automatically collaps the tabs on to a single tab with a dropdown menu if the available width is too small to contain the tabs. Prefect for mobile. In an Angular.js context... Bootstraps tabs require the ui-bootstrap module for the tabs to work... uikits tabs works out of the box with angular.js.
Here is a size comparison between uikit and a number of frontend frameworks: https://yootheme.com/component/blog/2013/08/13/how-big-is-uikit (might be a bit outdated now).
+1 for ditching angular ...
I mean, listen, I don't want to step on any toes here ... but Angular really made for a no fun Strider experience for me and my team. None of us knew Angular and its SUPER opinionated. I mean, seriously, do we have to learn Angular just to create our own plugins?! We finally gave up on it and just started circumventing all of things Strider is good for. We HAD to, we were burning through soooo much time.
Angular is popular and as I understand it, really great... but so are lots of other things... like Polymer. I mean, actually, keep Angular if you want, just show us how to make plugins without using it and we're fine. Just let us do simple HTML or whatever.. or, if you need to, cram a template engine down our throats... but please don't run with a super crazy revolutionary new anything. Keep it simple, easy as pie, make a form ... jQuery if you need it ... easy and simple.
As for the talks about front-end frameworks... we did this dance as well, for a long time. I've used most of the ones that most people know by now and there isn't a lot going on for any of them that isn't going for all of the others. Its kinda the same ole thing, no matter which way you go. Semanti-UI looks the best (to me) .. but otherwise its also the same.
However ... we ended up going 100% full-on Bootstrap because of how prolific it is. As an app developer you probably don't want to have to care about skinning or whatever if you don't have to. Bootstrap has HORDES of drop-in themes that users can download and it all just works. Like Bootswatch .. simple, free, easy, done ..
Go Bootstrap ..
Edit: In reading the comments above, also...
- Semantic UI might handle mobile better, but Bootstrap handles it perfectly so I really don't see how it could get any better. Bootstrap makes it easy and it works flawlessly for anyone who knows anything at all about responsive design or grid systems.
- I really don't think that static asset/resource size is THAT big of a deal on a CI server. Even if it were 10MB, it would be a mild inconvenience for some of us some of the time. Skip it, skip the whole discussion, and take the easy road.
How about just cleaning up and publishing a REST api for a start. Then various clients would work.
:+1: for React and their friends:
- React Router.
- Reflux or Redux for Flux stack.
- Webpack.
And for backend Meteor or Sails (which I like it more) and they have realtime features.
I'm also very pro API-first. The JS framework discussion will continue endlessly and the better Strider can support whatever you want to interface with sounds best to me. 2 new MV* JS frameworks were published by the time you finish reading this.
On the subject of REST API features I would really like to see the job logs accessible via an endpoint. Maybe with a call that has a ?since=101 param to get only the log from line 101 onward. Then this could support polling to fill the log out on the job pages. I'm finding the socket.io setup quite finicky and am sure it's what churns CPU when you leave job tabs open. On my 2012 MBA I get tab crashes in chrome if I watch a job thats putting out a lot of text quickly.
the parsing of live text streams for ANSI control characters is surprisingly slow unfortunately:-/
On Thursday, July 9, 2015, Kyle Newsome [email protected] wrote:
I'm also very pro API-first. The JS framework discussion will continue endlessly and the better Strider can support whatever you want to interface with sounds best to me. 2 new MV* JS frameworks were published by the time you finish reading this.
On the subject of REST API features I would really like to see the job logs accessible via an endpoint. Maybe with a call that has a ?since=101 param to get only the log from line 101 onward. Then this could support polling to fill the log out on the job pages. I'm finding the socket.io setup quite finicky and am sure it's what churns CPU when you leave job tabs open. On my 2012 MBA I get tab crashes in chrome if I watch a job thats putting out a lot of text quickly.
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-120180667.
Niall O'Higgins W: http://niallohiggins.com E: [email protected] T: @niallohiggins
Would love to see if there are better parsers, or if iojs has moved forward in this regard. Maybe even splitting out the phases some how so that the number of DOM elements isn't a big issue.
Talks about rewriting certain aspects of Strider, just to arrive at the same point with a different technology, seem misplaced.
Not being able to write a plugin for Strider, because you don't know Angular, is not a reason to rewrite Strider; it's a reason to learn Angular.
If you want to rewrite part of an application, you should have a better reason than "It's written in X and I don't like X, we should use Y, because I like Y".
Every framework sucks. If you don't think that the framework/tollkit/library/… you're using sucks, then you haven't been using it long enough.
@shaunc
How about just cleaning up and publishing a REST api for a start.
I agree completely, its hard to overstate this. If there's anything that practically every person in this thread shares, its knowledge on using REST.
@oliversalzburg
Not being able to write a plugin for Strider, because you don't know Angular, is not a reason to rewrite Strider; it's a reason to learn Angular.
Agreed. I only posted this because a "rewrite" was already on the table, seemingly, from the comments above mine. I would not suggest that Strider should begin refactoring to ease my personal pain, but if someone is going to rewrite it, then please consider not using Angular on the next round.
If you want to rewrite part of an application, you should have a better reason than "It's written in X and I don't like X, we should use Y, because I like Y".
Again, I agree, mostly, and I tried to make that point (from a different angle). To reconcile my statements with yours, all we need to do is agree that HTML, CSS, and JS are, in themselves, neither X, nor Y, nor any other letter. Angular might be X, React is Y, and Polymer is Z.. and I think Strider should use "none of the above" and make a plugin system that uses the fundamental technologies that everyone already knows how to use.
Also, my critique in this regard is limited to only the plugins. Strider can use a mixture of every client library out there for all I care, I just want to be able to write plugins without needing to add things to the already conflated "qualification requirements" we use to hire people. Strider plugins are, at their core, very simple devices.. and it seems inefficient to "invent" complexity by layering on alien technologies. Just do <form action="/config/myplugin" method="POST"> and walk away.....
Strider is precisely on the right track when it emphasizes plugins as a way to extend the product without requiring direct effort by its core team. This is one of the golden ideas of our age, in some ways. However, that is, at minimum, weakened by opinionated libraries like Angular.
I think its much worse than the minimum and if the Angular requirement can be removed, I expect Strider to take off to the levels of the other major CI/CD solutions on the market. I can at least be certain that we could (and would) share our ~10 or so unique Strider plugins with the world if we could ditch Angular. We only avoid sharing because we do not follow the community practices and guidelines.
I can at least be certain that we could (and would) share our ~10 or so unique Strider plugins with the world if we could ditch Angular. We only avoid sharing because we do not follow the community practices and guidelines.
Share them anyway, and the community will come through, otherwise everyone misses out.
Sure, we have them bundled as a single plugin.. the reason why will be a little more clear when you see it. As I said, we "circumvent" Strider in a particular way. I'll mirror the plugin to GitHub as soon as I can.
Easy to talk, but if anyone wants to start doing something, I have just the thing. Documenting the existing API. I've setup apidoc in this commit https://github.com/Strider-CD/strider/commit/f3c0ad2b3cea73d0c8ab4f7c606c26d23cb17aad and would love all the help to document our existing API.
Even though I might repeat what other people already mentioned: I think strider should increase its modularity. What I mean by this specifically is:
- Currently, Strider-CD/strider contains way to much functionality. In my opinion it would be good if strider itself would just be a central communication hub for plugins.
- Plugins itself should be cleanly decoupled from each other (a syntax error in a plugin or an exception must not be able to crash the rest of the system). This can be achieved by some kind of networked API and with a well defined data exchange format (I'd like to see something like ProtoBuf instead of JSON here). This would also:
- increase scalability (computing heavy plugins could run on a dedicated machine etc.)
- allow plugins to be written in arbitrary languages
- Frontend components should get no special treatment. They should also be plugins.
Additionally:
- I saw a lot of discussions about the frontend in this issue. I think it is way more important to have a stable and scalable backend first before we even talk about frontend stuff.
- By looking at the test coverage I'd say strider 2.0 should be architecturally designed in a way which makes it easier to test. What I mean by this is that people tend to write tests for stuff that is easy to test. If some architectural decision makes it hard to test certain functionality developers will typically not write tests for this functionality. BTW: using a networked plugin API with a well defined data exchange format would help a lot in this regard (IMHO).
- The cleanest way to do all this would be probably to write strider 2.0 from scratch (reusing code from current strider wherever possible).
I agree about everything, but not sure about ProtoBuf. I feel like that would hinder community involvement since JSON is well understood but not so much for ProtoBuf. Although I totally see the benefits of using that format.
The other concern is man-power. We don't have too many active contributors, but maybe a rewrite will solve that (developers love new things).
I would love to do a combination of json api and ProtoBuf. I think it will make it simple to have a nice convention and have the speed of ProtoBuf. In the end any kind of client could consume, since implementations are being written for many clients for JSON API.
If anyone is interesting in starting writing the API from scratch, let me know, and we can start a specific brain-storming issue (or a repo right off e.g. strider/api and discuss there).
I've really enjoyed reading this thread! Thanks all!
Ilya, I am very much for the idea of writing a new API from scratch.
As an aside, I currently use Buildkite for CI and I love that by design it does very little. There is an agent (open source and cross platform) that you must install on your workers, and then point them at the orchestrator (buildkite's closed-source cloud software). For example I have two types of workers... A digital ocean amd64 instance tests, builds, deploys my web code. An ARM beaglebone tests, builds, and uploads binary artifacts of my 'embedded' code.
As far as tools... I have been on a meteor stint for some time now. I have delivered several projects in meteor, and used it for a startup I'm currently working engaged in. However as my application grew more complex, the thing became slower and slower and the pub/sub stuff was getting in the way more often than helping. I scrapped it and rewrote...
I used hapi and sequelize with a postgres adapter. It has been so nice and such a breath of fresh air. I really enjoy hapi's simplicity and I missed SQL's predictability, maintainability, and enforcement of documented integrity of the underlying schema.
I have been using RabbitMQ (via CloudAMQP) and it's been just great for delivery work to downstream IoT devices. It's a good technology for delivering work. Redis and others are good too, but it was just interesting to me that I had been using Redis for pub sub so long (since I learned Rails) but that AMQP was what I was really after most of the time... I would keep that in my arsenal when designing a new CI...
As for frontend, my post-meteor development has been pretty much using AngularJS with proper usage of angular-ui ui-router and then calling back to Hapi with $http. Angular itself has been OK, but it is complicated and relatively hard to explain to people. I don't even fully understand ui-router and try to use it very minimally.
What I do agree with very much is that the core should not include UI -- if a new design can pull this off well I think it could be very special. I can imagine having a plugin that provides a base UI for those that want it. That plugin could easily be forked and have features added to it or use a totally different frontend framework or whatever.
That said, React seems quite interesting and I do believe (like we sorta have now in strider) in the idea of providing an easy way to generate plugin scaffolding. In it we can pre-bake some React boilerplate or Angular boilerplate, etc.
Also, would these plugins run on the orchestrator or on the worker? Strider currently does not have a super clear separation between orchestrator and worker and I think that was both a strength and a weakness. It was a strength because you got up and running quickly, but it was a weakness because it was difficult to figure out how to set up strider to actually deliver jobs down, to say, a beaglebone to produce ARM builds.
That said, React seems quite interesting and I do believe (like we sorta have now in strider) in the idea of providing an easy way to generate plugin scaffolding.
Sounds like a good plan. Write them as they come. I think that Yeoman is still a good option in this regard for writing something like that. It will allow for shared knowledge and a good platform for that exact use-case.
I used hapi and sequelize with a postgres adapter. It has been so nice and such a breath of fresh air. I really enjoy hapi's simplicity and I missed SQL's predictability, maintainability, and enforcement of documented integrity of the underlying schema.
I've yet to use Hapi. What does it bring to the table that Express doesn't? Same in regards to sequelize vs knexjs (which is what I've used in the past). I agree on postgres, and if we need json storage for plugin data, that can be done with the json column.
I know you like new things Keyvan :wink: Have you looked at rethinkdb (nosql)? It supports push notifications for an arbitrary query.
Also, would these plugins run on the orchestrator or on the worker? Strider currently does not have a super clear separation between orchestrator and worker and I think that was both a strength and a weakness. It was a strength because you got up and running quickly, but it was a weakness because it was difficult to figure out how to set up strider to actually deliver jobs down, to say, a beaglebone to produce ARM builds.
Plugins might have to be two faceted, a core portion and a drone portion. I can see some plugins implementing one or both, it would depend what the plugin needs to do I think. The drones can communicate back to core and report on what plugins they have, and those could be configured through core (or locally on the drone). Drone plugins could be simple node modules with a basic API, so that users can reuse the plethora of build/testing tools already out there.
I think we can start on a rough design on how this new system should work, and lay down the non-negotiables, and we can talk technology stack after.
I haven't looked at rethinkdb enough to comment on it.
The main thing that I like so far about hapi versus express has been the way you define routes. Routes are just objects that get passed into the server.route method. The method can take an object or an array of objects (routes), and the route object can contain keys that are used by other plugins or parts of hapi (e.g. using Joi to validate incoming data). This way you find yourself writing configuration objects most of the time, not so much functions that call other functions.
I also like the way 'auth strategies' work in Hapi -- it just makes sense to me. With express I vaguely remember doing some things with passportjs or something. On the other hand I might just be improving as a coder in general -- but I am pretty sure hapi has been a better toolset overall and it's not just me.
I don't know anything about knexjs, but I can say that sequelize has been well-documented and easy to use. I like how it has a tool for generating migration files and running them, I've been using it in production and doing migrations with it without any issues.
But anyway I agree with you about determining the non-negotiables, having an overall design we would like to achieve, and then looking at tools later. I'm pretty swamped with my current engagements but I will keep an eye on the new stuff :)
Multiple build environments and Simultaneous builds are a must-have for devops. If it's as easy to test that a whole infra on a superior version of
100% agree that we need true multiple build environments (this cannot be simply based on branches, it just doesn't work to tie the VCS into this).
I'd like more info about what blue green deployment is @jpic if you don't mind
Sure, is it good enough if I cite Fowler's article about BlueGreen ?
One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime. The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.
Blue-green deployment also gives you a rapid way to rollback - if anything goes wrong you switch the router back to your blue environment.
Guys, our 2c... Over at Panubo we are using Strider solely for deployments to CoreOS (Docker / Fleet) clusters. We chose Strider because it seems like a good lightweight, flexible tool that is focused on CD, rather than CI duties...
To get Strider to handle CD tasks we've needed to write a bunch of different tools and even a Docker container to host it all.
With all this we get an atomic (rolling) deployment capability. And our customers get visibility via a nice web UI.
But right now it's tricky to configure as everything must be configured manually. We are using the GitHub / BitBucket plugins (depending on customer), Custom Script runner, and Meta data plugins along with our tools above. Some of our customers have over 100 projects. So we envisage that we might need to write our own plugin to make this easier.
Given our usage, our wish list for Strider 2.0 would be:
- Rollbacks: We'd love to see the ability to replay a deployment for the purposes deployment workflow and recovery. For this I think the Strider core needs a concept of deployment artefacts and associated metadata that can be replayed verbatim, or reapplied with variables overridden.
- Focus on 'Continual Deployment'. (For most of our projects the deployment is the key, and the testing phase is not even required). By the time the code hits the appropriate branches in Git, it's already tested. And any secondary testing is done at the container level. I guess for this requirement, we'd love to see Strider take a more generic approach to 'CD', and not make steps mandatory. Ideally we'd be able to define the steps in a generic fashion, so we could adjust things to suite our customer's workflow.
- Ease of adding large numbers of new projects. As above, some of our customers have a lot of projects.
- Multithreading. It should be possible to build and deploy multiple projects at the same time.
Cheers, Andrew
After manually running deployment scripts for far too long, I started looking into CI applications and decided on Strider, largely because it runs on Node and seems a lot less complex to set up than Jenkins or whatever.
Since it's an easily-accessible portal to our private cloud, I've toyed with the idea of using it for more than just CI - like running database backups or running grunt tasks in arbitrary directories.
That being said, I like @bitwit's suggestion of supporting more plugin types. If things were set up correctly, that functionality could be accomplished via plugin.
I also think it makes a lot of sense to strip things down to just a REST API, and maybe have a separate repo for the "stock" UI, which, itself, could be a plugin orchestrator.
Given @macropin's comments about continual deployment, and my personal goal of using this for things other than CI, I think it would be good to get away from the idea of "builds" and think about generic, nestable "tasks". (Something like this is also mentioned in #233) So from what exists now, the "deploy" task would contain a "test" task and a final task to actually push files and such. The "test" task would contain a task to copy code, install code, and run the tests. Plugins could then publish their own tasks that could be "dragged" into other tasks to add them to the workflow. Other plugins could consume these tasks as well, paving the way for a generic task scheduler, etc. Is this idea too crazy?
Is this idea too crazy?
No, I think this is really the way we want to go.
Feel free to leave ideas in the core repo.
definitely. the data model is structured around jobs not builds per se. at least that was my goal :)
jobs can do more than just build and push code and can be triggered by different events - not just VCS events.
On Wednesday, August 19, 2015, Ilya Radchenko [email protected] wrote:
Is this idea too crazy?
No, I think this is really the way we want to go.
— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/667#issuecomment-132632777.
Niall O'Higgins W: http://niallohiggins.com E: [email protected] T: @niallohiggins
Hello @knownasilya ,Is the 2.0 development plan still going on? I use strider inside the enterprise, it is very good, but it seems that it is very necessary to upgrade, because it frequently appears BUG after the frequency is increased. I originally planned to submit support for the corresponding Tag push event for it, but found that there were more changes. If this plan is still going on, I am willing to participate.
