Support polling strategy for fully secured Jenkins boxes

[ppg]

If I want to have my Jenkins box totally secure so nothing external is hitting it I'd like to use polling instead of a webhook to trigger jobs, it'd be nice to have that as a feature/choice.

Two related questions:

1. Are there any workarounds to get webhooks working if I don't want to give Anonymous overall read access?
2. I noticed that DotCi seems to be circumventing Jenkins Git plugin for getting code, polling, etc. Why is that? It seems like I cannot just setup polling, because that would always build master on any change Jenkins Git plugin detects; plus DotCi seems to do funny deploy key stuff to be able to directly pull code, why not leave that up to the Git/Github plugin and just use the code it pulls to test?

[suryagaddipati]

1. Are there any workarounds to get webhooks working if I don't want to give Anonymous overall read access?

I haven't really researched this issue much. Perhaps there are solutions/workarounds for this ? Open to suggestions here. https://github.com/groupon/DotCi/issues/42

2. I noticed that DotCi seems to be circumventing Jenkins Git plugin for getting code, polling, etc. Why is that? It seems like I cannot just setup polling, because that would always build master on any change Jenkins Git plugin detects; plus DotCi seems to do funny deploy key stuff to be able to directly pull code, why not leave that up to the Git/Github plugin and just use the code it pulls to test?

We run a fairly large jenkins installation and the suggestion at the time was to not use polling for efficiency reasons http://kohsuke.org/2011/12/01/polling-must-die-triggering-jenkins-builds-from-a-git-hook/. Perhaps this has changed recently?

Git plugin doesn't support private repos AFAIK and the funny deploy key stuff happens only for private repos( despite it showing up on logs).

I am not quite sure what it would take to optionally use git plugin instead of hooks.

[suryagaddipati]

I am leaving this issue open. I would like to research 1 at the minimum.

[ppg]

I think polling is still definitely not as efficient as the webhooks, the push vs. pull model will always be superior. But I guess it depends on what 'large' means. I've seen it work fine (with the normal limitations of polling) with up to around 100 projects or so; github can handle the polling, its Jenkins that gets its (presumably) single threaded poller hung up. All that being said, if you can't and/or don't want to expose Jenkins to the public you don't really have much choice. I.e. the webhooks are a public entry point into Jenkins, and so hopefully they're hardened against injection attacks, etc. but I'm personally not super confident Jenkins with respect to that, and when we talk about the plethora of 3rd party plugins I get even less confident.

Side note, if anyone is reading this and didn't know you should use Jenkin's slightly altered cron syntax like H/5 * * * * for polling which says poll every 5 minutes, but randomly pick a time within a 5 minute window (instead of always at 0, 5, 10, 15, etc.); that helps disperse the polling of lots of projects.

The git plugin (or one of its extensions) definitely supports private repos, our old (non-DotCi) Jenkins uses it and polling and pulls from private repos. It I think forces the use of a global deploy user who has a key, but it'll pull the code just like normal. That would actually be a nice feature for DotCi too: I do like the per repo deploy keys, but it'd be nice to instead use a deploy user and have that do the pulling. I think really what my complaints/issues here are boiling down to a couple things:

1. DotCi uses a bunch of methods to checkout code, they're slightly different, they're different than Jenkins, and they aren't easy to inherit in new build types: https://github.com/groupon/DotCi/blob/4546e8b5613385e9eb7760403e644ac661015fbd/src/main/java/com/groupon/jenkins/buildtype/docker/CheckoutCommands.java#L33-L51 https://github.com/groupon/DotCi/blob/4546e8b5613385e9eb7760403e644ac661015fbd/src/main/resources/com/groupon/jenkins/buildtype/install_packages/template/Base/.ci.yml#L33-L57 https://github.com/groupon/DotCi/blob/7e56aee763708e20dc97b8715b190c713e0e7aa1/src/main/java/com/groupon/jenkins/buildtype/dockercompose/DockerComposeBuild.java#L88-L117 Maybe CheckoutCommands is what is supposed to be used? It seems like there's an opportunity here for a Base BuildType that will automatically do code checkout, and allow plugins and notifications configuration out of the box, so people are more altering the repo specific data specification in the yaml file instead of some of those more core concepts.
2. I'll have to poke around, but I'm thinking (hoping) that the git plugin has either some form of class that can generate/do the git checkout commands, or perhaps even better a way to expose on the Project object the git key, and let Jenkins handle the actual git commands to get the directory ready. That way there'd be SCM sync'ing that is handled by the git plugin, where it likely belongs. And then DotCi, just like someone manually writing in job steps, can depend on the projects SCM configuration to ensure files are in place when it starts its execution.