403 on Webhook

[bc24fl]

I'm having issues getting my web hook to work. On Caddy start it does pull the latest code from bitbucket as expected. But for some reason pulling using the webhook is not working. I'm thinking that the webhook should never reach the rewrite rule, no? The key file should be working because it is pulling on container start, just not on code push.

Any help would be much appreciated. Here is my info:

CADDYFILE VERSION 0.8.2:

mydomain
tls myemail
#browse
fastcgi / 127.0.0.1:9000 php # php variant only
startup php-fpm # php variant only

git {
       repo    [email protected]:mygitrepo.git
       branch   master
       key      /root/.ssh/id_rsa
       path     ../../src
       hook     /_webhook
       then    cp -pR /srv/src/laravel /srv/
       then    sh -c "cd /srv/laravel && composer update"
       then    sh -c "cd /srv && chown -R nobody.nobody laravel"
}

root /srv/laravel/public

log access.log
errors error.log

rewrite {
    if {file} not favicon.ico
    to {path} {path}/ /index.php?{query}
}

BITBUCKET SET:

Under integrations and Webhook i have the following hook: https://mydomain/_webhook

ERROR from bitbucket:

Response from https://mydomain/_webhook HTTP status: 403 Elapsed time: 1590ms Request time: 7 minutes ago (Tuesday, April 26th 2016, 3:56:10 pm)

[abiosoft]

Caddy 0.8.3 just got released. Can you upgrade to 0.8.3 and try again.

And yes, it is definitely not a key issue if it pulls the first time.

[abiosoft]

Hi, any success ?

[bc24fl]

Same 403 error. Not sure that it matters but i'm running Caddy inside a docker container. See conf below:

FROM alpine:3.2
MAINTAINER Abiola Ibrahim <[email protected]>

LABEL caddy_version="0.8.3" architecture="amd64"

RUN apk update && apk add --update openssh-client git tar php-fpm mysql-client php-phar php-openssl php-xml php-mcrypt php-curl php-dom php-gd php-json php-pdo_mysql php-pdo_sqlite php-iconv php-ctype php-common php-cli php-bcmath php-imap php-memcache php-ldap php-soap php-dba php-mcrypt php-xml php-dev php-mysqli php-mysql php-xmlrpc php-fpm php-pdo php-zip php-json

# change timezone to eastern
RUN apk add tzdata && cp /usr/share/zoneinfo/US/Eastern /etc/localtime && echo "US/Eastern" > /etc/timezone && apk del tzdata

# allow environment variable access.
RUN echo "clear_env = no" >> /etc/php/php-fpm.conf

RUN curl --silent --show-error --fail --location \
      --header "Accept: application/tar+gzip, application/x-gzip, application/octet-stream" -o - \
      "https://caddyserver.com/download/build?os=linux&arch=amd64&features=git" \
    | tar --no-same-owner -C /usr/bin/ -xz caddy \
 && chmod 0755 /usr/bin/caddy \
 && /usr/bin/caddy -version

RUN wget -O /root/installer https://getcomposer.org/installer && php /root/installer --install-dir=/usr/local/bin --filename=composer

EXPOSE 80 443 2015
VOLUME /srv
WORKDIR /srv

ADD Caddyfile /etc/Caddyfile
ADD phpinfo.php /srv/phpinfo.php

ENTRYPOINT ["/usr/bin/caddy"]
CMD ["--conf", "/etc/Caddyfile"]

[bc24fl]

Is there a debug option i can set or log I can view? I see nothing related in the error.log that Caddy produces.

[abiosoft]

Nothing really. Except you are willing to mingle with the source.

I wish BitBucket shows the response body.

[bc24fl]

BitBucket is showing a response of 403 Forbidden.

Response from https://mydomain/_webhook HTTP status: 403 Elapsed time: 181ms Request time: 36 minutes ago (Tuesday, May 3rd 2016, 7:28:52 pm)

Headers content-length 13 keep-alive timeout=15, max=100 server Apache connection Keep-Alive date Tue, 03 May 2016 23:30:40 GMT content-type text/plain; charset=utf-8

Response Body 403 Forbidden

[bc24fl]

I'll have to remove all variables (ie: docker, waf, etc..) to rule out anything other than Caddy & Caddy-Git. I'll keep yer updated.

[abiosoft]

I have attempted to reproduce this without success.

I hope you are not behind a reverse proxy that changes remote address before it gets to Caddy. That error is usually due to a request from a non bitbucket ip.

[diegobernardes]

I'm with the same problem. Caddy 0.8.3, running inside a docker, behind Cloudflare and using Bitbucket. All webhook requests return 403. It is exact the same thing as @bc24fl , when the process starts it do the 'git pull', download new commits, etc.. but, when bitbucket try to call the webhook, caddy return a 403.

Anything new on this? Got to caddy only because of this feature :'(

[abiosoft]

Could Cloudfare play a role in this ?

[diegobernardes]

Don't know. Already tried to disable the HTTP processing and use Cloudflare just as a DNS and still get the same error. Tomorrow gonna try to use the DNS from DigitalOcean to see if resolve the issue.

[regalstreak]

Cloudflare is the issue. Idk but i used it just as a DNS and still it does not work. So digitalocean ftw for DNS management here.

[saward]

I don't know if this is the same issue, but I am unable to use the Webhook via bitbucket. The error inside the docker container is:

17/Sep/2016:05:50:29 +0000 [ERROR 403 /webhook] the request doesn't come from a valid IP

Since these requests are routed via a load balancer, or perhaps from the host to the container, my guess is that the external IP does not match the bitbucket ones that are saved in https://github.com/abiosoft/caddy-git/blob/master/bitbucket_hook.go. Either that, or bitbucket has more IP's than just the ones listed.

[abiosoft]

@saward what's the proxy configuration in your Caddyfile ?

[regalstreak]

You should be using the ip instead of the domain if you do not want these cloudflare issues to be popping up.

[saward]

@abiosoft Here's my entire caddy file:

# https://caddyserver.com/docs/caddyfile
0.0.0.0:2020
log stdout
errors stderr

root /home/caddy

# Clone a public repo once/hour.

git bitbucket.org/<removed> {
    hook /webhook
}

[TotalLag]

Hi,

I'm running into the same 403 error as well when using bitbucket. I've tried with Cloudflare and without Cloudflare, even accessing just the IP of the server and it still fails. The error I get is:

1/23/2017 11:15:10 PM104.192.143.194 - [24/Jan/2017:05:15:10 +0000] "POST /update HTTP/1.1" 403 38

EDIT: I've found the fix for myself after accidentally stumbling across the "realip" plugin. I've updated my Caddyfile below to reflect what I changed to get it working.

@diegobernardes @bc24fl @saward can you try with the realip plugin? https://caddyserver.com/docs/realip

My Proxy Caddyfile:

http://staging.<domain>.com http://<ip>/site {
    proxy / staging-test:80 {
transparent
}
    gzip
    tls off
    log stdout
    errors stderr
    header / Cache-Control "max-age=0"

    #### THE FIX ####    
    realip cloudflare
    ###############
}

My site Caddyfile:

0.0.0.0:80
root /var/www/html
gzip
fastcgi / phpfpm:9000 php
log stdout
errors stderr

#### THE FIX ####
realip {
    from 10.0.0.0/8
}
###############  My docker network manager creates containers on the 10.x range

status 403 /forbidden

git {
    repo https://<username>@bitbucket.org/<path>.git
    path /var/www/repo
    key /deployment_key.rsa
    hook /update
    hook_type bitbucket
}

# Begin - Security
# deny all direct access for these folders
rewrite {
    if {path} match /(.git|cache|bin|logs|backups|tests)/.*$
    to /forbidden
}
# deny running scripts inside core system folders
rewrite {
    if {path} match /(system|vendor)/.*\.(txt|xml|md|html|yaml|php|pl|py|cgi|twig|sh|bat)$
    to /forbidden
}
# deny running scripts inside user folder
rewrite {
    if {path} match /user/.*\.(txt|md|yaml|php|pl|py|cgi|twig|sh|bat)$
    to /forbidden
}
# deny access to specific files in the root folder
rewrite {
    if {path} match /(LICENSE.txt|composer.lock|composer.json|nginx.conf|web.config|htaccess.txt|\.htaccess)
    to /forbidden
}
## End - Security

# global rewrite should come last.
rewrite {
    to  {path} {path}/ /index.php?_url={uri}
}

[pdinoto]

I have been hit by this same issue, but with a different use-case:

I am running an instance of Caddy in a docker container to act as :80 and :443 front-end (in order to host several Let's Encrypt-enabled sites under a single IP). Each site the is behind this front-end on different container, and the front-end does all the TLS magic, while proxying to the internal instances with tls off.

Some of the sites would benefit from this http.git plugin, and I could not make it work.

Clearly, the webhook requests always came from the front-end dockerized Caddy instance, not from the expected IPs of the repository. Using proxy /webhook { transparent } did not help, as the plugin seems not to pay attention to {>X-Real-IP} nor {>X-Forwarded-For}.

The :arrow_up: workaround by @TotalLag allowed me to use the plugin in this use case.

:bulb: An interesting issue with logging made me spend a lot of time focusing on the proxy instance while debugging: The POST request that arrived to the internal server was silently responded with 403, not logging it. So each time the webhook was fired, I only saw it on the proxy, but not on the internal server as other requests.

[abiosoft]

@pdinoto good to hear that you have a workaround.

Should the webhook handler worry about {>X-Real-IP} and {>X-Forwarded-For} ? I thought it is out of scope for it and it is better to leave that to a plugin/directive that is designed to that e.g. realip.

At least for now, we can recommend users to use realip workaround.

[bilalakil]

Howdy, I'm experiencing 403s from Bitbucket, but (as far as I know) am not sitting behind any proxies, etc. I'm running Caddy in a fresh DigitalOcean droplet with the following Caddyfile:

<domain>

gzip

tls {
        dns digitalocean
}

git <repo> {
        pull_args -s recursive -X theirs
        hook /__try_pull
        hook_type bitbucket
}

Caddy's response to a repo:push from Bitbucket looks like:

Date	Sun, 30 Dec 2018 13:14:15 GMT
Content-Length	14
Content-Type	text/plain; charset=utf-8
X-Content-Type-Options	nosniff
Server	Caddy
Body	403 Forbidden

Having read the above discussion, I'm not really sure what I'd do with realip to attempt a workaround.

[joebillings]

I'm in the exact same boat as @bilalakil. I've got a few servers all hosted on Digital Ocean, no proxies or CDNs (that I'm aware of), all with git repos on BitBucket and they're all returning 403 when Bitbucket fires the webhook.

[abiosoft]

This has been fixed with https://github.com/abiosoft/caddy-git/issues/94.

[SeaLife]

Heyho, as this should be fixed by #94, i tried this now with a newly downloaded caddy-binary with http.git enabled and im still getting a 403 from Bitbucket (self hosted)

Response Headers (in BitBucket)

Server: Caddy
Content-Length: 14
X-Content-Type-Options: nosniff
Date: Thu, 18 Apr 2019 11:15:38 GMT
Via: 1.1 localhost (Apache-HttpClient/4.5.5 (cache))
Content-Type: text/plain; charset=UTF-8

Response Body:

403 Forbidden

Caddy-File:

http://internal.ex.lan {
  tls off

  on startup docker start tasker

  log access.log
  errors error.log

  header / Cache-Control "max-age=0"

  realip 10.0.0.0/8

  git {
    repo https://bitbucket.ex.lan/scm/~sealife/repo.git
    path /applications/tasker
    branch master
    hook /webhook
    hook_type bitbucket
  }

  proxy / localhost:25100

  on shutdown docker stop tasker
}

Tried adding "realip" but im not using a proxy in front of Caddy at this point (BitBucket contacts Caddy directly)

[abiosoft]

@SeaLife since yours is self hosted, that means your IP will not be included in Atlassian's (https://ip-ranges.atlassian.com).

Looks like we'd have to either allow users to specify IPs or make IP verification optional.

[SeaLife]

I read somewhere something about a "hook_ips" setting which would be really helpful - unfortunate i dont know Go enough to implement it :( Probably adding 10.0.0.0/8 would help also, cause this is used in all the major companies who are self hosting their stuff? 🗡