Automated builds based on travis

[captn3m0]

Love the idea and project. I was wondering if it makes sense to move the build/deploy process to travis instead? I do the same at hackercouch and it currently runs the deploy on every commit to master.

In order to keep it updated, we also use nightli.es as a backup. Thinking if would be possible to pipe the github event for issues.* on the repo to trigger a build in travis.

[jayfk]

That's a great idea!

Thinking if would be possible to pipe the github event for issues.* on the repo to trigger a build in travis.

Any idea if that's possible? I did a quick search through the official documentation but couldn't find anything.

[captn3m0]

We can do the following:

1. Ask users to create a new webhook to heroku(?). I remember I saw some service that does jekyll builds on webhook notifications
2. This tiny service pokes travis on getting a webhook.

The scope for webhook is issues, issue_comment from this list: https://developer.github.com/webhooks/#events

You can also help users create a webhook using this API: https://developer.github.com/v3/repos/hooks/

[manavo]

It would obviously be a big change to the project, but theoretically the template/page could get built on the front-end, to avoid needing a re-build each time? It seems you can read issues via the API without auth: https://api.github.com/repos/jayfk/statuspage-demo/issues

That way you would only need a rebuild if adding a new system or something?

(just thinking out loud, sorry if too off topic for the original issue)

[jayfk]

I thought about something like this initially, but GitHub has a pretty low rate limit for unauthenticated requests.

curl -v https://api.github.com/repos/jayfk/statuspage-demo/issues < X-RateLimit-Limit: 60 < X-RateLimit-Remaining: 57

[manavo]

Ah, fair enough! Not a big enough limit for this then!

[paracycle]

@jayfk @manavo However that rate limit is per requesting IP address:

Unauthenticated requests are associated with your IP address, and not the user making requests.

That might still not be enough (basically 1 request/minute on average per IP address) but it is not so drastic either. Besides, the front-end can cache the response in local storage for (let's say) 2 minutes to ensure that the rate limit does not pose a problem.

[manavo]

Good find @paracycle! In that case it is definitely doable then!

[jayfk]

Good find @paracycle! In that case it is definitely doable then!

I am not 100% convinced ;).

Let's assume you have a repo with 3 systems, 2 collaborators and 5 issues with 4 comments each (20 comments total).

You'll need:

• 6 calls to get all labels (3 for the system, 3 for severity)
• 2 calls to fetch the collaborators
• 5 calls to fetch all issues
• 20 calls to fetch all comments (+ some more if the users are not collaborators)

That's a total of 33 calls. With a limit of 60 calls per IP you'll hit the limit with your second page refresh.

We can do the following:

Ask users to create a new webhook to heroku(?). I remember I saw some service that does jekyll builds on webhook notifications This tiny service pokes travis on getting a webhook.

Yup, but I really want to have something super simple for this. Something you don't have to maintain or care about. Don't get me wrong, I even wrote a toolkit for this two weeks ago at https://github.com/pyupio/octohook, but it involves setting up a server that you need to maintain etc.

What if we could bundle this thing, create an AWS API Gateway, an AWS Lambda function and a webhook for the repo that calls the function.

I was thinking about a simple command that takes your AWS credentials and creates everything automatically for you.

statuspage automate --name=foo --token=yourtoken

AWS Access Key: ********************
AWS Secret Key: *********************

And you are done with it.

AWS Lambda has 1M free requests per month. I don't know about the API Gateway, but they charge per million requests. That's like $0.01 in 20 years.

[captn3m0]

Another alternative for running code: https://rundexter.com/

On Wed, Mar 9, 2016 at 8:26 PM Jannis Gebauer [email protected] wrote:

Good find @paracycle https://github.com/paracycle! In that case it is definitely doable then!

I am not 100% convinced ;).

Let's assume you have a repo with 3 systems, 2 collaborators and 5 issues with 2 comments each (20 comments total).

You'll need:

• 6 calls to get all labels (3 for the system, 3 for severity)
• 2 calls to fetch the collaborators
• 5 calls to fetch all issues
• 20 calls to fetch all comments (+ some more if the users are not collaborators)

That's a total of 33 calls. With a limit of 60 calls per IP you'll hit the limit with your second page refresh.

We can do the following:

Ask users to create a new webhook to heroku(?). I remember I saw some service that does jekyll builds on webhook notifications This tiny service pokes travis on getting a webhook.

Yup, but I really want to have something super simple for this. Something you don't have to maintain or care about. Don't get me wrong, I even wrote a toolkit for this two weeks ago at https://github.com/pyupio/octohook, but it involves setting up a server that you need to maintain etc.

What if we could bundle this thing, create an AWS API Gateway, an AWS Lambda function and a webhook for the repo that calls the function.

I was thinking about a simple command that takes your AWS credentials and creates everything automatically for you.

statuspage automate --name=foo --token=yourtoken

AWS Access Key: ******************** AWS Secret Key: *********************

And you are done with it.

AWS Lambda has 1M free requests per month. I don't know about the API Gateway, but they charge per million requests. That's like $0.01 in 20 years.

— Reply to this email directly or view it on GitHub https://github.com/pyupio/statuspage/issues/5#issuecomment-194332019.

[paracycle]

@jayfk I am not sure I understand the request counts. A single request for issues of a repo returns up to 100 issues (which can be filtered for open/closed ones). Each issue returned includes the set of labels that are associated with it as well. If all the system types are predefined in the HTML file, then there won't be a need to read a list of all the labels in the repo (if really necessary this can also be done with 1 extra request and probably cached for up to 30 mins, after all we don't really expect system/severity identifier to change often).

That leaves us with comments. But for N issues that is only N extra calls, since a request for issue comments returns all the comments. Moreover, since comments will hardly be edited, they can be cached almost indefinitely (and thus not requested again).

With clever caching and creative API calls, it should be possible to obey the API limits. What do you think?

[jayfk]

@paracycle All right, I should have looked at the actual json before writing this. I assumed we need extra calls for that.

Leaves us with:

https://api.github.com/repos/jayfk/statuspage-demo/collaborators

{
  "message": "Requires authentication",
  "documentation_url": "https://developer.github.com/v3"
}

We need that because otherwise everyone that knows about the repo can at least comment on an issue.

If we predefine them in the HTML like the system labels it would indeed be possible.

That leaves us with comments. But for N issues that is only N extra calls, since a request for issue comments returns all the comments. Moreover, since comments will hardly be edited, they can be cached almost indefinitely (and thus not requested again).

Another good thing is that the /issue request returns the amount of comments for that issue, so that's even fewer requests if the comment count hasn't changed or is zero.

[captn3m0]

Ok, so we have 2 suggestions:

1. trigger rebuilds
2. Just handle it on the frontend and let users query it unauthed.

Another issue with the second option is that while gh-pages branch is necessarily public, the issues API might not be accessible because the repo might be private. Moreover, status pages are usually kept on auto-refresh to give the user immediate feedback, something which won't be possible if we are to stay within rate-limits.

However, the build flow and setup for the first option is very complicated as of now to make sense either. Somewhat related, not sure if it'd help is the prose/gatekeeper module which lets you do the oauth dance easily: https://github.com/prose/gatekeeper

[jayfk]

I've just found out that conditional requests against the public api don't count against the rate limit https://developer.github.com/v3/#conditional-requests

This way it would be possible to set the whole page to autoreload every 10 seconds without exceeding the rate limit.

[jayfk]

See https://jayfk.github.io/statuspage-prototype/ for an auto reloading prototype using the GitHub API :)

[paracycle]

Great work @jayfk. Thanks for looking into this. Looks amazing and will be much much simpler to run. Do you have plans for when you might release a version of this?

[jayfk]

@paracycle I've created a pre release: https://github.com/pyupio/statuspage/releases

[billiegoose]

The upgrade option didn't work for me:

$ statuspage upgrade --name=statuspage --token=$GITHUB_TOKEN
Uploading new files:  67%|            | 2/3 [00:00<00:00,  7.83it/s]
Traceback (most recent call last):
  File "<string>", line 193, in <module>
  File "site-packages/click/core.py", line 716, in __call__
  File "site-packages/click/core.py", line 696, in main
  File "site-packages/click/core.py", line 1060, in invoke
  File "site-packages/click/core.py", line 889, in invoke
  File "site-packages/click/core.py", line 534, in invoke
  File "<string>", line 61, in upgrade
  File "<string>", line 102, in run_upgrade
  File "github/Repository.py", line 1270, in create_file
  File "github/Requester.py", line 171, in requestJsonAndCheck
  File "github/Requester.py", line 179, in __check
github.GithubException.GithubException: 409 {'documentation_url': 'https://developer.github.com/v3/repos/contents/', 'message': 'refs/heads/gh-pages is at 42c62c307810d7d0d65fdca2bb3743bdc5149fa8 but expected 4e83ee987051913fb2020da82e1b8b11d31e2aa9'}
statuspage returned -1

But making a fresh repo worked! :heart_eyes: This is the best thing since sliced :cake: :exclamation:

[jakirkham]

We are using Heroku with GitHub webhooks to get this kind of functionality. It actually works pretty well thus far. Though there are some events that still are not supported. See issue ( https://github.com/isaacs/github/issues/746 ).

[yashsway]

Is there a way to set this up in Buddy CI or Codeship?

I've already made the statuspage on my desktop and have been updating it. I'm moving to automated builds but I get this error when I execute 'statuspage update ...' in the Python docker container. (after pip installing statuspage) Is there a way to pass in my repo URL instead of just the name of my repo?

I think it doesn't know my git credentials? Do I have to make a new statuspage repo?

Generating..
Traceback (most recent call last):
  File "/usr/local/bin/statuspage", line 9, in <module>
    load_entry_point('statuspage==0.8.1', 'console_scripts', 'statuspage')()
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/statuspage/statuspage.py", line 73, in update
    run_update(name=name, token=token, org=org)
  File "/usr/local/lib/python3.5/site-packages/statuspage/statuspage.py", line 184, in run_update
    sha = repo.get_git_ref("heads/gh-pages").object.sha
  File "/usr/local/lib/python3.5/site-packages/github/Repository.py", line 1525, in get_git_ref
    self.url + prefix + ref
  File "/usr/local/lib/python3.5/site-packages/github/Requester.py", line 172, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers, input, cnx))
  File "/usr/local/lib/python3.5/site-packages/github/Requester.py", line 180, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.UnknownObjectException: 404 {'documentation_url': 'https://developer.github.com/v3', 'message': 'Not Found'}

[jakirkham]

Did you give it a valid GitHub token with the right permissions? It needs that to run.

[yashsway]

Yes I ran the 'statuspage update' with a GitHub token with all 'repo' perms ticked and 'admin:repo_hook' perms ticked.

[jakirkham]

Sorry I'm still unclear. Did you pass --token=$GITHUB_TOKEN where GITHUB_TOKEN contains your GitHub token?

Also you may need the --org and --name arguments passed with reasonable values as well.

[jakirkham]

I've already made the statuspage on my desktop and have been updating it.

Also a little confused by this statement. statuspage only runs remotely on GitHub. It doesn't actually modify any local repo. Could you please explain what this means?

[jakirkham]

Is there a way to pass in my repo URL instead of just the name of my repo?

Guessing you are looking for --org then.

[yashsway]

Yep! So in my CI Python Docker container, this was the build step which produced the error I referenced earlier.

pip install statuspage
statuspage update --name={I put my repo name here} --token={I put my token here}

[yashsway]

I've already made the statuspage on my desktop and have been updating it.

This is what I meant: I did 'statuspage create... ' (as documented in this project readme) on my local machine which created a repo in my account for the page. Every time I make an issue on the repo, I've been running 'statuspage update' on my local machine, which runs a python script to update my repo on GitHub, which is then reflected in the status page site.

Instead of me having to run statuspage update on my terminal everytime I make a new issue, I wanted to automate this using Buddy CI

[yashsway]

What would an example 'statuspage update' command with the --org token look like?

Something like this? statuspage update --name={I put my repo name here} --org={.git URL of github repo?} --token={I put my token here}

[yashsway]

Also a little confused by this statement. statuspage only runs remotely on GitHub. It doesn't actually modify any local repo.

I was just wondering what statuspage create actually does other than make the repo on the GH account. Someone's solution in the thread above was to create the statuspage over again and move issues over. Does it modify the pip package to store the refs for the created status page ?

[jayfk]

statuspage create creates the repository and basically runs statuspage update internally afterwards to populate the initial template.

@jakirkham could you contact me at [email protected]? I wasn't able to find a way to contact you directly.

[jakirkham]

Something like this? statuspage update --name={I put my repo name here} --org={.git URL of github repo?} --token={I put my token here}

Sort of. --org should be the username or org name where the repo lives. So if it is your personal repo it would MightyRevenge. For instance, this is what we do for our status page. Does that work for you?

[jakirkham]

@jakirkham could you contact me at [email protected]? I wasn't able to find a way to contact you directly.

Yep, emailed.

[yashsway]

Thanks for clearing that up @jayfk! Hmm, I wonder why it's erroring out then. And thanks @jakirkham, that clears it up.

I hope to contribute to this project once I get a better handle on Python :/

[jayfk]

Do you see this also locally when running statuspage update?

[yashsway]

Nope, it works fine locally.

So I'm trying something new now; I've updated the script in the CI to the following and I'm making the Docker container use the latest Python release:

statuspage upgrade --name=**** --token=*****
statuspage update --name=**** --token=*****

It upgrades successfully. Now during the update step, it returns a different error. I'm just following the traceback and it now gives me a different error:

Upgrading...
Traceback (most recent call last):
  File "/usr/local/bin/statuspage", line 9, in <module>
    load_entry_point('statuspage==0.8.1', 'console_scripts', 'statuspage')()
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/statuspage/statuspage.py", line 81, in upgrade
    run_upgrade(name=name, token=token, org=org)
  File "/usr/local/lib/python3.5/site-packages/statuspage/statuspage.py", line 150, in run_upgrade
    files = get_files(repo=repo)
  File "/usr/local/lib/python3.5/site-packages/statuspage/statuspage.py", line 353, in get_files
    return [file.path for file in repo.get_dir_contents("/", ref="gh-pages")]
  File "/usr/local/lib/python3.5/site-packages/github/Repository.py", line 1410, in get_dir_contents
    parameters=url_parameters
  File "/usr/local/lib/python3.5/site-packages/github/Requester.py", line 172, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers, input, cnx))
  File "/usr/local/lib/python3.5/site-packages/github/Requester.py", line 180, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 404 {'message': 'No commit found for the ref gh-pages', 'documentation_url': 'https://developer.github.com/v3/repos/contents/'}
Build failed !!!.

'No commit found'. I recently transferred ownership of the entire repo to a different account, maybe I have to make a dummy commit (or make a dummy issue) from the new owner of the account, to the repo?

Currently browsing https://developer.github.com/v3/repos/contents/ and https://github.com/PyGithub/PyGithub/blob/master/github/Repository.py to find an answer related to this specific error message...

[jayfk]

I recently transferred ownership of the entire repo to a different account, maybe I have to make a dummy commit (or make a dummy issue) from the new owner of the account, to the repo?

No. This means your gh-pages branch is empty. When running statuspage create for the first time, it should do that automatically for you.

[jayfk]

Oh, and you shouldn't run statuspage upgrade. This command is intended to be used when upgrading the statuspage binary, e.g if you do an upgrade from 0.8 to 0.9 etc.

[yashsway]

But I have 37 commits on my gh-pages branch though! 😕 and 1 in the master. And okay, I'll remove that, thank you! We should specify that in the docs as well, if that might break things?

If this is the format for a content request using the GitHub API: GET /repos/:owner/:repo/contents/:path

When I transfered the repo, I have a feeling the new owner (:owner) technically doesn't have any commits in the gh-pages branch. MightRevenge, my personal account does, the new owner doesn't.

I'm going to test out my theory, brb

[yashsway]

Yep that was it!!! 🎉 I had to make a dummy commit using the new owner 👏

[paracycle]

@jayfk what happened to the purely client-side version that you had a pre-release for back in the day? did you choose to not go down that road at the end?

[jayfk]

@paracycle yeah. The public GitHub API is too limited. I was constantly rate limited during development so I've decided to abandon that approach.