Wishlist: User Configurations

[jankeromnes]

Today, for every new contribution on the Janitor, there are a number of repetitive steps you need to perform in order to configure your environment properly. It would be great to automate these steps, e.g. by letting users add their desired configurations once and for all to their Janitor account, and automatically apply them to all your contributions.

Here are the configurations we'll aim to support:

• [x] ~~Cloud9 username (to link your Cloud9 and Janitor accounts)~~ (we no longer use the third-party c9.io service)
• [x] ~~Cloud9 SSH public key (to authorize Cloud9 IDE to access to your contributions)~~ (same as above)
• [ ] Personal SSH public keys (to authorize SSH access to your contributions from anywhere)
• [x] Dedicated SSH key pair (to allow pushing from your contribution environment e.g. to GitHub)
• [x] ~~Git/Mercurial preference (to prefer one over the other for contributions where both are supported)~~ (not really a user preference, but a different project setup)
• [x] .gitconfig (to preconfigure e.g. user.name, user.email, core.editor and credentials like bz.username and bz.apikey)
• [x] .gitignore (to make sure contribution internals are ignored, e.g. the .c9/ folder in your workspace)
• [x] .hgrc (to preconfigure ui.username, enable extensions like mq, specify the desired patch format)
• [x] .config/hub and .netrc to auto-configure hub like this
• [ ] .gdbinit (e.g. to ignore SIG38 and SIG39 by default, and add add-auto-load-safe-path /home/user/firefox/.gdbinit or set auto-load safe-path /)
• [ ] .eslintrc
• [ ] .vimrc
• [ ] .emacs
• [ ] .nanorc
• [ ] Bash/Zsh preference
• [ ] .bashrc (e.g. for aliases, completion behavior, and to set a custom PS1 prompt)
• [ ] .zshrc

If you feel that something important for your work is missing, please comment below and I'll update the list.

[jankeromnes]

@beaufortfrancois @Coder206 @mantaroh @notriddle @rcatolino Please take a look: would this list cover all your configuration needs for your Janitor workflows?

[Coder206]

@jankeromnes Based on my workflow and knowledge it looks like it would work well for Servo!

[rcatolino]

A custom bashrc would be nice (for aliases, completion behavior etc.), and you could set your prompt there.

[beaufortfrancois]

👍

[notriddle]

This would cover all the needs, yeah, but could we add the hub CLI app? The hub command allows you to manage pull requests and stuff directly from the CLI.

If you did set up hub, since it requires you to get a personal access token anyway, you might as well use that for git, too, and skip the SSH stuff for GitHub users.

[jankeromnes]

Very interesting, I didn't know about hub, thanks for the suggestion! I will look into adding it to all project images by default.

[jankeromnes]

Added hub 2.2.8 to the base project image (janx/ubuntu-dev) in https://github.com/jankeromnes/dockerfiles/compare/80ee361499c1...c73991778198, so it should be available in all project images after the next full rebuild (probably in 2-3 weeks).

[davidaparicio]

@jankeromnes @notriddle the hub CLI makes me remember a very old project of hexanome: https://github.com/hexanome/github/blob/master/github 👍

[jankeromnes]

@davidaparicio Yes, it reminded me too! Great project this :D

[jankeromnes]

Dedicated SSH key pairs are now automatically generated for active users since 3ef28f9cb4b30b0a6b8bab8d4f19e5dfa20d77dd.

[etiennewan]

Maybe a .eslintrc file, as c9 IDE uses it for JS linting. Sometimes, I need to specify explicitely the environment (node, browser, workers) because eslint triggers warnings on globals if the environment is not set. Edit: wrong thread, as a .eslintrc file is not a per-user configuration

[notriddle]

I just created a new Servo contribution, and there's no key in ~/.ssh/. How do I use it with git?

[notriddle]

It also doesn't seem to have hub on it. Do you still need to do another deploy?

[jankeromnes]

@EtienneWan Great point, figuring out IDE linting would be very nice, but I think some projects tend to redefine many rules (e.g. Firefox). It'd be great to get the IDE to lint your code according to just the right rules (i.e. merging your own preferences with Firefox code style). I'll add .eslintrc to the list here, but most of the work will probably happen in the "Code feedback in the editor" plugin for #30.

[jankeromnes]

@notriddle Dedicated SSH key pairs are being generated for active Janitor users (so you already have one), but they're not yet copied into all project containers (at some point they will be added to new containers, and your user settings will have a feature like "Propagate my SSH keys to all containers" with a warning about overwriting previous configurations).

Also, I haven't been able to deploy the new project images with hub included yet, because of disk space problems (host 1 is 97% full, and I'm rushing to finish Janitor's transformation into a multi-server cluster so that we can use host 2, which is a new AWS instance with a lot of free space). Host 2 already has all the new images, and the clustering mechanism is almost finished, it just misses single sign-on and a new noVNC proxy. I hope to be finished this week, and I'll talk about in the upcoming newsletter... almost there!

[notriddle]

Ah, okay.

[jankeromnes]

@notriddle As a workaround, please feel free to install and configure hub in your container (you have root privileges), or to generate a new SSH keypair as explained here and here, so that you can push your Servo contribution. Sorry that this is still such a hassle, but I'm working around the clock to automate all of this as fast as possible.

[notriddle]

Now that hub is working, there are three things I run to set up a Janitor container.

• Run hub config --global hub.protocol https, because I don't want to mess with setting up SSH when I can just use HTTPS.
• Run hub fork, which:
  • Prompts me for my GitHub username, password, and 2FA token. It uses these to request a Personal Access Token from GitHub, and stores that in ~/.config/hub. If Janitor were a GitHub OAuth2 app, it could just drop its token in there.
  • Adds a Git remote pointing at my fork of the Servo repository.
• Copy the oauth token from the ~/.config/hub file into ~/servo/.git/config. This changes the URL from https://github.com/notriddle/servo to https://notriddle:[email protected]/notriddle/servo. Janitor could generate this file just as easily as it generates ~/.config/hub. I wish I understood why hub didn't just do this.

The nice part about this (other than getting around firewalls by running everything on port 443) is that I can revoke a Janitor container's access to my repos by removing one record from inside the GitHub web app.

[jankeromnes]

On Tuesday, February 28th, 2017, we also had an interesting IRC discussion about hub. Pasting it here because it contains useful ideas:

17:01:16 (notriddle) There are missing features. The CLI uses vim instead of just using Cloud9's text editor. hub browse uses lynx instead of opening a new tab in my local Firefox instance. And I need to manually log in to GitHub and set my git committer credentials by hand for every new container. And you forgot to alias git=hub in my bashrc. 17:02:18 (notriddle) It's nitpicky stuff at this point, though. 17:24:55 (janx) notriddle: interesting suggestions! You can already npm install -g c9 and then set EDITOR env variable to "c9", and I tried that for a while, but it got somewhat confusing during git rebase -i and git add -p and such (or maybe I'm just used to vim for these) 17:26:09 (janx) notriddle: but nitpicky stuff is great! [...] 17:27:19 (janx) hub browse sounds tricky, a Cloud9 plugin is probably required to support this (just like Coder206's IDE notifications)

(Note: It's no longer necessary to do npm install -g c9, because c9 is already installed on recent Janitor project images, so in Cloud9's Terminal you can use c9 path/to/file.js to open a file directly in Cloud9's Editor instead of vim).

[jankeromnes]

Note: Some users might want to have a small bootstrap script for their containers, e.g. to check out a personal "dotfiles" repository. Maybe this can be done by appending some instructions to the ~/.bashrc file.

Edit: Thanks @La0 for the suggestion!

[jankeromnes]

Great news: @nt1m just started implementing a .gitconfig that would be editable from a new "Configurations" section in the "Settings" page.

Welcome Tim! And thank you for jumping on such a long-awaited feature! 🎉

Edit: The editable .gitconfig was merged as f1cc7db871b1819e1d468f1632b5f2c147b1896b.

[jankeromnes]

I also contributed an editable .gitignore in 349807a6f22f0a1d072637a3467dbcaf377bc2cf, and both features are now live on https://janitor.technology ! 🎉🎆🍾

If you care about this, please give them a try in your https://janitor.technology/settings/configurations/ (@Coder206 has already found a browser cache issue, so be sure to refresh a few times so that you get the latest /js/janitor.js file from the server.)

[jankeromnes]

@nt1m mentioned he'd be interested in adding a .hgrc template, which could be very useful for both Firefox and Thunderbird.

FYI, here is a .hgrc template draft that I worked on a few months ago. Hope it can be useful to you! (But please make sure it is up-to-date by comparing it with the ~/.hgrc produced by a fresh ./mach bootstrap).

Also worth noting, a Firefox-oriented .hgrc should in theory work for Thunderbird devs as well without requiring any additional changes.

[jankeromnes]

Marked this as a Good First Bug because since the system is in place, it's both pretty quick and very useful to add a new user configuration.

To do that, you can simply add a useful default template in templates/configurations/ (you can look at this commit for inspiration).

[jankeromnes]

FYI, I asked about ~/.config/hub, ~/.netrc and Personal Access Tokens for hub in https://github.com/github/hub/issues/1668.

[jankeromnes]

We've landed configuration templates for ~/.config/hub and ~/.netrc in be78b4dbd9696c5ac2f1e7c818314c22e5f84738, and hub does pick up the new token, but for some reason hub fork works but not hub push, even though we have request the public_repo scope:

user@e7377ffb4b38:~/services (master) $ hub fork
Updating jankeromnes
remote: Counting objects: 281, done.
remote: Total 281 (delta 159), reused 160 (delta 159), pack-reused 121
Receiving objects: 100% (281/281), 71.71 KiB | 2.66 MiB/s, done.
Resolving deltas: 100% (210/210), completed with 19 local objects.
From https://github.com/mozilla-releng/services
 * [new branch]        dev_auth              -> jankeromnes/dev_auth
 * [new branch]        mapper_tests          -> jankeromnes/mapper_tests
 * [new branch]        master                -> jankeromnes/master
 * [new branch]        notification_services -> jankeromnes/notification_services
 * [new branch]        production            -> jankeromnes/production
 * [new branch]        releng_slavehealth    -> jankeromnes/releng_slavehealth
 * [new branch]        staging               -> jankeromnes/staging
 * [new branch]        testing               -> jankeromnes/testing
 * [new branch]        update_documentation  -> jankeromnes/update_documentation
 * [new tag]           v35                   -> v35
new remote: jankeromnes

user@e7377ffb4b38:~/services (master) $ hub push jankeromnes 'origin/master:master'
remote: Permission to jankeromnes/services.git denied to jankeromnes.
fatal: unable to access 'https://github.com/jankeromnes/services.git/': The requested URL returned error: 403

Note, the public_repo scope is requested here, and I do see it in GitHub's authorization URL parameters, but the GitHub confirmation prompt says that the Janitor OAuth app only needs access to "public information" (and not "read/write access to public repositories").

Could this be a GitHub bug? Or maybe I don't understand what the public_repo scope does?

[ishitatsuyuki]

It may be better to use write:public_key and SSH. That seems to be how Che is doing it.

[jankeromnes]

It may be better to use write:public_key and SSH. That seems to be how Che is doing it.

Thanks for the suggestion! We do generate custom SSH keys for each Janitor user today (they're not used for anything yet, but we could potentially deploy them in user containers and authorize them with GitHub). However, we'd like to limit the scope of developers' credentials in every container, because Janitor is a shared infrastructure and can be hacked. A GitHub OAuth token with scope public_repo is somewhat less dangerous if stolen than an SSH key that's authorized with potentially multiple remote services and hosts (compare "write access to non-protected branches of public GitHub repositories" with "full access to any authorized remote service, e.g. all repositories on GitHub/GitLab/BitBucket/... and any servers that you SSH into").

[jankeromnes]

Maybe we should also support file modes for configuration files, to fix warnings like these:

WARNING: netrc file /home/user/.netrc cannot be used because its file permissions are insecure.  netrc file permissions should be 600.