Long term planning for the Frontend

[yuvipanda]

Currently, we have one JS file + 1 HTML file (rendered with Jinja2) that does all the frontend work. This includes:

1. Client for the BinderHub API
2. Main page (when people see mybinder.org)
3. Loading page (when people go to a mybinder.org/v2/gh/user/repo launch URL)

(2) and (3) are particularly intertwined - they're restyled versions of the same page (HTML + JS).

In the future, I'd like us to improve on the following aspects:

1. JS Tests (we have none)
2. Export a BinderHub client API library that others can re-implement. Right now, everyone does their own implementation, and particularly run into problems against eventsource retries.
3. Implement both v2 (current) and newer API (whenever #358 lands)
4. Split the main & loading pages
5. Make main page a lot more dynamic - ideally showcasing useful / featured repos & other forms of discovery
6. Better (privacy conscious) frontend instrumentation to see how people use the main page & how we can improve it.

Doing a refactor of the current JS in some form or way while it's still smallish (~400 lines) seems like a good idea to me!

Factors

Some factors to consider:

1. How much is it used inside the Jupyter community? We want to make it as easy as possible for other Jupyter contributors to work on this.
2. Does it help bring new people with different skillsets into the project? We should balance this against (1) - we wanna diversify who comes in and contributes to the project, and specific tech might help / hinder that.
3. How much is it used in the frontend dev world? We wanna use an approach that's likely to be well debugged & with a lot of resources out there on how to use it.
4. Is it appropriately scoped for our size? We shouldn't overcomplicate our lives by choosing tools meant for 100k lines projects while ours is much smaller.
5. Does it make working on the frontend a happy & joyful thing for many people in the long run? Often projects end up with convoluted untested frontend code that is hard to touch since it might break.

Options

Here are some options

Option 1: React + TypeScript

We can switch to using TypeScript as a language & use React for frontend components + tests.

Option 2: React + JS

Similar to above, but with just plain JS (JSX) instead of TypeScript.

Option 3: Next.js

Recommended by @betatim: https://nextjs.org/docs

Option 4: No React

We'll refactor and move away from monolithic single JS file, but keep using our current setup of pure DOM manipulation rather than use a library.

There are probably more options than this, but this should be a starting point to discuss!

[yuvipanda]

Fun fact - when I first started writing BinderHub, the very first line of JS I wrote was:

/* If this file gets over 200 lines of code long (not counting docs / comments), start using a framework

It's still there, and the file is over 400lines long now :)

[yuvipanda]

I personally really like TypeScript, and the JupyterLab / nteract communities seem to be using it heavily as well. However, @ellisonbg pointed out it might be overkill for us - and I agree.

[yuvipanda]

My current preference is to start with Option 4, and then see how it goes.

[yuvipanda]

/cc some people who might be interested: @minrk, @betatim, @ellisonbg, @ian-r-rose, @choldgraf, @willingc

[sgibson91]

Hi @yuvipanda

I've often thought it would be nice to have some kind of checkbox on the main page for if you wanted to launch in a JupyterLab/RStudio environment. Rather than selecting URL path and type lab/rstudio as that's not overly clear to beginners. Just a thought 🙂

[choldgraf]

IMO option 4 will also make it easier to adopt options 1-3 in the future if we want to move in that direction, I'm +1 on option 4. It feels more like an iterative step rather than a total re-work.

put another way:

I think that we should do number 4 regardless. The question to me is "should we spend the extra time taking care of 1, 2, or 3 while we do number 4, or should we put off that work to a potential future time"

[betatim]

I'd vote for option 3. The main factor is that it used to be/is possible to use https://github.com/segmentio/create-next-app to get all the webpack/uglify/hot-reload/css shrinking etc stuff "for free". As someone who knows nothing about JS and web dev I enjoyed being taken by the hand and given something that looked useful, worked and wasn't super heavy weight. Since I setup bndr.it it seems things have changed so maybe next.js now has this built in??

https://nextjs.org/learn/ was very useful as a place to look up and (somewhat blindly) follow to get some good practices.

I'd split the "more dynamic front page" out from this refactor. Mostly because I think we can get some very easy wins there even if we don't make it dynamic. Having some popular repos shown, "syntactic sugar" like @sgibson91 mentioned, etc could be done already without any JS refactoring. Splitting this out means we could work on it in parallel and maybe pull in a new contributor?

[choldgraf]

If option 3 is "add no new features, just replicate the current structure of things using a simple framework anybody can use" then I'd be +1 on that. @betatim do you think that a person who has medium experience with HTML but no JS would be able to pick up next.js about as quickly as they'd pick up the current page setup?

[betatim]

For some more shameless lobbying: https://github.com/zeit/next.js/tree/master/examples is a good place to find out how to do stuff. So we can start lightweight but find inspiration and guidance. The recommendation right now is to start with an empty repo, follow nextjs.org/learn and pull stuff from the examples as needed.

I think on "do frontend devs use this" next is Ok and not a toolkit that is going away soon.

There are also react components we might be able to reuse from the nteract project (@rgbkrk tells me we should check out rx-binder + rx-jupyter)

[betatim]

I found it easy to build bndr.it with next.js, I find it very hard to understand our JS and DOM stuff on mybinder.org. However I never really got into using jQuery and went straight from writing HTML+CSS 15 years ago to react via a big break of doing "nothing web whatsoever".

I'd support the idea (no matter which option) to do the refactor without changing any functionality/looks/design. Then after the switch change design/functions/etc. It splits it into two smaller tasks. Smaller diffs to review, less potential for the refactor to get held up by discussion about changing looks/feel/functionality.

[choldgraf]

Sounds good - I am just trying to tease apart the level of difficulty that these tools will introduce for people of all technical backgrounds, not just those with web-dev experience. Y'all know more about this than I do, so I trust the community's input on this!

[yuvipanda]

I'm unsure what nextjs gives us over using react, but I guess these are implementation details...

I think we should definitely split this into two issues maybe - one for refactoring the current JS, and another for seeing if we should move to something.

[rgbkrk]

nextjs is still react, it mostly just gives you an opinionated scaffolding around the webpack so that you're (hopefully) not having to muck with all the JS config pain. Note: it can still be used with typescript if you want that.

[captainsafia]

Based on my experience, I think the best flow is Option 4 -> Option 2 -> Option 3.

The refactor in Option 4 should split up the contents of the single JS file into different files by UI element. For example, a file for repository input field, the file path input field, etc.

Once this refactor is complete, we can go through file by file and convert the set of jQuery events/actions into a React component. We don't have to opt-in to React completely. With the way that React is setup, we can bit by bit render more and more React components on the page as everything is migrated.

Once that is done, we can move over to Option 3 to leverage some of the performance improvements that Next.JS gives.

IMO, this process allows us to refactor and migrate things bit-by-bit and opt in to the tools as we need them. I'm happy to lead this effort, starting with a lightweight refactor so the site can continue to be deployed while we iterate towards our ultimate end goal.

[rgbkrk]

That sounds like a great way to make progress on this @captainsafia and allow others to help in the process iteratively. Feel free to ping me for review on PRs.

[choldgraf]

+1 to @captainsafia 's idea, thanks for the input!

[yuvipanda]

<3 @captainsafia would love for you to lead the effort!!!

[minrk]

That sounds like a great plan! Thanks to the js experts.

[jtpio]

2. Export a BinderHub client API library that others can re-implement. Right now, everyone does their own implementation, and particularly run into problems against eventsource retries.

A standalone @binderhub/client npm package would be really useful, that other (alternative) frontends to BinderHub could reuse. This client would abstract away the EventSource and other details, and could listen to events:

import BinderImage from '@binderhub/client';

var image = new BinderImage();
image.onStateChange('building', () => { ... });

Similar to: https://github.com/jupyterhub/binderhub/blob/23eeb4313d96890e3cc777e614135fa224da868c/binderhub/static/js/index.js#L168-L225

[choldgraf]

@jtpio I think there's general agreement that splitting up the mega JS file is a great idea (@captainsafia made some progress on this in https://github.com/jupyterhub/binderhub/issues/777). That PR has stalled, but perhaps we can look to it for inspiration? Or make some more progress on those items if @captainsafia is still interested? I personally am still very pro-this-idea, we all just have too many other things going on :-)

I think the example you give would also be useful for tools like Thebelab to utilize

[rgbkrk]

It looks like #777 was an issue tracking steps to take. Three were done in bite sized chunks, with several available for takers.

[choldgraf]

@rgbkrk ah you're right, for some reason my brain processed that as a PR 😊 (which makes way more sense... I was wondering why there was a PR that was referencing other bite sized PRs lol).

more bite-sized PRs like the ones on that issue to chip away at that would be great!

[yuvipanda]

I'm picking this back up :)

I think my first goal is to actually cleanup the @jupyterhub/binder-client enough to get it to 100% unit test coverage. I've a WIP PR for it, and it does lead to getting us out of some of the weird and buggy callback stuff we have going on.

[willingc]

@yuvipanda You may want to try Playwright for testing. Nice ergonomics and fast.

[yuvipanda]

Thanks for the pointer, @willingc! It does look pretty cool! I will pick it up when we get to doing end to end testing. Do you know if it can be used purely for unit testing? From looking at the docs, it looks like jest is still a better fit for unit testing while this is great for end to end.

[minrk]

I learned about playwright from @bl-aire and we now use it for the browser tests in jupyterhub. It's quite nice. I've since used it to script some tedious tasks I have to do for teaching with a web service that doesn't have an API.

I don't think it's really for unittesting, though. But since there's so little that our js does other than talking to BinderHub, maybe full browser tests are all we need?

[willingc]

More useful for component testing. Jest is fine for unit tests. IIRC playwright can execute jest tests too.