Blockers for using retrolab in data8

[yuvipanda]

JupyterLab retro is a great alternative to the classic notebook, keeping the simple document oriented UI while giving us new features like RTC. We want to get it to a point where it can be used for data8. This issue will track whatever needs to be fixed for that to happen.

• [x] https://github.com/jupyterlab/retrolab/issues/153
• [x] https://github.com/jupyterlab/retrolab/issues/152
• [x] Add retrolab as an option in nbgitpuller.link - depends on https://github.com/jupyterlab/retrolab/issues/153
• [x] https://github.com/jupyterlab/retrolab/issues/157
• [x] Get everything into a 2.x release
• [x] Install release with all these fixes on our hubs
• [ ] Figure out how JS for saving notebooks will work (otter does this)
• [ ] Wite a blog post about Data 8 using Retro Lab!
• [x] Create a documentation for launching retro lab (articulate the why for this transition)

/cc @Yanay1 and @ericvd-ucb

[yuvipanda]

https://github.com/jupyterlab/retrolab/pull/159 and https://github.com/jupyterlab/retrolab/issues/157 should fix two of these issues. Once merged, I'd want to backport them to the 2.x release so we can deploy them.

[yuvipanda]

@Yanay1 @ericvd-ucb all the blockers we have identified have been fixed. Can we try giving it a shot again?

[ericvd-ucb]

@Yanay1 has left Berkeley to go across the bay I think this is a question for Data 8 course staff - eg Wagner or DeNero @yuvipanda is the point that this is future-proofing strategy? and if so how urgent is it?

[yuvipanda]

@ericvd-ucb I think I'd like us to use this for the next semester. Perhaps we can have a meeting where we go through some of the workflows but with retrolab to see what else needs fixing? I think we've fixed all the reported issues so far.

[balajialg]

@yuvipanda @ericvd-ucb This is awesome! Any specific reasons to start with Data 8? Was this a prior request from Data 8 staff? I wonder whether it would be a viable strategy to begin testing with a smaller class like CMP BIO (Genomics class) and test how the retro lab functionality works before moving it to Data 8? Thoughts?

[yuvipanda]

@balajialg we could do that as well - but I suspect some of those classes might just want to use JupyterLab straight up instead? I think data8 is the big convert here, as it is teaching retrolab's target audience. We could definitely start smaller, but ideally we won't wait one more full semester before starting data8 out. @fperez has been teaching data100 to 1200 students with JupyterLab, and that already shook out a lot of issues with otter and other systems that data8 also uses (JupyterLab and retrolab share most code)

[ericvd-ucb]

So I guess some of the elements would be

• could we "test it out somewhere"
• we could test it out on Data 88 - 75 students ?
• some percent of Data 8 students?
• make up a set of nbgitpuller links
• probably need to convince DeNero of the need to do this
• Most importantly need to talk to Data 8 course staff who does assignment generation

[fperez]

@yuvipanda thanks for the ping! I'm 💯 on this, and happy to try to move things forward. I had a quick look and just opened a couple of issues in the retro repo for things that stood out to me. I think jupyterlab/retrolab#259 should be added to your blockers list at the top, not so sure about the file manager ones: those might be easier to live with for now, as I don't know how critical the FM is to Data 8 workflows.

[yuvipanda]

That's a good list, @ericvd-ucb. I thought we had already convinced David Wagner of this last semester. I think coupling this with real time collab is perhaps an important aspect to highlight.

If you think that starting with a smaller class is the way to go, let's definitely do that.

@fperez the data8 workflow doesn't use the file manager at all, so shouldn't affect it.

[davidwagner]

For Fall 2021, Melissa Wong and Stephanie Djajadi are the leads of the Data 8 staff team who generate assignments.

[ericvd-ucb]

OK well lets get student staff to try it once more with Data 8 materials @yuvipanda can you direct us to the retrolab nbgitpuller syntax

[Yanay1]

I made the change to retro lab at the beginning of summer (before reverting back) so I have the paths which I guess I forgot to commit:

demos_base = 'http://data8.datahub.berkeley.edu/hub/user-redirect/git-sync?repo=https://github.com/data-8/materials-su21&urlpath=retro/notebooks/materials-su21'
demos_file = '&subPath=lec/%s'

datahub_base = 'http://data8.datahub.berkeley.edu/hub/user-redirect/git-sync?repo=https://github.com/data-8/materials-su21&urlpath=retro/notebooks/materials-su21'
datahub_path = '<a href="' + datahub_base + '/materials/su21/'

These would replace the existing variables in site/content/pages/calendar.ipynb (have to change su21 to new semester)

[yuvipanda]

Thanks a lot, @Yanay1! Does this help, @ericvd-ucb?

[yuvipanda]

I also opened https://github.com/jupyterhub/nbgitpuller/pull/225 to add retrolab support to the link generator.

[ericvd-ucb]

Hi all. Yep I think I can figure out the syntax from @Yanay1 code. I am guessing this is actually referring to Data8 assignment generation scripts

[yuvipanda]

@ericvd-ucb yeah. for a simple string replace, you can replace urlpath=tree/ with urlpath=retro/tree/ in all current nbgitpuller URLs and they will open with retrolab instead.

[yuvipanda]

@ericvd-ucb nbgitpuller.link now supports opening in retrolab

[yuvipanda]

Looks like otter relies on classic notebook javascript execution with display (https://github.com/ucbds-infra/otter-grader/blob/6a33a7519c8015201989250a1d185d4b03638892/otter/check/utils.py#L18) to help save notebooks before submitting them. This is extremely important functionality, but has already been a bit flaky (been debugging issues with problems saving with @davidwagner). It also won't work with retrolab as it is.

We must find a way to allow python code to ask for the current notebook to be saved. This is a blocker for retrolab / lab adoption I think. I think a more robust solution will also fix the current issues around failure to save.

@jtpio any thoughts on how we can address this?

[davidwagner]

Can you help me figure out how I'd test a notebook with retrolab? You mention doing a string replace on urlPath, but our current nbgitpuller links that we're currently using in Data 8 don't use urlPath. A sample one is http://data8.datahub.berkeley.edu/hub/user-redirect/git-sync?repo=https://github.com/data-8/materials-fa21&subPath=materials/fa21/hw/hw10/hw10.ipynb. Is there a simple tweak to that, which will take me to a retrolab version of the notebook? If there is, we could probably start testing some Data 8 assignments with retrolab.

I tried checking out the nbgitpuller page that yuvipanda linked to, but that doesn't have an option I can see for opening in retrolab (only JupyterLab).

[davidwagner]

Hmm, it looks like what works is to create a nbgitpuller link with https://jupyterhub.github.io/nbgitpuller/link?hub=https://data8.datahub.berkeley.edu&repo=https://github.com/data-8/materials-fa21&branch=master, fill in something like materials/fa21/hw/hw10/hw10.ipynb as the file to open, select JupyterLab as the application to open, then manually edit the JupyterHub URL to change lab to retro. That's a bit kludgy but it seems to work.

@yuvipanda, Was there supposed to be built-in support to create retrolab links on that page (e.g., another option under Application to Open for Retrolab)?

[yuvipanda]

@davidwagner yeah, i'm not sure why it isn't showing up in that page. I'll investigate.

[yuvipanda]

also, I definitely don't expect that display call to work in retrolab!

[ericvd-ucb]

So for testing Nbgitpuller Links for Fall 2021 Data 8 Lab9 in Classic Notebook https://data8.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fdata-8%2Fmaterials-fa21&branch=master&urlpath=tree%2Fmaterials-fa21%2Fmaterials%2Ffa21%2Flab%2Flab09%2Flab09.ipynb

Data 8 Lab9 in RetroLab Notebook https://data8.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fdata-8%2Fmaterials-fa21&branch=master&urlpath=retro%2Ftree%2Fmaterials-fa21%2Fmaterials%2Ffa21%2Flab%2Flab09%2Flab09.ipynb

[ericvd-ucb]

@yuvipanda as per your Autosaving blocker commentary above, It seems to me that Data 100 has been running Otter in JupterLab for 3 semesters now and somehow Otter and Lab are getting along there?

[yuvipanda]

@ericvd-ucb yeah, that's a great point. I think maybe they ask students to manually save before submitting? @chrispyles @fperez do either of you know if you manually instruct users to save?

[chrispyles]

Yes. Instructions to save are included in every notebook that Otter Assign outputs that include export cells. In Data 100, we also highlight the saving requirement in assignment instructions and make it explicit that regrades due to unsaved work that wasn't included in the submission will not be accepted.

[yuvipanda]

Ah, ok! @chrispyles would you consider this kinda functionality allowing you to save the notebook from python a blocker for retrolab adoption?

[chrispyles]

I think that's a question for the instructors/course staff to answer. Personally, I'm not a huge fan of Otter's autosave feature as it's finicky and unreliable. However, I would imagine that courses like Data 8 that use the autosave feature would consider this functionality a requirement.

[davidwagner]

@yuvipanda Another blocker item for your list at the top: https://github.com/berkeley-dsep-infra/datahub/issues/2989

[davidwagner]

I reviewed Data 8's lecture demos for lectures 01-10, and I recorded the annoyances that I spotted: #2990, #2991, #2992, #2993, #2994.

John DeNero (@papajohn) is teaching Data 8 next semester, so he'll be the one to decide which ones are acceptable and which are stow stoppers, but I wanted to document what I've found so far.