Colab integration

[lorenzoh]

Ideally, all the documentation pages based off of Jupyter notebooks should have a button to launch them in Colab.

For example, fastai's documentation has such a button:

First step would be a tutorial on how to set up Colab with Julia and install the dependencies.

See this Discourse post that shows how to get Julia running on Colab.

[rozeappletree]

Hi, I want to take this up.

[lorenzoh]

Cool! Here's what I think needs to be done.

• Conversion of regular Julia .ipynb notebooks to Colab notebooks This consists of

  • Adding cells with instructions and shell script that installs Julia, packages and the Julia kernel (see the Colab template for an example)
  • Updating the metadata of notebook with kernel name and accelerator info so that GPU is used by default

  This functionality may be best created as a separate repository (Colab.jl?) that does this conversion; that way the functionality will be easier to reuse but also maintain when the specifics of running Julia on Colab change

• Integration with the documentation system (Pollen.jl)

  During documentation build, the following needs to happen for every source notebook file

  • create a Colab version of the notebook using the above conversion functionality
  • generate a Colab link that will auto-open the Colab version that is built to the GitHub Pages branch during doc build
  • insert the link with a badge into the documentation html output

  Since Pollen.jl is still undocumented, I could take care of this step once the first is done

Let me know if you need any other pointers 👍

[Chandu-4444]

Hey, I'm interested in doing this. Can I try to do this?

[lorenzoh]

Sure! Let me know if you need any other pointers aside from the above

[Chandu-4444]

I could come up with this. But is there any way to speed up the initialization process (The time taken to run the first cell). Running this for every time a notebook that has been opened is kind of inefficient? Or is there a workaround?

[lorenzoh]

One though I've had is using PackageCompiler.jl to create a sysimage once that is then saved somewhere and loaded into the Colab session which would allow reducing installation time significantly and almost completely removing compilation overhead.

However, there are a lot of problems that need to be dealt with, and I'm not sure it's worth the effort:

• the sysimage needs to be built on Colab since it needs to be compiled for that hardware and software
• it needs to be manually rebuilt on every update since Colab doesn't allow automation
• the sysimage file needs to be stored somewhere and served (these are easily > 100MB)
• I'm not sure if the randomly different GPU hardware you get on Colab would mess with the sysimage.

[Chandu-4444]

Ok then. I'll add the Launch Colab button to all document pages that are based off a notebook.

Cool! Here's what I think needs to be done.

• Conversion of regular Julia .ipynb notebooks to Colab notebooks This consists of

  • Adding cells with instructions and shell script that installs Julia, packages and the Julia kernel (see the Colab template for an example)
  • Updating the metadata of notebook with kernel name and accelerator info so that GPU is used by default

  This functionality may be best created as a separate repository (Colab.jl?) that does this conversion; that way the functionality will be easier to reuse but also maintain when the specifics of running Julia on Colab change

• Integration with the documentation system (Pollen.jl) During documentation build, the following needs to happen for every source notebook file

  • create a Colab version of the notebook using the above conversion functionality
  • generate a Colab link that will auto-open the Colab version that is built to the GitHub Pages branch during doc build
  • insert the link with a badge into the documentation html output

  Since Pollen.jl is still undocumented, I could take care of this step once the first is done

Let me know if you need any other pointers +1

Sure! I started working on the first one. Is it possible to update the metadata of a notebook file and add the extra Julia installation to the start of the notebook as soon as the user clicks on Launch Colab? Or is it better to manually add this information to all notebook files and store them in a separate repo (Colab.jl) and we can link the corresponding notebook from Colab.jl as soon as a user clicks Launch Colab in FastAI documentation page.

[lorenzoh]

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

This code should probably live in a separate repo, but the notebook files themselves will be built during the documentation building step in FastAI.jl's CI and stored on the gh-pages branch.

[Chandu-4444]

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

I'll first try writing that conversion script. Thanks 👍

[Chandu-4444]

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

I'll first try writing that conversion script. Thanks +1

Hey, I made a repo that contains the code for converting a normal ipynb notebook to the one that can run on Colab. https://github.com/Chandu-4444/Colab.jl This is a sample one, there are few improvements to be made (Specifying the output file location etc).

Please look at the repo and let me know the next step.

[Chandu-4444]

This code should probably live in a separate repo, but the notebook files themselves will be built during the documentation building step in FastAI.jl's CI and stored on the gh-pages branch.

I have an idea to do this. Is it ok if the configuration cells were hidden from getting displayed on the docs website and reveal when people click view on colab on colab page?

[lorenzoh]

Sorry for letting this fall through the cracks!

Hey, I made a repo that contains the code for converting a normal ipynb notebook to the one that can run on Colab. https://github.com/Chandu-4444/Colab.jl This is a sample one, there are few improvements to be made (Specifying the output file location etc).

Thanks for starting on this

I have an idea to do this. Is it ok if the configuration cells were hidden from getting displayed on the docs website and reveal when people click view on colab on colab page?

They should be added to the notebook, and the documentation system will take care of hiding the setup cells as well as the "Open in Colab" button

[Chandu-4444]

They should be added to the notebook, and the documentation system will take care of hiding the setup cells as well as the "Open in Colab" button

So, shall I replace all the current notebooks with the notebooks containing the config cells that can run on Colab?

[Chandu-4444]

Colab.jl is now a registered Julia package. I think this can now be integrated into docs?

[lorenzoh]

Hey, sorry for letting this sit again, integrating it into the docs is bottlenecked on my time making an integration with Pollen.jl right now. In any case, thanks for getting this running! I'll open an issue on the repo to propose some changes when I get around to it