Anubis icon indicating copy to clipboard operation
Anubis copied to clipboard

Published 20 hours ago • verified publisher icon AnubisLMS

ADD Machine Learning IDE(WIP)

Open AlpacaMax opened this issue 3 years ago • 11 comments

Creating an IDE for Machine Learning class at NYU Tandon. Everything else except Dockerfile is updated. Haven't tested it due to difficulties starting a Theia IDE locally. I'll update Dockerfile and see if there're additional plugins or dependencies that need to be added.

AlpacaMax avatar Feb 14 '22 02:02 AlpacaMax

Which class will use this ?

On Sun, Feb 13, 2022 at 9:41 PM Maxwell Yang @.***> wrote:

Creating an IDE for Machine Learning class at NYU Tandon. Everything else except Dockerfile is updated. Haven't tested it due to difficulties starting a Theia IDE locally. I'll update Dockerfile and see if there're additional plugins or dependencies that need to be added.

You can view, comment on, or merge this pull request online at:

https://github.com/AnubisLMS/Anubis/pull/341 Commit Summary

File Changes

(5 files https://github.com/AnubisLMS/Anubis/pull/341/files)

Patch Links:

  • https://github.com/AnubisLMS/Anubis/pull/341.patch
  • https://github.com/AnubisLMS/Anubis/pull/341.diff

— Reply to this email directly, view it on GitHub https://github.com/AnubisLMS/Anubis/pull/341, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMEVVFWTMBNGV3Y2GH4D5LU3BTWJANCNFSM5OKEULAA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

GusSand avatar Feb 14 '22 03:02 GusSand

We were thinking pretty much anything that uses jupyter notebooks. I think the plan is for it to have jupyter notebooks, with:

  • matplotlib
  • pandas
  • numpy
  • scikit-learn

Nothing too crazy there. It could be used by the intro to machine learning class (which i believe max is in)

wabscale avatar Feb 14 '22 03:02 wabscale

Yep. And if it's for any class that uses jupyter notebooks, I guess I'll add PyTorch, Keras, TensorFlow, and stuff like that as well. My roommate is in DS major from CAS. I'll ask him what library he's been using in his classes.

There's one problem though: assignments from these classes are usually quite resource-consuming. Are we sure that digitalocean can provide that amount of computation resource to these classes?

AlpacaMax avatar Feb 14 '22 04:02 AlpacaMax

We should test how large the image is before and after adding those extra libraries. If it blows up in size, then we may not want to include them.

Don't worry about the resources. We can reserve whatever we need for specific sessions. If the default resources for a playground session aren't enough, I can add something so we can change the resources for specific IDE images

wabscale avatar Feb 14 '22 14:02 wabscale

Question:

what does this buy us over say Collab/Paperspace and others that already provide this service with GPUs or TPUs included even on the free tier?

On Sun, Feb 13, 2022 at 11:07 PM Maxwell Yang @.***> wrote:

Yep. And if it's for any class that uses jupyter notebooks, I guess I'll add PyTorch, Keras, TensorFlow, and stuff like that as well. My roommate is in DS major from CAS. I'll ask him what library he's been using in his classes.

There's one problem though: assignments from these classes are usually quite resource-consuming. Are we sure that digitalocean can provide that amount of computation resource to these classes?

— Reply to this email directly, view it on GitHub https://github.com/AnubisLMS/Anubis/pull/341#issuecomment-1038613633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMEVVCE5F5AEQCWZXGEVUDU3B5W5ANCNFSM5OKEULAA . You are receiving this because you commented.Message ID: @.***>

GusSand avatar Feb 14 '22 15:02 GusSand

That is a good point for the higher level ML stuff that uses GPUs and TPUs. The main value add would be the fact that this will be a vscode + jupyter together. It may not be much of a distinction, but whatever. If people want to use collab, thats fine.

Also one of the things Max is working out is how to use a theia plugin that is basically a mini-browser window that is right in the vscode IDE. We definitely will want to take that and put it in the flask / devops / golang IDEs.

wabscale avatar Feb 14 '22 15:02 wabscale

@wabscale Are there any changes that need to be made besides adding this theia-ml directory? Also I apparently cannot test this cuz I don't have access to the theia-base image on digitalocean.

AlpacaMax avatar Feb 15 '22 03:02 AlpacaMax

Oh the base image is just the theia-base directory. Just do docker-compose build theia-base-39 then build the theia-ml image. The base image takes a while to build so be ready to sit around for a while lol.

wabscale avatar Feb 15 '22 04:02 wabscale

There is some issue with the jupyter plugin:

running docker build -f ./Dockerfile -t theia-ml .
Sending build context to Docker daemon  12.29kB
Step 1/8 : ARG PY_VERSION=3.9
Step 2/8 : FROM registry.digitalocean.com/anubis/theia-base:python-${PY_VERSION} as theia
 ---> 498edd2c888c
Step 3/8 : USER root
 ---> Running in c5e422b095d0
Removing intermediate container c5e422b095d0
 ---> 1274518df658
Step 4/8 : COPY latest.package.json package.json
 ---> 84b1895229c3
Step 5/8 : RUN set -ex;   yarn --pure-lockfile;   yarn theia download:plugins;   yarn --production;   yarn autoclean --force;   yarn cache clean;   find / -depth     \( -name .cache -o -name __pycache__ -o -name '*.pyc' -o -name .git -o -name .github \)     -exec 'rm' '-rf' '{}' '+';
 ---> Running in db9a175fdd64
+ yarn --pure-lockfile
yarn install v1.22.5
info No lockfile found.
[1/4] Resolving packages...
warning @theia/callhierarchy > @theia/monaco > @theia/filesystem > trash > [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
warning @theia/callhierarchy > @theia/core > @theia/application-package > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning @theia/callhierarchy > @theia/core > @theia/application-package > request > [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
warning @theia/callhierarchy > @theia/core > @theia/application-package > request > [email protected]: this library is no longer supported
warning @theia/callhierarchy > @theia/monaco > @theia/filesystem > trash > xdg-trashdir > @sindresorhus/df > execa > [email protected]: cross-spawn no longer requires a build toolchain, use it instead
warning @theia/debug > [email protected]: This package has been renamed to @vscode/debugprotocol, please update to the new name
warning @theia/file-search > [email protected]: This package has been renamed to @vscode/ripgrep, please update to the new name
warning @theia/git > dugite-extra > dugite-no-gpl > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning @theia/plugin-ext > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning @theia/plugin-ext > [email protected]: This package has been renamed to @vscode/debugprotocol, please update to the new name
warning @theia/plugin-ext > @theia/search-in-workspace > [email protected]: This package has been renamed to @vscode/ripgrep, please update to the new name
warning @theia/plugin-ext-vscode > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning @theia/preview > @types/[email protected]: This is a stub types definition. highlight.js provides its own type definitions, so you do not need this installed.
warning @theia/search-in-workspace > [email protected]: This package has been renamed to @vscode/ripgrep, please update to the new name
warning @theia/cli > mocha > [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
warning @theia/cli > [email protected]: Version no longer supported. Upgrade to @latest
warning @theia/cli > mocha > chokidar > [email protected]: "Please update to latest v2.3 or v2.2"
warning @theia/cli > @theia/application-manager > electron-rebuild > node-gyp > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "@theia/debug > [email protected]" has unmet peer dependency "request@2.*.*".
warning "@theia/process > [email protected]" has unmet peer dependency "node-gyp@^8.3.0".
warning "@theia/scm > [email protected]" has unmet peer dependency "react@^0.14.0 || ^15.0.0 || ^16.0.0".
warning "@theia/scm > [email protected]" has unmet peer dependency "react-dom@^0.14.0 || ^15.0.0 || ^16.0.0".
warning "@theia/messages > [email protected]" has unmet peer dependency "react@>=16.3.3".
warning "@theia/messages > [email protected]" has unmet peer dependency "react-dom@>=16.3.3".
warning "@theia/preview > [email protected]" has unmet peer dependency "markdown-it@^8.4.1".
[4/4] Building fresh packages...
Done in 71.36s.
+ yarn theia download:plugins
yarn run v1.22.5
$ /opt/theia/node_modules/.bin/theia download:plugins
--- downloading plugins ---
+ vscode-builtin-css: downloaded successfully 
+ vscode-builtin-ini: downloaded successfully 
+ vscode-builtin-html: downloaded successfully 
+ vscode-builtin-less: downloaded successfully 
+ vscode-builtin-shellscript: downloaded successfully 
+ vscode-builtin-make: downloaded successfully 
+ vscode-builtin-log: downloaded successfully 
+ vscode-builtin-pug: downloaded successfully 
+ vscode-builtin-sql: downloaded successfully 
+ vscode-builtin-debug-auto-launch: downloaded successfully 
+ vscode-builtin-json: downloaded successfully 
+ vscode-builtin-markdown: downloaded successfully 
+ vscode-builtin-python: downloaded successfully 
+ vscode-builtin-configuration-editing: downloaded successfully 
+ vscode-builtin-xml: downloaded successfully 
+ vscode-builtin-merge-conflicts: downloaded successfully 
+ vscode-builtin-icon-theme-seti: downloaded successfully 
+ vscode-builtin-javascript: downloaded successfully 
+ vscode-builtin-yaml: downloaded successfully 
+ vscode-builtin-theme-defaults: downloaded successfully 
+ vscode-builtin-cpp: downloaded successfully 
+ vscode-editorconfig: downloaded successfully 
+ vscode-asm: downloaded successfully 
+ vscode-builtin-docker: downloaded successfully 
+ vscode-builtin-json-language-features: downloaded successfully 
+ webfreak-debug: downloaded successfully 
+ cdt-gdb-vscode: downloaded successfully 
+ vscode-clangd: downloaded successfully 
+ vscode-builtin-html-language-features: downloaded successfully 
+ plantuml: downloaded successfully 
+ vscode-jupyter: downloaded successfully 
--- collecting extension-packs ---
--- resolving 1 extension-packs ---
Error: Extension with id ms-toolsai.jupyter-renderers not found at https://open-vsx.org/api/-/query?extensionId=ms-toolsai.jupyter-renderers&includeAllVersions=true
    at OVSXClient.getAllVersions (/opt/theia/node_modules/@theia/ovsx-client/lib/ovsx-client.js:81:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async OVSXClient.getLatestCompatibleExtensionVersion (/opt/theia/node_modules/@theia/ovsx-client/lib/ovsx-client.js:99:28)
    at async /opt/theia/node_modules/@theia/cli/lib/download-plugins.js:68:35
    at async Promise.all (index 1)
    at async Object.downloadPlugins [as default] (/opt/theia/node_modules/@theia/cli/lib/download-plugins.js:67:13)
    at async Object.handler (/opt/theia/node_modules/@theia/cli/lib/theia.js:200:13)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
The command '/bin/sh -c set -ex;   yarn --pure-lockfile;   yarn theia download:plugins;   yarn --production;   yarn autoclean --force;   yarn cache clean;   find / -depth     \( -name .cache -o -name __pycache__ -o -name '*.pyc' -o -name .git -o -name .github \)     -exec 'rm' '-rf' '{}' '+';' returned a non-zero code: 1

wabscale avatar Feb 22 '22 04:02 wabscale

@wabscale John can you take over this PR? I'm sorry but there's a ton of stuff that I need to do right now and I can't test this image on my machine.

AlpacaMax avatar Feb 23 '22 04:02 AlpacaMax

Yeah, no problem. Good luck with your other stuff

wabscale avatar Feb 23 '22 15:02 wabscale