Initial Issues Encountered While Setting Up miqa

[davidshq]

Just wanted to share the issues I ran into as I began setting up miqa:

1. I'm using Windows 10 Pro w/Windows Subsystem for Linux 2 running Ubuntu 20.04.
2. VS Code 1.56.0 is my editor and I'm using the Remote - Containers extension 0.177.0.
3. I cloned the miqa repository into my WSL2 Ubuntu instance and then from within the miqa folder launched VS Code (code .). This launches VS Code from within WSL.
4. I prefer to use an isolated Docker container for actual development and VSC's Remote - Containers extension handles this nicely (Command Palette --> Remote-Containers: Reopen in Remote Container --> allows to select from prebuilt templates). See later for contents of devcontainer.json and Dockerfile.
5. I reopen the project in the container.
6. I open VSC's Terminal (which is running within the container) and execute pip install -e miqa/server and receive the following warning/error:

WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /usr/local/include/python3.9/UNKNOWN
sysconfig: /usr/local/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Defaulting to user installation because normal site-packages is not writeable
ERROR: miqa/server is not a valid editable requirement. It should either be a path to a local project 
or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, 
git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, 
svn+ssh, svn+http, svn+https, svn+svn, svn+file).
WARNING: You are using pip version 21.1; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

1. Eliminating the first two warnings can be accomplished by heeding the third and upgrading pip: python -m pip install --upgrade pip.
2. Rerunning pip install -e miqa/server returns the same error without warnings:

Defaulting to user installation because normal site-packages is not writeable
ERROR: miqa/server is not a valid editable requirement. It should either be a path to a local project 
or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, 
git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, 
svn+ssh, svn+http, svn+https, svn+svn, svn+file).

1. I realize that in the repository there is no miqa folder only a server folder so I run instead pip install -e server and this works. I assume if I had navigated outside of the repo folder and run pip install -e miqa/server it would have worked...and strictly speaking the instructions in development.md don't say to enter the repo, may be worth adding verbiage noting that if one does enter the repo one needs to remove miqa/ from the pip command.
2. I run girder build, the first steps - installing npm packages run fine. As does @girder/meta-build@ build /home/vscode/.local/lib/python3.9/site-packages/girder/web_client.
3. girder build next attempts to run grunt "--girder-version=3.1.4" "--static-path=/usr/local/share/girder/static" --"static-public-path=/static" "--no-progress=false" "--env=prod" and while running the "webpack:core_lib" (webpack) task throws an error at 94% asset optimization:

Warning: EACCES: permission denied, mkdir '/usr/local/share/girder' Use --force to continue.

Aborted due to warnings.
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! @girder/meta-build@ build: `grunt "--girder-version=3.1.4" "--static-path=/usr/local/share/girder/static" 
"--static-public-path=/static" "--no-progress=false" "--env=prod"`
npm ERR! Exit status 3
npm ERR! 
npm ERR! Failed at the @girder/meta-build@ build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/vscode/.npm/_logs/2021-05-07T16_08_20_911Z-debug.log
Traceback (most recent call last):
  File "/home/vscode/.local/bin/girder", line 8, in <module>
    sys.exit(main())
  File "/home/vscode/.local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/vscode/.local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/vscode/.local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vscode/.local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vscode/.local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/vscode/.local/lib/python3.9/site-packages/girder/cli/build.py", line 104, in main
    check_call(buildCommand, cwd=staging)
  File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['npm', 'run', 'build', '--', '--girder-version=3.1.4', 
'--static-path=/usr/local/share/girder/static', '--static-public-path=/static', '--no-progress=false',
'--env=prod']' returned non-zero exit status 3.

This is where I am at the moment. I'll update if I make further progress.

Contents of devcontainer.json:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.166.1/containers/python-3
{
	"name": "Python 3",
	"build": {
		"dockerfile": "Dockerfile",
		"context": "..",
		"args": { 
			// Update 'VARIANT' to pick a Python version: 3, 3.6, 3.7, 3.8, 3.9
			"VARIANT": "3.9",
			// Options
			"INSTALL_NODE": "true",
			"NODE_VERSION": "lts/*"
		}
	},

	// Set *default* container specific settings.json values on container create.
	"settings": { 
		"terminal.integrated.shell.linux": "/bin/bash",
		"python.pythonPath": "/usr/local/bin/python",
		"python.linting.enabled": true,
		"python.linting.pylintEnabled": true,
		"python.formatting.autopep8Path": "/usr/local/py-utils/bin/autopep8",
		"python.formatting.blackPath": "/usr/local/py-utils/bin/black",
		"python.formatting.yapfPath": "/usr/local/py-utils/bin/yapf",
		"python.linting.banditPath": "/usr/local/py-utils/bin/bandit",
		"python.linting.flake8Path": "/usr/local/py-utils/bin/flake8",
		"python.linting.mypyPath": "/usr/local/py-utils/bin/mypy",
		"python.linting.pycodestylePath": "/usr/local/py-utils/bin/pycodestyle",
		"python.linting.pydocstylePath": "/usr/local/py-utils/bin/pydocstyle",
		"python.linting.pylintPath": "/usr/local/py-utils/bin/pylint"
	},

	// Add the IDs of extensions you want installed when the container is created.
	"extensions": [
		"ms-python.python"
	],

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "pip3 install --user -r requirements.txt",

	// Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
	"remoteUser": "vscode"
}

Contents of Dockerfile

# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.166.1/containers/python-3/.devcontainer/base.Dockerfile

# [Choice] Python version: 3, 3.9, 3.8, 3.7, 3.6
ARG VARIANT="3"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}

# [Option] Install Node.js
ARG INSTALL_NODE="true"
ARG NODE_VERSION="lts/*"
RUN if [ "${INSTALL_NODE}" = "true" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi

# [Optional] If your pip requirements rarely change, uncomment this section to add them to the image.
# COPY requirements.txt /tmp/pip-tmp/
# RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
#    && rm -rf /tmp/pip-tmp

# [Optional] Uncomment this section to install additional OS packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
#     && apt-get -y install --no-install-recommends <your-package-list-here>

# [Optional] Uncomment this line to install global node packages.
# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g <your-package-here>" 2>&1

[dzenanz]

I remember that some component didn't work with Python newer than 3.7 (I originally tried it with 3.8).

If you are interested in doing development on Windows, I could update the readme with my Windows-specific notes soon (rather than later).

[davidshq]

[re windows steps: If the notes are for using Windows and WSL2, seeing them would be great! I'm trying to avoid installing directly onto my Windows OS.]

I started fresh with a copy of Python 3.6 and ran into the same problem. The issue was that the user VSC automatically creates for use with the machine is not root and does not have permissions to create/edit files in /usr/local/share. I temporarily commented out the line "remoteUser": "vscode" in devcontainer.json to force it to make root the default user.

After that girder build almost seems to work...It succeeds in running "webpack:core_lib" (webpack) task as well as webpack:core_app, webpack:plugin_jobs, webpack:plugin_autojoin, copy:swagger, and copy:girder-swagger.

It then runs stylus:swagger and reports "1 file created" before throwing several warnings and finishing with "Done.":

Running "stylus:swagger" (stylus) task
>> 1 file created.
(node:7697) Warning: Accessing non-existent property 'lineno' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:7697) Warning: Accessing non-existent property 'column' of module exports inside circular dependency
(node:7697) Warning: Accessing non-existent property 'filename' of module exports inside circular dependency
(node:7697) Warning: Accessing non-existent property 'lineno' of module exports inside circular dependency
(node:7697) Warning: Accessing non-existent property 'column' of module exports inside circular dependency
(node:7697) Warning: Accessing non-existent property 'filename' of module exports inside circular dependency

I tried running girder serve and it starts up okay but as soon as someone (me) hits http://127.0.0.1:8080 it throws an error because it can't find /usr/local/share/girder/static/miqa/index.html:

[07/May/2021:19:25:23] HTTP 
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/usr/local/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond
    response.body = self.handler()
  File "/usr/local/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/workspaces/py/miqa/server/miqa_server/client_webroot.py", line 17, in GET
    file = open(os.path.join(constants.STATIC_ROOT_DIR, 'miqa', "index.html"), "r")
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/share/girder/static/miqa/index.html'
[07/May/2021:19:25:23] HTTP 
Request Headers:
  Remote-Addr: 127.0.0.1
  HOST: 127.0.0.1:8080
  CONNECTION: keep-alive
  SEC-CH-UA: " Not A;Brand";v="99", "Chromium";v="90", "Microsoft Edge";v="90"
  SEC-CH-UA-MOBILE: ?0
  DNT: 1
  UPGRADE-INSECURE-REQUESTS: 1
  USER-AGENT: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.51
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
  SEC-FETCH-SITE: none
  SEC-FETCH-MODE: navigate
  SEC-FETCH-USER: ?1
  SEC-FETCH-DEST: document
  ACCEPT-ENCODING: gzip, deflate, br
  ACCEPT-LANGUAGE: en-US,en;q=0.9
127.0.0.1 - - [07/May/2021:19:25:23] "GET / HTTP/1.1" 500 513 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.51"
127.0.0.1 - - [07/May/2021:19:25:23] "GET /favicon.ico HTTP/1.1" 405 257 "http://127.0.0.1:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.51"

I've verified the file does not exist at the given path but am not sure yet why it isn't being created...

[curtislisle]

In this application, girder is running only as a backend server and is not hosting the web page. Please try http://127.0.0.1:8080/girder to see if you see Girder's internal UI. The MIQA GUI is run separately using npm or yarn serving from the mica/client directory. Depending on your browser CORS settings you might have issues connecting between client and server since the ports are different. Please cd into the 'client' directory and do something similar to 'yarn install' and 'yarn serve'.

[curtislisle]

Also, girder uses mongoDB for internal local storage. You will need a mongoDB server running for 'girder serve' to be successful. In my experience, girder will usually stop within a few seconds if it cannot find a mongoDB instance to use for internal state management. This is probably not in the documentation.

[dzenanz]

Updated instructions (part of #54 and #52) are here: https://github.com/dzenanz/miqa/blob/with_learning-rebase/development.md

[davidshq]

Thanks @curtislisle, once I went to /girder it worked. girder does stop running after a few seconds if mongoDB is not available, not sure about the docs.

Thanks @dzenanz, I'll take a look at these too

[davidshq]

@curtislisle, @dzenanz:

TLDR; With a few tweaks I'm up and running. I have not attempted enabling the active learning components at this juncture.

• I had issues with CORS across multiple browsers. Was able to get past it by running Chrome with the --disable-web-security flag as described here
• Creating a user in girder, logging in with the user to the Vue front-end, creating a collection, adding an assetstore, and mapping to a local directory all worked fine.
• The Settings --> JSON path config took me a little while. In VSC one's code resides at /workspaces/rest_of/path. In my instance this meant changing the default ~/miqa/sample_data/sample.json (and export path similarly) to /workspaces/py/miqa/sample_data/sample.json. I was able to save the path after doing so.
• Attempting to import I ran into a related issue. I had to edit sample.json data_root from /miqa/sample_data to /workspaces/py/miqa/sample_data, then I was able to import.
• It appears I'm up and running at this point although I have not attempted to setup the active learning components as outlined in @dzenanz's updated development doc.

I've written this all out in this issue in case there is an opportunity for me (or someone) to smooth over a few of the bumps I encountered.

[curtislisle]

@davidshq , glad to hear you have the system built and running. The issues you encountered largely reflect on our incomplete developer documentation. Thanks for the comments and testing.