jupyterlite build fails if --contents dir has any dir with a `.` in name

[yuvipanda]

Description

Running jupyter lite build --contents contents fails with the following error if there is any directory whose name starts with a ..

❯ jupyter lite build --contents contents                                                                             130 ↵
static:jupyter-lite.json
.  pre_status:static:jupyter-lite.json
    tarball:         jupyterlite-app-0.1.0-beta.5.tgz 8MB
    output:          /Users/yuvipanda/code/jupyterlite-data8/tmp/_output
    lite dir:        /Users/yuvipanda/code/jupyterlite-data8/tmp
    apps:            
    sourcemaps:      True
    unused packages: True
archive:archive
contents:contents
lite:jupyter-lite.json
mathjax:status
pyodide:pyodide
serve:contents
settings:overrides
translation:translation
.  status:archive:archive
[LiteBuildApp] No archive (yet): tmp-jupyterlite.tgz
.  status:contents:contents
    contents: 1 files
.  status:lite:jupyter-lite.json
[LiteBuildApp]     jupyter-lite.(json|ipynb): 0 files
.  status:mathjax:status
     jupyter-server-mathjax: /Users/yuvipanda/.local/share/virtualenvs/textbook/lib/python3.9/site-packages/jupyter_server_mathjax/static 
     configured MathJax dir: /Users/yuvipanda/code/jupyterlite-data8/tmp/traitlets.Undefined
     effective MathJax dir: /Users/yuvipanda/.local/share/virtualenvs/textbook/lib/python3.9/site-packages/jupyter_server_mathjax/static
.  status:pyodide:pyodide
     URL: None
 archive: []
   cache: 0 files
   local: 0 files
.  status:serve:contents
    url: http://127.0.0.1:8000/
    server: tornado
    headers:
.  status:settings:overrides
    overrides.json: 0
.  status:translation:translation
    translation files: 0 files
static:output_dir
.  pre_init:static:output_dir
static:unpack
.  init:static:unpack
federated_extensions:copy:ext:jupyterlab_pygments
.  pre_build:federated_extensions:copy:ext:jupyterlab_pygments
contents:copy:.binder/wat
mathjax:copy
translation:copy
.  build:contents:copy:.binder/wat
.  build:mathjax:copy
.  build:translation:copy
contents:contents:.binder
contents:contents:.
federated_extensions:patch
mathjax:patch
.  post_build:contents:contents:.binder
TaskError - taskid:post_build:contents:contents:.binder
PythonAction Error
Traceback (most recent call last):
  File "/Users/yuvipanda/.local/share/virtualenvs/textbook/lib/python3.9/site-packages/doit/action.py", line 461, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/Users/yuvipanda/.local/share/virtualenvs/textbook/lib/python3.9/site-packages/jupyterlite/addons/contents.py", line 170, in one_contents_path
    listing = fm.get(listing_path)
  File "/Users/yuvipanda/.local/share/virtualenvs/textbook/lib/python3.9/site-packages/jupyter_server/services/contents/filemanager.py", line 370, in get
    raise web.HTTPError(404, "No such file or directory: %s" % path)
tornado.web.HTTPError: HTTP 404: Not Found (No such file or directory: binder)

Reproduce

Create a directory named .binder, and just put any random file inside it.

mkdir -p contents/.binder
touch contents/.binder/runtime.txt
jupyter lite build --contents contents

Expected behavior

Context

Output from pip list:

❯ pip list                                                                                                           130 ↵
Package            Version
------------------ -------
cloudpickle        2.0.0
doit               0.36.0
entrypoints        0.4
importlib-metadata 4.11.3
jupyter-core       4.10.0
jupyterlite        0.1.0b5
pip                22.0.4
setuptools         60.10.0
traitlets          5.1.1
zipp               3.8.0

[bollwyvl]

nice: all's fair in love and reuse when pretending to be a server, i guess.

we have a bunch of preset ignored paths, and could add .binder, but i bet it would work if we did the equivalent of in jupyter_lite_config.json:

{
  "ContentsManager": {
    "allow_hidden": true
  }
}

of note though, some static hosts, no doubt much like us, ignore . files, so they might not end up getting served properly...

[yuvipanda]

Just getting back to this, @bollwyvl do you think allow_hidden will actually work if the server serves these?

[bollwyvl]

will actually work

To clarify, what is is your use case? Do you want these files, or not want these files indexed (which is what's failing) and just don't want the build to fail.

allow_hidden

Not sure, as there are more things at play, and the actual jupyter_server traitlets aren't in play... the above was just an example, and we have to fool the equivlant of jupyter_server and jupyterlab_server, an maybe @jupyterlab/services itself. Last I checked, there's also an overrides.json entry needed:

{
  "@jupyterlab/filebrowser-extension:browser": {
    "showHidden': true
  }
}

[yuvipanda]

@bollwyvl currently the build process just completely crashes if the content directory has any hidden files. I mostly just want it to not crash :)

[yuvipanda]

@bollwyvl I can't seem to implement your suggestion in https://github.com/jupyterlite/jupyterlite/issues/624#issuecomment-1112783184 - how would I specify the 'equivalent' of that in config? In jupyter_lite_config.json? elsewhere? sorry if I'm being dense

[bollwyvl]

It probably needs python changes to not break, and schema and potentially ts changes to actually work... Will have a look.

[bollwyvl]

https://github.com/jupyterlite/jupyterlite/pull/860

Haven't tried it out with all the overrides, etc. but at least it doesn't asplode.

[yuvipanda]

@bollwyvl yay! I tried to set that PR up locally to test, but got thwarted by not really being able to understand what is going on and why pip install -e . doesn't work (and the doit commands also seem to give me strange nested failures) :( I'm grateful y'all are keeping on top of this build system, the fact that it's doing all these complex things and works as well when released is amazing to me <3

Thank you very much for working on it, and I'll just test it out when it's out! \o/

[bollwyvl]

pip install -e . doesn't work

Welp, when pip install -e . doesn't decide to break that for pure python packages (has happened to flit, hatchling, etc. in the last few months) that is a nice property to have. Once we've gotten #854 out of the way, we'll only have one python package here, unless we still need a monorepo setup, because [extras] are hot trash... only thing worse is undeclared deps the user gets to find at ~~run~~ fail time.

doit commands also seem to give me strange nested failures

Such as?

doing all these complex things

Yerp, at present, there's like 10min of nested python and node stuff happening to get to a working build due to all the layers (e.g. .whl files hoisted to typescript imports :scream:). For this reason, I don't like pip install doing node stuff at this scale, because when it breaks, the extra layers of flavor-of-the-week-python-build-backend errors hide what might be far, far worse (node-gyp errors for some esoteric image optimizer, etc).

[yuvipanda]

@bollwyvl nvm, I got it to work! doit produced a lot of output and I was overwhelmed :D turns out I just needed to have yarn and node installed. I tested the PR and works :)