Add jsTree based FileTreeSelector

[philippjfr]

Adapts the panel-jstree components to support selecting both local and remote filesystems.

import s3fs
import panel as pn

from panel.widgets.tree import RemoteFileProvider

fs = s3fs.S3FileSystem(anon=True)
provider = RemoteFileProvider(fs=fs)

pn.widgets.FileTree(directory='s3://datasets.holoviz.org', provider=provider).servable()

Done

• [x] Clean up RemoteFileProvider
• [x] Display loading icons while fetching directories

ToDo

• [x] Add composite component that adds additional controls for navigating different directories
• [x] Figure out how to efficiently update nodes as you are navigating directories (without losing state)
• [x] Allow navigating up and down a directory tree while respecting a max_depth parameter setting
• [x] Consider whether to create file provider internally and simply let the user provide a fsspec FileSystem
• [ ] Add tests
• [x] Add docs
• [ ] Dark mode handling
• [ ] Make the generic Tree implementation usable (or hide it for now)

Nice To Have

• [ ] Allow asynchronously fetching directories
• [ ] Automatically infer the protocol (e.g. s3://) and create a FileSystem

[madeline-scyphers]

This is great. It was always my hope that panel-jstree would get put into panel someday. I am gonna put some comments on some parts for the things I was trying to work on when I had time.

The next thing I was trying to do for a side project I help with that was trying to use panel-jstree was create a composite widget, which here https://github.com/madeline-scyphers/panel-jstree/blob/1e364fcf5ff1947912ce8ba2b1f4d9753c4498d8/src/panel_jstree/widgets/jstree.py#L290

    def _set_data_from_directory(self, *event):
        self._data = [{"id": fullpath(self.directory),
                       "text": Path(self.directory).name,
                       "icon": self._folder_icon,
                       "state": {"opened": True},
                       "children": self._get_children_cb(Path(self.directory).name, self.directory,  depth=1)
                       }]

is a FileTree cb triggered on the directory change to swap out the entire data from scratch. I found this worked, but was a bit laggy (lag came from recreate the tree in the browser, not the directory search). Maybe the asynchronous tree will help, but if not, I was trying to play around with the massload plugin to see if that could also help but I never finished it with trying to finish my master's https://www.jstree.com/api/#/?f=$.jstree.defaults.massload

One thing I also wanted to mention was that I was trying really hard to make sure that there was a general tree implementation someone could use that would be independent of a FileTree if they just wanted to explore Tree data. It looks like that is still working properly, but I just want to underscore that I think that is really important, and ideally as many features can be generalized to work for a generalized Tree, not just a FileTree.

I added support for a number of jsTree's plugins, but there are a few more that might be cool to do. drag and drop was one I wanted to do next, which allows the user to just rearrange the Tree, though by default it can move a leaf to another node, so maybe not for the FileTree. Sort also is another one that might be nice. There is a search plugin as well.

That is most of most of what I had left that I wanted to do. I think the remote file provider is super cool, as well as the other features you all are adding. I would love to help with the last things to get this in.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 64.01766% with 163 lines in your changes missing coverage. Please review.

Project coverage is 81.54%. Comparing base (70a27c9) to head (3b5f675).

Files	Patch %	Lines
panel/widgets/file_selector.py	71.69%	75 Missing :warning:
panel/widgets/tree.py	55.55%	52 Missing :warning:
panel/models/jstree.py	0.00%	34 Missing :warning:
panel/compiler.py	0.00%	1 Missing :warning:
panel/util/__init__.py	50.00%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6837      +/-   ##
==========================================
- Coverage   81.73%   81.54%   -0.19%     
==========================================
  Files         326      328       +2     
  Lines       48006    48376     +370     
==========================================
+ Hits        39236    39448     +212     
- Misses       8770     8928     +158     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[hoxbro]

Example of max_depth:

import s3fs
import panel as pn

fs = s3fs.S3FileSystem(anon=True)
pn.widgets.FileTree(directory="s3://datasets.holoviz.org", fs=fs, max_depth=2).servable()

[philippjfr]

A little FileTreeSelector demo:

[madeline-scyphers]

So for my job, I am building a dashboard for some genome taxonomy data that is the output of a model. And I am trying to use my version of panel-jstree and eventually this version to filter based on taxonomy rank (domain, phylum, etc.). While doing this, I have a little function that allows me to convert an edgelist dataframe into a format that jstree needs.

So this edgelist df

  source target
0      a    a-1
1      a    a-2
2      b    b-1
3      b    b-2
4      b    b-3
5    b-1  b-1-1
6    b-2  b-2-1
7    b-2  b-2-2

Gets transformed into this node list

>>>build_tree(edge_df, state={"opened": False, "selected": False})
[{'text': 'a',
  'children': [{'text': 'a-1',
    'children': [],
    'state': {'opened': False, 'selected': False}},
   {'text': 'a-2',
    'children': [],
    'state': {'opened': False, 'selected': False}}],
  'state': {'opened': False, 'selected': False}},
 {'text': 'b',
  'children': [{'text': 'b-1',
    'children': [{'text': 'b-1-1',
      'children': [],
      'state': {'opened': False, 'selected': False}}],
    'state': {'opened': False, 'selected': False}},
   {'text': 'b-2',
    'children': [{'text': 'b-2-1',
      'children': [],
      'state': {'opened': False, 'selected': False}},
     {'text': 'b-2-2',
      'children': [],
      'state': {'opened': False, 'selected': False}}],
    'state': {'opened': False, 'selected': False}},
   {'text': 'b-3',
    'children': [],
    'state': {'opened': False, 'selected': False}}],
  'state': {'opened': False, 'selected': False}}]

And I was thinking it would be nice to be able to construct the generic Tree class from a edgelist df, maybe as a class_method? Tree.from_edgelist(df)

I can put in a PR into this branch to add that. It isn't that complicated of a function, and I tried to do it without something like NetworkX to not add any dependencies. Though I imagine that would speed it up. So there could be an optional dependency on NetworkX to speed it up?

[philippjfr]

Thanks for all your helpful review comments @madeline-scyphers! Will try to get back to this PR soon.