Gradual fetching of data by scrolling down in File Manger Application

[mahsashadi]

According to this issue, I am implementing gradual fetching by scrolling in FM app. I have two issues.

1. I need to detect when the whole data is fetched to stop fetching the next time scroll hits the bottom. So I think I need to initially get total number of directories and files of a path (without calling readdir api and getting the list) and save it into some state. I checked the stat api, but it does not provide this attribute.
2. I need to have the last file returned from server, but the list returned here is not in the same order as server response, due to some sorting done here.

[mahsashadi]

BTW, What is your idea about adding this pagination feature to your file manager app? As you suggest here, it can be used by adapters supported this interface in their api. If you agree, I can make a pull request when it is totally done.

[andersevenrud]

Hm.

Relying on either the total number of files or the last one to figure out when to stop does not sound like a good idea.

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

[mahsashadi]

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

Yes, seems a better idea, thanks :)

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

[andersevenrud]

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

It is possible just to use a page number :thinking:

const pageSize = 10
let currentPage = 0
let reachedEnd = false

function changedDirectory() {
  currentPage = 0
  reachedEnd = false
}

async function scrollReachedBottom() {
  if (reachedEnd) return
  const nextPage = currentPage + 1
  const result = await readdir('home:/', { page: nextPage, pageSize })
  reachedEnd = result.length < pageSize
  currentPage = nextPage
}

Now you have a offset on the server: page * pageSize.

[mahsashadi]

But it seems my storage pagination API does not support offset. https://docs.openstack.org/swift/latest/api/pagination.html

[andersevenrud]

I see. I'm not really sure how to solve this then without also adding sorting as an option (default true).

[mahsashadi]

So do you think you can add this feature? Or if it is needed to be done by myself, I can start working on that.

[andersevenrud]

Feel free to have a look at it. This would also mean that the file manager would have to run the sort manually to make it behave the same.

[mahsashadi]

Great :) Do you have any idea which parts of codes are potential for changing and also where this boolean option should be set.

[andersevenrud]

I was just thinking of adding it to the options and add the logic to the VFS function:

https://github.com/os-js/osjs-client/blob/385b38f3c4b113d1afaf5775c8402e5f22436a96/src/vfs.js#L84

https://github.com/os-js/osjs-client/blob/385b38f3c4b113d1afaf5775c8402e5f22436a96/src/vfs.js#L70-L75

[mahsashadi]

This would also mean that the file manager would have to run the sort manually to make it behave the same.

About this part, when a new page is fetched, I think sorting the accumulative list (previous pages+ new page) leads to a bad UX. The files of previous pages and the new one might be mixed, and the user can not scroll just among files of new page.

[andersevenrud]

I was only thinking about applying sorting per result.

Because in adapters that does not support this you'd only get one result.

[mahsashadi]

So by adding sortFiles option to this method as below:

// Handles directory listing result(s)
const handleDirectoryList = (path, options) => result =>
  Promise.resolve(result.map(stat => createFileIter(stat)))
    .then(result => transformReaddir(pathToObject(path), result, {
      showHiddenFiles: options.showHiddenFiles !== false,
      filter: options.filter,
      sortFiles: options.sortFiles
    }));

We just need to check sortFiles option in transformReaddir method and do sorting if it is true. This way it works :)

The question is where should I config this boolean option? Maybe it is needed to add this config to src/client/config.js:

  fileManager:{
    pageSize: 1000,
    sortFiles: false
  }

[andersevenrud]

The question is where should I config this boolean option?

Doesn't this just fit right into the FM readdir function ?

[mahsashadi]

BTW, What is your idea about adding this pagination feature to your file manager app? As you suggest here, it can be used by adapters supported this interface in their api. If you agree, I can make a pull request when it is totally done.

So when we complete this feature, are you going to add it to your FM app? If so is it possible to set in readdir as below?

const options = {
  showHiddenFiles: proc.settings.showHiddenFiles,
  sortFiles: false,
  page:{
    size:1000,
    number: state.currentPage,
    lastFetchedFile: state.lastFetchedFile
  }
};

In this way when sorting (deafult is true) is going to happen?

[andersevenrud]

If so is it possible to set in readdir as below?

Yes, that will be possible. However, it would be nice to agree on a default option and properties for "paging" so that it can work for all adapters, not just the one that you're making.

[mahsashadi]

For sure we should agree on paging options. I've just only used some options for testing the functionality.

Up to now, Three options seem necessary in my mind:

1. size of each page
2. number of current page (for storage services that use offset)
3. last fetched file name (for storage services like our case)

[mahsashadi]

Up to now, Three options seem necessary in my mind:

So what is your opinion ?

[mahsashadi]

I again find some time to work on paging project :) So if you have any time, it's good to agree on paging properties. Thanks a lot.

[andersevenrud]

So if you have any time

Sadly, I've been busy with a lot of other things. Those three options are fine, but I can also thing of another one: "pagination token". This is for examplle have in Google Drive. Maybe there are other options there to consider as well.

[mahsashadi]

Sometimes I fetch some pages in File manager app, i faced the following error (the page list is returned correctly)

Do you know what can cause this error?

[andersevenrud]

Hm. Pretty sure that this can occur if you remove something from the VDOM while it's updating. Does it cause any issues ?

[mahsashadi]

It causes new page list items does not show on the file manager app.

[andersevenrud]

Can you share the code that causes this ?

[mahsashadi]

And there is also another question. By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

If you agree, what is your suggestion for its implementation?

[mahsashadi]

Can you share the code that causes this ?

I do not have access right now. I will share it tomorrow 🙏

[andersevenrud]

By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

This would require getting some additional information, which is now not really possible in the VFS :thinking:

I'm starting to think maybe having a new scandir endpoint that is designed for doing things like this so that you get a proper server response:

{
  count: number, // Files in this response (page)
  totalFiles: number, // Total files in the directory
  totalPages: number, // Total number of pages
  currentPage: number | string, // An identifier that is used to identify the current page
  next?: {}, // Options for scandir to give next page ?
  prev?: {}, // Options to scandir give previois page ?
  data: VFSFile[]
}

I do not have access right now. I will share it tomorrow pray

Great!

[mahsashadi]

How about providing number of files and directories of each dir in stat response.

So we should call stat whenever a new mountpoint or directory is selected in app.

[andersevenrud]

Oh, yeah. I didn't think of that :man_facepalming: That could work!

[mahsashadi]

These are changes in filemanger application index file (till now):

const PAGE_SIZE = 10;

/**
 * Update some state properties when selected directory/file changed
 */
const updateState = (state) => {
  state.currentList = [];
  state.currentPage = 0;
  state.fetchAllPages = false;
  state.currentLastItem = '';
};

/**
 * Detect pagination capability when selected mountpoint changed
 */
const isPagingActive = async (core, path) => {
  const vfs = core.make('osjs/vfs');
  const capabilities = await vfs.capabilities(path);
  return capabilities.pagination;
};

/**
 * Create files list by concating new page items by previous fetched pages items
 */
const createPagesList = async (proc, state, vfs, dir) => {
  const options = {
    showHiddenFiles: proc.settings.showHiddenFiles,
    page:{
      size:PAGE_SIZE,
      number: state.currentPage,
      marker: state.currentLastItem,
      token: ''
    }
  };

  let list = await vfs.readdir(dir, options);
  // FIXME: the special file `..` should be handled in a better way (not add per page).
  if(list.length === 0 || (list.length === 1 && list[0].filename === '..')) {
    state.fetchAllPages = true;
  }
  if(list.length !== 0 && list !== state.currentList) {
    // FIXME: the special file `..` should be handled in a better way (not add per page).
    if(state.currentList.length !== 0 && list[0].filename === '..' && state.currentList[0].filename === '..') {
      list = list.filter(item => item.filename !== '..');
    }
    state.currentLastItem = list.length !== 0 ? list[list.length - 1].filename : null;
    state.currentList = state.currentList.concat(list);
    list = state.currentList;
  } else {
    list = state.currentList;
  }
  return list;
};

/**
 * Create total list of files when pagination is not supported
 */
const createTotalList = (proc, vfs, dir) => {
  const options = {
    showHiddenFiles: proc.settings.showHiddenFiles
  };
  return vfs.readdir(dir, options);
};

/**
 * VFS action Factory
 */
const vfsActionFactory = (core, proc, win, dialog, state) => {
  // ...
  const readdir = async (dir, history, selectFile) => {
    if (win.getState('loading')) {
      return;
    }
    // if calling by scroll
    if(dir === undefined) {
      dir = state.currentPath;
      state.currentPage += 1;
    }

    try {
      const message = __('LBL_LOADING', dir.path);
      win.setState('loading', true);
      win.emit('filemanager:status', message);
      let list;
      if(state.pagination) {
        list = await createPagesList(proc, state, vfs, dir);
      }else {
        list = await createTotalList(proc, vfs, dir);
      }
    // ...
};

/**
 * Creates a new FileManager user interface
 */
const createApplication = (core, proc, state) => {
   const createActions = (win) => ({
     // ...

    mountview: listView.actions({
      select: async ({data}) => {
        await updateState(state);
        state.pagination = await isPagingActive(core, data.root);
        win.emit('filemanager:navigate', {path: data.root});
      }
    }),

    fileview: listView.actions({
        // ...
        activate: async ({data}) => {
             await updateState(state);
             win.emit(`filemanager:${data.isFile ? 'open' : 'navigate'}`, data);
      },
        scroll: (ev) => {
        if (state.pagination) {
          const el = ev.target;
          if (state.fetchAllPages) {
            return;
          }
          const hitBottom = (el.scrollTop + el.offsetHeight) >= el.scrollHeight;
          if(hitBottom) {
            win.emit('filemanager:navigate');
          }
        }
      }
)}
 // ...
}

/**
 * Creates a new FileManager window
 */
const createWindow = async (core, proc) => {
  let wired;
  const state = {
    currentFile: undefined,
    currentPath: undefined,
    currentList: [],
    currentPage:0,
    fetchAllPages: false,
    currentLastItem:'',
    pagination: false
  };
  const {homePath, initialPath} = createInitialPaths(core, proc);
  state.pagination = await isPagingActive(core, initialPath);
  // ...
}

And I changed transformReaddir of src/utils/vfs.js as below:

const sortedSpecial = createSpecials(path)
   .sort(sorter(sortBy, sortDir))
   .map(modify);

 const sortFiles = files.filter(filterHidden)
   .filter(filter)
   .map(modify);

 return [
   ...sortedSpecial,
   ...sortFiles
 ];