metacatui
metacatui copied to clipboard
NCEAS
Show a message in the Metadata View when a package is in the index queue
After creating or updating a data package with several data files the files do not show up on the metadata view page after saving and clicking "View your data package" in the "Files in this dataset" download section.
The files do eventually show up but not always right away.
After saving and clicking "View your data package"
Notice that no files show up
After refreshing the page many times
After refreshing the page many times the files slowly show up.
This is likely an indexing backlog issue. The delay is in waiting for the index queue to clear. We encounter it frequently, and its high on our priority list to be able to scale indexing more effectively. Can you check the indexing backlog when you see this delay to confirm that is the issue?
@mbjones I doubt there is much of a backlog as the example I posted here was from my local development box. Are there tuning parameters that could speed this up? Also from a UX standpoint what about having the UI take this potential for a backlong into account? Is it possible to give the user a message that the files are being indexed or some thing else that gives the user the confidence that their files weren't lost?
Yes, we've had those exact same thoughts and @laurenwalker could probably point you at other tickets on this issue. Definitely something we are working on. There are several lines of attack: 1) speed up individual index ops so they are much faster; 2) avoid index re-work on package updates (we recheck things for safety more than needed); 3) parallelize indexing workers to scale workers with the rate of incoming data sets; 4) provide inspection APIs to know what the queue looks like to provide user feedback. All of 1-3 are in different states of work right now. Its going to be a bit before this gets developed and tested, and is somewhat complicated by our need to test and deploy in multiple scenarios.
@mbjones Thanks for the update! Can you send me the issue numbers for these other tickets so that I may update the ESS-DIVE team?
We should be able to use the new ResourceMapResolver introduced in v 2.34.0 to find the resource map even if it's still being indexed.
