Discussion: auto-updating analyses, and expressing Microsetta processing as Qiita analysis

[wasade]

On the construction of the Microsetta results pages, @rob-knight suggested allowing participants to link directly to an analyses in Qiita depicting meta-analyses used in the results report.

To do this, I believe the following items would need to be assessed for feasibility and resolved:

• Automatically updatable analyses, where the URL for an analysis can remain static but the artifacts/visuals within the analysis are recomputed from a triggering event (such as the creation of a new preparation)
• A way to express an analysis workflow that is in common between Microsetta and Qiita, perhaps through something like CWL, or we figure out how to migrate Microsetta processing steps into a Qiita workflow
• Either manual updates to metadata variables for studies used in meta-analysis for presentation purposes, or an ability for Qiita to perform a configurable metadata variable creation within an analysis
• The ability to open up a Qiita analysis such that on page load it shows the PCoA artifact with a predefined orientation and coloring

This may create an opportunity to move the entirety of Microsetta processing into Qiita, which would be very nice.

[rob-knight]

Is the Microsetta processing currently completely independent of qiita? If yes, is that a feature or a bug?

On Mar 31, 2021, at 1:19 PM, Daniel McDonald @.***> wrote:

On the construction of the Microsetta results pages, @rob-knight https://urldefense.com/v3/__https://github.com/rob-knight__;!!Mih3wA!Un6D6ogiLHXujRNK1al-d2iCfI7hmJdQpSbD9PdmtTrm9ocQWyAfZ03oY0jRMczwlg$ suggested allowing participants to link directly to an analyses in Qiita depicting meta-analyses used in the results report.

To do this, I believe the following items would need to be assessed for feasibility and resolved:

Automatically updatable analyses, where the URL for an analysis can remain static but the artifacts/visuals within the analysis are recomputed from a triggering event (such as the creation of a new preparation) A way to express an analysis workflow that is in common between Microsetta and Qiita, perhaps through something like CWL, or we figure out how to migrate Microsetta processing steps into a Qiita workflow Either manual updates to metadata variables for studies used in meta-analysis for presentation purposes, or an ability for Qiita to perform a configurable metadata variable creation within an analysis The ability to open up a Qiita analysis such that on page load it shows the PCoA artifact with a predefined orientation and coloring This may create an opportunity to move the entirety of Microsetta processing into Qiita, which would be very nice.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/qiita-spots/qiita/issues/3088__;!!Mih3wA!Un6D6ogiLHXujRNK1al-d2iCfI7hmJdQpSbD9PdmtTrm9ocQWyAfZ03oY0h424bo2A$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAPL2KJ47HBYQVYDVOPKDJ3TGN7WDANCNFSM42FOY66Q__;!!Mih3wA!Un6D6ogiLHXujRNK1al-d2iCfI7hmJdQpSbD9PdmtTrm9ocQWyAfZ03oY0iDZv7ThA$.

[antgonza]

• A way to express an analysis workflow that is in common between Microsetta and Qiita, perhaps through something like CWL, or we figure out how to migrate Microsetta processing steps into a Qiita workflow

Perhaps it will be "easy" to use the DefaultWorkflow; I think adding a flag to differentiate between sequence processing and analysis would be a good start. Note that the steps are created and stored in the DB and we could add whatever exporter needed. In case you want to see how's displayed in the GUI: https://qiita.ucsd.edu/workflows/

• Automatically updatable analyses, where the URL for an analysis can remain static but the artifacts/visuals within the analysis are recomputed from a triggering event (such as the creation of a new preparation)
• Either manual updates to metadata variables for studies used in meta-analysis for presentation purposes, or an ability for Qiita to perform a configurable metadata variable creation within an analysis

This might be easier if we consider what you are suggesting here: a full update of the analysis vs. just updating the metadata; where the same jobs/artifact ids are simply rewritten. Questions here: do we need to keep a backup of the old artifact? what should happen if a job fails (for example cause a metadata column doesn't exist or the values are now wrong)?

• The ability to open up a Qiita analysis such that on page load it shows the PCoA artifact with a predefined orientation and coloring

Note that all the visualizations have direct links, for example so if the tool that it's being used for the visualization accepts the configuration via the URL we can store and add it via the DB. I can imagine the example above where you also have a link to the main analysis page that generated that plot.

[wasade]

@rob-knight, it is dependent on Qiita as it sources processed feature tables through redbiom. However it does not utilize Qiita for derived components from feature tables. I think that we should implement it fully in Qiita, but we have not finalized all of the processing steps.

Thanks, @antgonza! I'll begin to review the DefaultWorkflow -- that is super helpful. With the metadata variables, we do not need to keep a backup. At the moment, it is important to create new metadata columns for the metadata served by the API. And that is excellent with the direct link with emperor!