Project-MONAI
Create a Survey of Institutional Workflows and Use Cases for MONAI Deploy
MONAI Deploy working group should produce a generalized survey that existing institutional members and new members contribute to that collects information related to the MONAI Workloads for (non-exhaustive)
- Use Cases with priority of deployment
- Types of imaging studies modalities
- Statistics on data characteristics
- other key AI data characteristics
The goal of the effort is to begin building a useful profile and catalog of best practices and state of the art descriptions for MONAI Deploy related workflows where this information can be further utilized with regular reports on MONAI Deploy use case and workflows and their characteristics in the community.
See draft here: https://docs.google.com/forms/d/1zHSS9exLLfivQlh9HxRUI3S8FLnznhlmShr5IIvARWo/edit
What's the best way to develop this? Happy for people to put comments here and I can make changes or can try convert to markdown (somehow?) and we can work on that until finalised and then convert back to Google Form.
I like the survey. Do we have any members of the working group who haven't yet provided this info, who could dry run the survey and see if it fulfills the purpose?
It's possible since we've had a few new members. We can ask during this Thursday's meeting.
Is it possible to add a question to institutions that, If you wanted to deploy AI based solutions at you institutions and you have not deployed it, what was the reason ? Some of the options are
- Lack of Resources (Financial and Man Power)
- Lack of acceptance because of ethical issues
- Lack of knowhow.
- Not enough financial benefits from AI based deployment.
- Lack of trust.
Some of the questions are asking about information that is probably publicly available (e.g. Number of studies per year etc.) We may want to consider sourcing this information independently, which will also make it consistent from different folks in the same institution, who also may not know all those figures.
To add to @vikashg's comment, how would people feel if the answers to those questions could be on a scale (1-5 + N/A) of how large an obstacle to deployment each factor is, then we could monitor improvements in each category in the case of this becoming an annual survey. e.g. could have evidence that in 2023 lack of trust was less of an obstacle to deployment than in 2022.
1 = very difficult obstacle 5 = not an obstacle e.g. Lack of resources | 1 2 3 4 5 N/A
Is the Radiology focus deliberate? AI can obviously be a part of other MONAI Deploy pipelines such as pathology, endoscopy etc. I appreciate that Radiology is probably the simplest option - just don't want to miss a trick unless it's deliberate.
I would be interested to know whether the respondents anticipate that AI will be provided as predominantly as: a) Cloud Services b) On Premises c) Hybrid
I think it is deliberate. It would be a much bigger endeavour to try and map things like pathology , endoscopy etc. It might be worth adding to next year's audit.
I think on-premise question is a good one. Should we allow some fuzzyness in options (a) and (b) by adding something like (>90% of deployments)? Let me know and I'll add it to the questionnaire.
Some of the questions are asking about information that is probably publicly available (e.g. Number of studies per year etc.) We may want to consider sourcing this information independently, which will also make it consistent from different folks in the same institution, who also may not know all those figures.
I agree we should do this where available.
To add to @vikashg's comment, how would people feel if the answers to those questions could be on a scale (1-5 + N/A) of how large an obstacle to deployment each factor is, then we could monitor improvements in each category in the case of this becoming an annual survey. e.g. could have evidence that in 2023 lack of trust was less of an obstacle to deployment than in 2022.
1 = very difficult obstacle 5 = not an obstacle e.g. Lack of resources | 1 2 3 4 5 N/A
I think this is a good idea, and seeing as it has some thumbs up, i'll add it
Is it possible to add a question to institutions that, If you wanted to deploy AI based solutions at you institutions and you have not deployed it, what was the reason ? Some of the options are
- Lack of Resources (Financial and Man Power)
- Lack of acceptance because of ethical issues
- Lack of knowhow.
- Not enough financial benefits from AI based deployment.
- Lack of trust.
Can we have some thoughts about the best set of 5 options we want to offer? @ristoh @laurencejackson @JHancox @dbericat @slbryson
I think the question should be something like : A major challenge in the routine deployment of AI is:
- funding (1 | 2 | 3 |4 | 5)
- technical infrastructure (integration, hardware, software) (1 | 2 | 3 |4 | 5)
- viable products (return on investment, performance) (1 | 2 | 3 |4 | 5)
- local expertise (1 | 2 | 3 |4 | 5)
- business process change (1 | 2 | 3 |4 | 5)
Another interesting response might be: Insufficient Clinical Demand, but I think what you have @hshuaib90 is good
I think what you've got here is great. The only thing I'd add is how do we capture general mistrust/resistance to AI from institutions? I think safety and liability concerns could be one of the primary obstacles to deployment, but not immediately clear which category this falls under? I think probably local expertise - in which case we could use some bracketed examples e.g.
- local expertise (technical, clinical and regulatory) (1 | 2 | 3 |4 | 5)
I think what you've got here is great. The only thing I'd add is how do we capture general mistrust/resistance to AI from institutions? I think safety and liability concerns could be one of the primary obstacles to deployment, but not immediately clear which category this falls under? I think probably local expertise - in which case we could use some bracketed examples e.g.
- local expertise (technical, clinical and regulatory) (1 | 2 | 3 |4 | 5)
Thanks, Laurence. I think we don't want to allow a general answer/option because it is of limited use when interpreting the results. I think what you're describing could be either local expertise or it could be viable products (i.e. there are no products that allay my fears around AI). I'll amend option 4 with your suggestion whilst awaiting other comments
I think on-premise question is a good one. Should we allow some fuzzyness in options (a) and (b) by adding something like (>90% of deployments)? Let me know and I'll add it to the questionnaire.
Yes, I think that would be fine @hshuaib90
I like the set of questions posed. Definitely keep the format of allowing the response to comment on "All" the choices with 1|2|3|... adding that 5 is strongly impacted or whatever the desired meaning of the range.
Survey looks good!
@hshuaib90 previously you mentioned, we could incentivize different ways to contribute to this effort. Should that be included in the survey?
i.e. Would you like to contribute more in the AI survey and MONAI Deploy WG effort?
- Yes, I could contribute as a reviewers of research [your name gets listed in the acknowledgement on the published paper]
- Yes, I could contribute a response and my opinion and insights on how my institution uses AI [your institution gets acknowlement on the published paper]
- Yes, I would like to join the MONAI Deploy WG as an active participant to define the future of AI deployment across institutions and industry
- Yes, other: (suggest ways to contribute)
- No, I'd rather focus on other things
@hshuaib90 if you think this is diluting the message, feel free to leave out.
Survey looks good!
@hshuaib90 previously you mentioned, we could incentivize different ways to contribute to this effort. Should that be included in the survey?
i.e. Would you like to contribute more in the AI survey and MONAI Deploy WG effort?
- Yes, I could contribute as a reviewers of research [your name gets listed in the acknowledgement on the published paper]
- Yes, I could contribute a response and my opinion and insights on how my institution uses AI [your institution gets acknowlement on the published paper]
- Yes, I would like to join the MONAI Deploy WG as an active participant to define the future of AI deployment across institutions and industry
- Yes, other: (suggest ways to contribute)
- No, I'd rather focus on other things
@hshuaib90 if you think this is diluting the message, feel free to leave out.
I think this is a great idea. I think it would be good to follow up with respondents with this ask as opposed to putting it in the initial survey, just to keep the survey as focussed as possible.
This is really cool effort! Thank you for this work! If I may, I have a few questions:
- Do you have aspirations for future surveys?
- How are you recruiting respondents? Wondering if we could help at all, or if you've already covered the folks we might bring along (e.g. the MICCAI crowd).
- Would you have interest in similar surveys aimed at infosec, IT, or IRBs?
Ralf Floca can elaborate here and on the issue #49 that we should be careful to clarify (if not already) on the survey they we are interested in responses for research workflows as well as clinical workflows. In some regions, when mentioning clinical workflows it may limit the response to Federally approved AI workflows (like FDA algorithms in the US) vs a broader category of research applied workflows please let me know if I captured correctly. And this would be research hospitals.
This is really cool effort! Thank you for this work! If I may, I have a few questions:
- Do you have aspirations for future surveys?
- How are you recruiting respondents? Wondering if we could help at all, or if you've already covered the folks we might bring along (e.g. the MICCAI crowd).
- Would you have interest in similar surveys aimed at infosec, IT, or IRBs?
Thanks @msheller ! Yes the plan is to update it annually, potentially expanding the scope. There's a list in the appendix of the Google doc , please add to that table! Currently just went through list from International Societies of Radiology.
Yes, I think it would definitely be worthwhile exploring other dimensions around AI deployment where we can either set a standard or report an analysis of the current state.
Ralf Floca can elaborate here and on the issue #49 that we should be careful to clarify (if not already) on the survey they we are interested in responses for research workflows as well as clinical workflows. In some regions, when mentioning clinical workflows it may limit the response to Federally approved AI workflows (like FDA algorithms in the US) vs a broader category of research applied workflows please let me know if I captured correctly. And this would be research hospitals.
Yes, in the end it revolves around the question which kind of AI workload you want to cover with the survey. If you also want to get things that are done with AI, but not in prospective manner in the regulatory sense (e.g. because it's the research and not yet in a regulatory compliant product), you should stated more explicitly in the introduction. And you should ask yourself about the granularity you want for those use cases. Currently you only have "retrospective study" in your list, which automatically could imply that everything else is prospectively. But then you could not differentiate between different types of projects in a retrospective setting (not sure if this is of interest for you though); e.g. in context of the research project you also could run "emergency diagnostic" in a "retrospective" setting (thus not impacting clinical decision) to see how it performs.
Hope that I got the point across.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.
Sorry @rfloca of course you are correct, I would consider RT planning in scope - i guess i see that as a medical imaging workflow (even though it does not happen in Radiology). Can you think of a good general way to describe that work so we can add as an option?
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.
Sorry @rfloca of course you are correct, I would consider RT planning in scope - i guess i see that as a medical imaging workflow (even though it does not happen in Radiology). Can you think of a good general way to describe that work so we can add as an option?
@rfloca We can add it within the Workloads document itself and we already have an open issue to add an example "streaming" use case to the Workloads document. Do you think it fits in that category? https://github.com/Project-MONAI/monai-deploy/issues/37 I can help with where we want to add it into the document if you can draft and initial description.
Just wanted to share this article about information-blocking rules: https://www.sciencedirect.com/science/article/pii/S0363018822000123
In particular, and concerning MONAI, some related articles have suggested that in order for providers to handle the volume of requests and comply with a reduced embargo period, AI might assist in generating patient-facing reports (from an AI perspective, the target being patients probably adds an extra layer of complexity, beyond the ethical issues mentioned in the article) This goes beyond the diagnostics stages mentioned in the survey, but maybe is a good idea to keep track of it, since providers could want to add this functionality to their clinical workflows in the future.
