Pilot Testing & Evaluation Frameworks for Digital Health Solutions

Testing is a major component of an XPRIZE, and understanding the current best practices of digital health solution testing is essential in order to identify which parameters should be applied for this competition.

• What were your experiences with piloting digital health solutions in low-resource environments?
• What were your criterion for evaluation and success?
• What frameworks can be used to evaluate a digital health program?

Hello @RahulJindal, @SArora and @Fractalman - As you all have been involved with project Echo, We feel you will be able to answer Terry’s questions on piloting digital health solutions. Please share your experiences.

Hi @Davisthedoc, @Debbie_Rogers and @aassif_lg - As you all have successfully designed and implemented a Digital Health Solution programs - Please share your experience on a framework used to evaluate such a program / service.

Hi @Shashi, @TerryMulligan, At Project ECHO, with over 400 hubs, 850 programs in over 40 countries, we have a fair amount of experience piloting programs, and scaling them across disease areas and geographies in LMIC’s. One of the most important strategies is to engage partners and local experts, so that the cultural context is integrated and the the program is lead by people that are known and trusted.
Impact Evaluation is critical as well, and we use as part of a closed-loop process of continuous improvement. We use a tiered outcome model to assess the impact of telementoring for healthcare and/or other professionals. These levels are common across multiple training/education evaluation frameworks like Moore’s 7-level outcome model and Kirkpatrick’s model.

  • First we make sure people participate regularly, engage, and feel like they get something out of the program.
  • Then ​we see if they learned something or are more confident in their knowledge
  • Do they intend to use that knowledge to change their behaviors, or ideally, did they actually change some behaviors or practices?
  • Ultimately, we want to see if these changes in practice have an impact on patients. Are more patients treated? Do patients spend less time/money and miss less work to get treated? Do patients get better treatments? Do measures of disease severity improve? Are patients more satisfied with their care?
  • Finally, we want to see if any of these changes go beyond individual practitioners or patients. For example, do rates of disease burden go down in healthcare facilities or communities where a lot of practitioners participate in ECHO? Is there evidence that best practices are adopted by organizations or systems such as public insurance?
After a specific intervention like ECHO has been shown to be effective, we also use other frameworks to evaluate how well it is being scaled such as the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. This framework helps us think about why certain entities or practitioners do or do not adopt ECHO, and what factors need to be in place for an ECHO program to be sustainable. These frameworks help us assess the public health impact of an intervention. For example, you could have a really effective intervention but it's either difficult to access or very expensive, so it doesn't scale well. The goal is to have the most effective intervention that ALSO can scale at the level of having a public health impact.

@SArora - Thanks Dr. Sanjeev for sharing your experience on project ECHO and highlighting important points. It’s highly insightful.

Hello @shamakarkal, @dykki, @joshnesbit, @ClaireM and @dollendorf - As you all have experience with digital health initiatives and programs, we feel you may have inputs on piloting digital health solutions. Please share your experience on a framework or criterion used to evaluate such a program / service. Thanks.

Hi @Nvargas2, @acowlagi, @anitasmoore, @marschenrj and @baldhame9 - Please share your experience on piloting digital health solutions. Thanks.

Hello @SArora - thank you for the helpful evaluation methodologies. The Moore’s 7-level Outcome Model, Kirkpatrick’s Model and RE-AIM framework are especially helpful for us as we begin to design our competition parameters. Our team will be sure to follow up at a later time as we define benchmarks for our competition parameters.
Also, we completely agree “that the goal is to have the most effective intervention that ALSO can scale at the level of having a public health impact.” Thank you so much for your contribution.

Hi @rajpanda, @jda, @synhodo, @jonc101, @aroamer, @LeeStein - We feel you might have some thoughts on piloting a digital health solutions and/or evaluating a framework used for a digital health program. Please share your experiences. Thanks.

The outcome/evaluation frameworks @SArora and @TerryMulligan addressed are all relevant and sound good, I think.

I also think it depends on what the interventions are and who the target population is. Curing diseases vs. promoting health and wellness through digital health solutions? Children, mothers, older people, and/or all of them? I am not sure whether or not the XPrize team has already decided those specific features of the competition.

Effectiveness, process, and cost evaluations are often planned and written in most study protocols for digital health solutions, although not all studies cannot keep the promises.

Digital literacy can or should be considered as an outcome again depending on the target population and programs.

Thanks @Hongsoo for sharing your thoughts.

@Hongsoo ~ Thank you for those insights! You are correct, we haven’t yet determined the types of interventions or target populations, but I do think we are leaning towards promoting health and wellness as well as treatment. Still so much to flesh out, but it’s experts like you who help us narrow it all down. Thank you so much for your contribution!

@Shashi @TerryMulligan I am reaching out to invite you and the Xprize community to the upcoming virtual event, Reducing Disparities: Democratizing Access to Healthcare Globally, this **Friday 10:00am - 11:00am **MT. I will be joined by Dr. Soumya Swaminathan, Chief Scientist from WHO, and the session will be moderated by Dr. Naveen Rao, formerly head of Merck for Mothers, and now SVP, Health, of the Rockefeller Foundation. The event is being hosted by Catalyst2030 (founded by the Schwab Foundation, Skoll Foundation, Ashoka, and Echoing Green to spur collaboration and attain the SDG’s). It coincides with the United Nations General Assembly. Join the conversation and please feel free to share beyond the Xprize community. You can register here.
Let me know if you have any questions and hope to see you on Friday.

Thank you @SArora for Inviting us. I have registered.

@TerryMulligan @Shashi - thanks for your patience! On testing and assessment of digital tools, there has been work in India to create some assessment criteria for looking at digital tools for front-line workers. Amnesty LeFevere at Johns Hopkins can give you more details but I can email some of the early work; there will be a publication on it later this year. I am sure you have seen the WHO guidance/documents? Sharing links here -

  1. https://www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en/;
  2. https://www.who.int/reproductivehealth/publications/mhealth/classification-digital-health-interventions/en/;
  3. https://apps.who.int/iris/bitstream/handle/10665/252183/9789241511766-eng.pdf;jsessionid=B7457633E61BAB49A91CF9939267525B?sequence=1;
  4. https://apps.who.int/iris/bitstream/handle/10665/185238/9789241509510_eng.pdf?sequence=1

Thanks @shamakarkal for sharing these insightful resources and also dropping name of Dr. Amnesty LeFevere. Its definitely helping us understand the evaluation process of digital health solutions.

@Shashi - thanks - ca do an email connect with Amnesty if the team doesn’t know her already.

@Shashi can you also tell me what platform you are using for hosting this community?

@shamakarkal - Thanks Shama. It would be great if you could connect us with Amnesty.
We are using Vanilla platform for hosting this community.

@shamakarkal ~ Yes! Thank you so much for these resources! I’ve added them to our list for further research. Also very much appreciate the connection with Amnesty. Have a great week ahead!

@HeatherSutton - the connection to Amnesty should be in your inbox! Thanks and wishing you the same