Furthermore… Evolving Thoughts on Evaluating Connected Health Interventions
Friday, August 20, 2010
| Adam Kaufman, PhD
About the Author: Adam Kaufman, PhD, is the President of DPS Health, and Adjunct Assistant Professor, Department of Economics, University of Southern California.
It has been almost a year since we last shared thoughts on the not always exciting, but crucial topic of program evaluation. Since that time, we here at DPS Health, have only seen an accelerating interest in Connected Health interventions that engage patients to adopt and sustain healthier behaviors. However, we see an increased interest and focus on outcomes. As we move past the innovation phase and through early adoption and quickly into the early majority we move past the evangelists – those who intuitively believe that these programs work – and into more skeptical audiences. This transition – what some would call acceleration through the S-curve – is NOT a given despite what appears a natural progression on the graphs.
Throughout the history of technology diffusion, many cases of innovations have climbed the adoption curve only to fail at the point I believe we now find Connected Health interventions.
So what can we do to ensure the continued momentum? First is to continue to develop great solutions that truly engage people and improve health. But I would argue that this will not be enough; we must continue to build the evidence that Connected Health programs improve health at an affordable cost in the context of clinical processes. And this remains a unique and profound challenge.
I want to applaud two initiatives, one ongoing and one new. First, DMAA continues to help lead the way in terms of models of program evaluation. The DMAA Outcomes Guidelines Report (Volume 4) is freely available from their website. And a team of staff and volunteers is at work on updating those guidelines. Second, The Leapfrog Group is about to begin offering a new due diligence service where they will review and confirm vendors’ outcome claims. These are important initiatives and many working on Connected solutions are involved, but these are not specific to Connected Health solutions.
I continue to believe that technology-based solutions offer unique challenges in terms of evaluation. In addition to the issues standard in population health programs such as selection bias, technology-based interventions present these additional evaluation challenges:
- Technology evolves faster than research studies can be completed and analyzed. You’ll encounter health interventionists who are used to slowly evolving interventions over decades while the pace of changes in
technology intervention is in months; and this has lead to consumer-oriented technologists who believe that we
can simply analyze system use trends to confirm effect and to guide intervention updates.
- Measuring program costs for development and process changes to calculate cost-effectiveness can be a real challange. And, as so much of the real costs are upfront, how do we consider this when determining the
cost-benefit and Return on Investment once programs go to scale?
I continue to be impressed by the energy and insights of the community revolving around the Center for Connected Health. I will continue to advocate that we spend more of our precious resource of time considering the methodological issues of evaluation. I believe we will all be well served and look forward to hearing your thoughts.