Evaluating Connected Health Interventions

Wednesday, December 16, 2009  | Adam Kaufman, PhD

About the Author: Adam Kaufman, PhD, is the Chief Operating Officer at DPS Health, and Adjunct Assistant Professor, Department of Economics, University of Southern California.

What gets measured gets done!  An expression used across many industries and processes to remind us of the important focus on quantifying and expressing outcomes.  It seems a fitting phrase to guide the self-management support interventions our industry builds and also, I would argue, to ensure we maintain our focus on quantifying and proving the outcomes of our solutions.  

Connected Health interventions, are seen as HEALTH interventions subject to the rigors of evidenced based medicine.  At DPS Health, we both develop new interventions and commercialize solutions in a Software as a Service model to healthcare provider organizations.  Every clinician who evaluates our software asks “what is your evidence?”  And the fact that our industry takes evidence seriously is what distinguishes us from the myriad of B2C offerings available directly on the web.

However, evaluating Connected Health programs is not simple and those of us who develop these interventions need to lead the way in terms of how we describe and measure effectiveness.  As I see it, there are four issues which complicate the measurement of Connected Health interventions:

1. The standard issue of population health programs and meaningful comparison groups
2. The selection bias which results from enrollment and user choice processes
3. The pace of technological evolution which evolves faster than the pace of research and publication
4. The issues of cost-effectiveness around accounting for development costs and staff time

At the symposium in October, we began a conversation about these issues and discussed how to (1) define the common elements of a Connected Health intervention and (2) work towards a structure to evaluate cost-effectiveness.  My intention in this post is to energize the community towards these issues.

Perhaps an illustration will serve as a talking point.  DPS Health has a research project with the University of Waterloo to automate the creation of draft coaching notes.  The idea is to develop an algorithm that coalesces the patient’s experience and success on the program to date with an effective rhetorical model and generate draft texts for a clinician.  The clinician would modify the text and send it to a patient through a secure messaging framework.  The objective is to help clinicians deliver more effective coaching more efficiently.  So, here is the challenge.  We intend to measure the impact of the solution in one domain -- most likely physical activity promotion.  We’ll run a small randomized trial and measure both clinician time and intervention effectiveness.  But, of course, we want to use the tool in other interventions as well.  With what rigor can we use the results of the small trial to claim the other solutions have been improved? 

Another intervention, the Stanford Chronic Disease Self Management Program has strong evidence for its in-person group program.  Several years ago, the University developed an online version with as much fealty as possible to the original. How much can we conclude about the efficacy of the online version from the evidence on the face-to-face version?  Stanford has conducted separate research on the online version but clearly this takes time and money.   It also freezes the technology of the online version to correspond to those trials.

Does each new intervention need its own research study?  Each extension or translation?  How large does each study need to be to show important results?  We probably cannot conduct RCTs for everything so what are appropriate levels of proof?  Since the digital tool is only one component of an multi-level intervention how are we to understand  its relationship to other components of the intervention? How do we measure cost-effectiveness as costs change overtime?  Just a small set of the questions.

I look forward to the continued conversation.

Member Comments

Adam: you've laid out the challenge(s) beautifully. Our experience is that rigorous RCT-like data is important early in the adoption process. Usually the question to be answered is 'does this intervention do anything to decrease the quality of care?" I say this because so many of the interventions we propose alter clinical work flow and challenge deeply ingrained assumptions about how health care is delivered. We have found that later in the adoption process, less rigorous evaluation criteria are imposed on us by those who might adopt our solutions. At that stage, there is a core belief that the solution 'works' and will provide high quality care. In this phase, the nuanced questions being asked have to do with finance, efficiency, optimal patient selection, optimizing workflow, etc. and the evaluative designs to support answering those questions often do not involve an RCT.

Just some experience to add to your well articulated post.

Joseph Kvedar

Director
Center for Connected Health

Joseph C. Kvedar, M.D., is the Founder and Director of the Center for Connected Health, applying communications technology and online resources to increase access and improve the delivery of quality medical services and patient care. Dr. Kvedar is also a board-certified dermatologist and Vice-Chair of Dermatology at Harvard Medical School. In his role with the Center for Connected Health, Dr. Kvedar launched the first physician-to-physician online consultation service in an academic setting. He is also leading important research into novel approaches for connected health in a variety of medical specialties, including post-operative care in the home, wound care, and remote monitoring of patients with chronic diseases. Dr. Kvedar is a past President and member of the governing Board of the American Telemedicine Association (ATA). Dr. Kvedar is also Chair of the American Academy of Dermatology (AAD) Task Force on Telemedicine. Dr. Kvedar is internationally recognized for his leadership and vision in the field of connected health and the application of communications technologies to improve healthcare to patients. Dr. Kvedar is co-editor of a new book, Home Telehealth, which was published in May 2006. He is a frequent lecturer and has authored over 60 publications on telemedicine and connected health. In 2003, Dr. Kvedar was honored by the New England Business and Technology Association for his extraordinary leadership in the field.

 

Posted by: Joseph Kvedar
12/17/2009

 

A fifth issue to consider is the lack of a clear, consistent way to describe and measure the barriers people experience when using the interventions. Chris Gibbons' recent systematic reviewfor AHRQ included a discouragingly long and motley list of barriers that were inconsistently described and measured across the studies in the review.

We need good standardized information on barriers to better understand how different groups use interventions, to inform intervention redesign and to track trends over time.

To the extent that Web-based interventions increasingly constitute a "field," identifying key barriers and standards for measuring them and then publishing them / enourage others to use them would contribute to its credibility and the value of its findings.

Is this something Connected Health could take on?

Jessie Gruman

President
Center for Advancing Health

 

Posted by: Jessie Gruman
12/17/2009

 

Jessie - thank you for post and adding the concept of barriers to the conversation. Certainly we need to think systematically about challenges people have in accessing and using the solutions we build.

I particularly appreciate your suggestion about being systematic. During the conversation on evaluation at the Symposium we talked about trying to define a meta-structure for a Connected Health intervention. The goal would be to define them in a way that we could develop evidence around particular elements and a rigorous way to describe what is in an intervention and what is not. I think the issue of the particular user demographics and segment is certainly part of that structure. We certainly would not want to claim that a solution shown to work with educated office workers would necessarily work for homeless individuals.

What we talked about as a group is trying to define those elements. Let me offer the following structure as a starting point for the conversation:
1. User Population
2. User Identification
2. Recruitment
3. Enrollment
4. Orientation
5. Early engagement (probably 1st month)
6. Intensive engagement (generally through 4-6 months)
7. Maintenance engagement
8. Discharge/graduation

The goal of the structure is to provide a language so we can talk about what is common in an intervention and what might be different. For example just looking at point 3 - recruitment. An intervention could recruit from:
- A clinician visit
- Proactive outreach from a registry
- General marketing to a population

When we talk about evidence for an intervention working and its evaluation, I would argue that we certainly want to describe the particular recruitment strategy. We might not trust that evidence on an intervention with recruitment through a clinician visit and referral is translatable to recruitment through general marketing.

Adam

Adam Kaufman

COO
DPS Health

 

Posted by: Adam Kaufman
12/21/2009

 

Your Comments

This feature is available for Connected Health members only. Please login now or register to continue.

Email Address

Password

Login