Measuring a ‘Moving Target’

Posted Maio 4, 2022
Impact Area Adolescents & Youth

This article originally appeared on HCDExchange.

Human-centred design (HCD) is increasingly being used as a complementary approach to traditional global health programming methods to design and optimise innovative solutions. HCD has a unique ability to bring new ideas to entrenched problems, integrate multiple stakeholder perspectives, and bring in a strong human lens. Differences between conventional evaluation approaches and the inherent fluidity that is required for HCD and adaptive implementation invariably makes it challenging for practitioners to think about how to best evaluate HCD or adaptive programs.

To address this knowledge gap, our webinar examined some of the roadblocks in evaluating health projects rooted in iterative approaches with experts from PSI’s Adolescents 360 (A360) , YLabs’ CyberRwanda and Vihara Innovation Network.

Program interventions such as PSI’s A360 and YLabs’ CyberRwanda’s use HCD and adaptive implementation approaches to ensure that they are designing and implementing the right solutions.

Dive into webinar highlights and engaging discussion questions below!

Combining HCD and AI approaches to attain robust evidence and greater impact

The HCD process paves a path toward implementation and evaluation by ensuring that the ultimate intervention is desirable among many key stakeholders, and is feasible for implementation and scale-up. Using the adaptive implementation approach, the findings from routine user testing informs iterations to the program.

Applying HCD during the implementation and evaluation stages of the intervention lends itself well to adaptive learning and iteration; allowing for new information to be continuously introduced. This is because HCD is inherently collaborative. For instance, CyberRwanda has been able to use interviews, focus group discussions, and co-design sessions to better understand the quantitative data.

For A360, HCD and adaptive implementation approaches are complimentary.

“We’ve found the two approaches very complementary. They rely on a culture of curiosity and ‘human-centeredness,’ that harnesses diverse perspectives to elicit and interpret insights and evidence, employ a collaborative and iterative approach to generating solutions, and entrust those closest to the point of impact to lead.”
—Matthew Wilson, Project Director, A360

Chiefly, HCD may help to support the long-term sustainability and cost-efficiency of the intervention by addressing implementation challenges early and often, rather than relying solely on endline results.

Challenges of conventional outcome evaluations

The design of the intervention & the implementation strategy is not known at the outset; ranging from who the intervention is directed at, how it will be implemented, where it will be implemented, and with what intensity it will be implemented.
Practitioners cannot assume consistency in the intervention design and implementation strategy as it evolves over the life of the project.
Documentation of the iterative design process, decision making, testing along the way is either absent or incompatible with what is an acceptable measurement metric.
Limitations with financial resources and almost linear sequencing of each discipline.
A tenuous relationship between different disciplinary teams, each silently questioning the validity of the other.
Differing learning priorities and styles of different disciplines, thus limited common language
Historically designers have limited understanding and no incorporation of measurement priorities in the design process

“The dream would be to see these disciplines kind of bring their individual strengths to bear but for the boundaries to blur, small cross-disciplinary, hybrid design and measurement teams working on a day to day basis together, optimising implementation strategies for outcomes.”
—Divya Datta, Director of Design Strategy and Innovation, Vihara Innovation Network

Mitigating challenges to rigorously evaluate AI and HCD interventions

Keeping the ‘essence’ of the intervention in mind: Through finding the balance between adaptation and staying true to the ‘essence’ of the intervention, drifting can be avoided, despite constant iteration. This requires a shift in the mindset of implementers, becoming systematic, and an investment in capabilities.
Evaluation designs need to be responsive and adaptive too: This has implications for implementers, evaluators and donors. This means implementers and evaluators would benefit from having a high touch engagement and open communication so there is an intimate and timely understanding of the intervention design and implementation strategy.
Implementers also need to have a robust understanding of evaluation protocols: This will enable them to understand the implications of potential or actual changes in the intervention design or implementation strategy.
Implementers need to manage adaptation effectively: This will enable them to maintain fidelity to the ‘essence’ of their interventions and avoid drift.
Think beyond outcome evaluations: Outcome evaluations essentially tell two parts of the story – the baseline and the endline. But we also need process/developmental evaluations that help tell the story of what’s happened in between.
Evaluators need to be resourced to reconcile results emerging from different evaluation approaches: the sense making, with more emphasis placed on evaluating how well a programme learns and adapts.