Measuring HCD is Hard. Let’s Talk about Best Practices.

This piece was originally featured on the HCD Exchange blog.

Adolescents 360 (A360) is a project aimed at increasing access to and demand for modern contraception for girls aged 15 to 19, through country-specific interventions developed using human-centered design alongside other disciplines. The project which started in 2016, is run in Ethiopia, Nigeria and Tanzania. 

The project is implemented by Population Services International (PSI) and worked in partnership with IDEO.org, Society for Health Nigeria and the Center on the Developing Adolescent. It is externally evaluated by Itad, Avenir Health and the London School of Tropical Hygiene.

The ASRH interventions developed are Smart Start in Ethiopia, 9ja girls in Southern Nigeria, Matasa Matan Arewa in Northern Nigeria and Kuwa Mjanja in Tanzania. A360 is currently in its second phase and is focused on the sustainability of the interventions. 

In this webinar, A360, Itad and PSI explore the project phase one process evaluation and share reflections on their findings, challenges and opportunities for evaluating HCD programs. 

Panelists

• Meghan Cutherell, Senior Program Manager, Adolescents 360, PSI
• Stefanie Wallach, Associate Partner, Itad
• Heran Birhanu, Adolescent Insights Associate, HCDexchange & Young Designer, Smart Start
• Ibrahim Kamaldeen, ASRH Advocate and former Young Designer, Adolescent 360
• Izzy Quilter, Senior Consultant (Human Development), Itad

Highlights from the Webinar 

The evaluation of phase one is made up of three studies that were undertaken to understand how effective A360 has been and what changes are needed for future programming.

The studies are:

1. Process evaluation
2. Impact evaluation
3. Cross effectiveness studies

The webinar is focused on the process evaluation which explores how the A360 program was designed and if it achieved what it set out to achieve. The process evaluation was theory based and started out looking at how the A360 theory of change played out in practice. As the project progressed, the team developed user journeys for each country to look at the experience of the users as they interacted with the solutions. The process evaluation therefore ended up looking at the user journeys rather than the original theory of change. 

Some of the process evaluation key findings

• The human-centered design process brought rigor and innovation to A360 providing space to try out new ideas and bring them to fruition, and creating a mechanism to involve young people in design. However, engaging a multi-disciplinary consortium led to challenges.
• A360’s use of adaptive implementation and meaningful youth engagement complemented the HCD process and added value by building on the skills and mindsets fostered through the design process to help country teams respond to variable performance, challenges and opportunities.
• A360’s strong performance is underpinned by several common success factors: 
  • life and vocational skills messaging
  • flexible service delivery models
  • working through public health facilities
  • engaging government, trusted local stakeholders and husbands/parents. 

Challenges of that came up in the process evaluation 

1. Since HCD is fast moving and dynamic, decisions are made very quickly with less documentation around decision-making than might be seen in a more traditional design approach. This created some challenges for the evaluation especially when the team wanted to go back and look at what happened during the design phase. 
2. The team was aware of the potential of research fatigue because community members participated in both the design and evaluation process. They therefore tried to overcome this by combining both research and evaluation activities as much as possible. 
3. The implementing team (PSI) did not always have the time to fully engage with evaluation findings so the evaluation team (Itad) revamped how it shared findings. It used slide decks with key insights and sense-making workshops with country teams to discuss findings. 

Developing prototypes to help share process evaluation findings with a young audience 

Prior to the webinar, the HCDExchange and Itad teams collaborated to answer the question “How might we share the process evaluation findings with a young/youth audience?” To find some answers, we held a prototyping workshop with 8 young professionals who worked on the A360 project, from all three countries.

What is prototyping, you might ask?

“Design thinking is actually more about doing, and as an aspect of design thinking, prototyping was selected to model this workshop because it allowed us to build our thoughts and ideas into tangible forms. With prototyping the end result is not a generation of ideas but a tangible form of that idea.

Prototyping is also an experimental process that gives room for those participating to explore real world solutions on the spot. This also feeds into building the creative confidence of designers and all those engaged in prototyping.” – Heran Birhanu

We decided to use this approach rather than a focus group discussion or interview, to give us something tangible that we could then take and test. The youth participants broke out into two groups: young designers and young evaluators. Each was given a list of key findings from the evaluation and asked them to work towards a prototype for how we might share those findings with their respective audiences.

In this webinar, participants from the workshop shared an overview of their approach and the two prototypes that they developed.

Check out the prototypes and the workshop discussion flow here:

• Young designers prototype
• Young evaluators prototype

Webinar related links 

• Watch the webinar here
• Check out the  presentation on the A360 project
• Check out the presentation on the process evaluation
• Explore A360’s website 
• Keep the discussion going on the Community Learning Forum