Evaluating Program Effectiveness: Key Concepts and How to Use Coarsened Exact Matching

CEMGuide.zip

CEMGuide1.zip

PSI uses impact evaluation to determine if our programs are working. One approach to impact evaluation is the use of coarsened exact matching, a statistical technique that allows researchers to compare program participants to nonparticipants, controlling for factors believed to influence outcomes of interest. Because it clearly shows comparisons based on program exposure, CEM analysis is a useful method for demonstrating the impact of our work.

This guide defines impact evaluation and CEM further and provides instruction on how to design studies to facilitate CEM, analyze the data using CEM, and interpret and report the results.

Resource Types: Guides and Toolkits
Languages: English

Your Investment,
Real Impact

92%

Program Expenses

Management, General, Fundraising Expenses

$38.96

is all it takes to give a healthy mother and her child a year of healthy life.

Donate

Careers

Recursos

1120 19th Street, NW, Suite 600, Washington, DC 20036 P (202) 785-0072 F (202) 785-0120

GDPR Compliance

Anti-Slavery Statement

Integrity and Protection Hotline

PSI

Digitalizing contraceptive counseling to reach rural women and girls in Ethiopia

By: Fana Abay, Marketing and Communications Director, PSI Ethiopia

In rural Ethiopia, women and girls often face significant barriers in accessing healthcare facilities, which can be located hours away. Moreover, there is a prevailing stigma surrounding the use of contraception, with concerns about potential infertility or the perception of promiscuity. To address these challenges, the Smart Start initiative has emerged, linking financial well-being with family planning through clear and relatable messaging that addresses the immediate needs of young couples—planning for the lives and families they envision. Smart Start takes a community-based approach, utilizing a network of dedicated Navigators who engage with women in their localities. These Navigators provide counseling and refer interested clients to Health Extension Workers or healthcare providers within Marie Stopes International-operated clinics for comprehensive contraceptive counseling and services.

In a significant development, PSI Ethiopia has digitized the proven counseling messaging of Smart Start, expanding its reach to more adolescent girls, young women, and couples. This approach aligns with the priorities set by the Ethiopian Ministry of Health (MOH) and is made possible through funding from Global Affairs Canada. The interactive and engaging digital messaging has revolutionized counseling services, enabling clients to make informed and confident decisions regarding both their finances and contraceptive choices.

Clients who received counseling with the digital Smart Start tool reported a higher understanding of their options and were more likely to choose contraception (74 percent) compared to those counseled with the manual version of Smart Start (64 percent). Navigators also found the digital tool more effective in connecting with clients, leading to higher ratings for the quality of their counseling.

By December 2023, PSI Ethiopia, working in close collaboration with the MOH, aims to reach over 50 thousand new clients by leveraging the digital counseling tool offered by Smart Start. This innovative approach allows for greater accessibility and effectiveness in providing sexual and reproductive health services, contributing to improved reproductive health outcomes for women and couples across the country.

Building community health worker capacity to deliver malaria care

By: Christopher Lourenço, Deputy Director, Malaria, PSI Global

Community health workers (CHWs) are critical lifelines in their communities. Ensuring they have the training, support, and equipment they need is essential to keep their communities safe from malaria, especially in the hardest to reach contexts.

For example, in Mali, access to formal health services remains challenging, with four in ten people living several miles from the nearest health center, all without reliable transportation or access. In 2009, the Ministry of Health adopted a community health strategy to reach this population. The U.S. President’s Malaria Initiative (PMI) Impact Malaria project, funded by USAID and led by PSI, supports the Ministry with CHW training and supervision to localize health services.

In 2022, 328 thousand malaria cases were recorded by CHWs); 6.5 thousand severe malaria cases were referred to health centers, according to the national health information system.

During that time, the PMI Impact Malaria project (IM) designed and supported two rounds of supportive supervision of 123 CHWs in their workplaces in the IM-supported regions of Kayes and Koulikoro. This included developing and digitizing a standardized supervision checklist; and developing a methodology for selecting which CHWs to visit. Once a long list of CHW sites had been determined as accessible to supervisors for a day trip (including security reasons), the supervisors telephoned the CHWs to check when they would be available to receive a visit [as being a CHW is not a full-time job, and certain times of the year they are busy with agricultural work (planting, harvesting) or supporting health campaigns like mosquito net distribution].

Supervisors directly observed how CHWs performed malaria rapid diagnostic tests (RDTs) and administered artemisinin-based combination therapy (ACT). They recorded CHW performance using the digitized checklist, interviewed community members, reviewed records, and provided on-the-spot coaching. They also interviewed the CHWs and tried to resolve challenges they expressed, including with resupply of commodities or equipment immediately or soon afterwards.

Beyond the observed interactions with patients, supervisors heard from community members that they were pleased that CHWs were able to provide essential malaria services in the community. And the data shows the impact.

In IM-supported areas of Mali, 36% of CHWs in the first round were competent in performing the RDT, which rose to 53% in the second. 24% of CHWs in the first round compared to 38% in the second were competent in the treatment of fever cases and pre-referral counseling. Between both rounds, availability of ACT increased from 80 percent to 90 percent.

Supportive supervision with interviews and observations at sites improved the basic competencies of CHWs between the first and second rounds, and additional rounds will help to understand the longer-term programmatic benefits.

Taking a market-based approach to scale sanitation in Ethiopia

By: Dr. Dorothy Balaba, Country Representative, PSI Ethiopia

In Ethiopia, PSI leads the implementation of USAID Transform WASH (T/WASH) activity with consortium partners, SNV and IRC WASH. Contrary to traditional models that rely on distribution of free or heavily subsidized sanitation products, T/WASH utilizes a market-based sanitation approach. This approach creates sustainable and affordable solutions, by integrating market forces and supporting businesses to grow, while creating demand at the household level.

During the last six years, T/WASH has worked alongside the private sector and government (Ethiopia’s Ministry of Health, Ministry of Water and Energy, and Ministry of Labor and Skills), among other stakeholders, to increase household access to affordable, quality sanitation products and services. For example, more than 158 thousand households have invested in upgraded sanitation solutions with rapid expansion to come as the initiative scales and market growth accelerates.

T/WASH has successfully trained more than 500 small businesses, including community masons and other construction-related enterprises, with technical know-how in sanitation product installation, operational capacities, and marketing and sales skills needed to run successful, growing businesses. The Ethiopian government is now scaling the approach to all districts through various national, regional, and local institutions with requisite expertise. T/WASH has also worked the One WASH National Program, Ministry of Health, Ministry of Water and Energy, and Ministry of Labor and Skills to examine policies that influence increased household uptake of basic WASH services, such as targeted sanitation subsidies, tax reduction to increase affordability, and increased access to loan capital for business seeking to expand and households needing help to improve their facilities.

To share the journey to market-based sanitation, representatives of the Ethiopian Ministry of Health and the USAID Transform WASH team took to the stage at the UN Water Conference in 2023.

“Rather than relying on traditional aid models that often distribute free or heavily subsidized sanitation products, market-based sanitation creates sustainable and affordable solutions, integrating market forces and supporting businesses to grow.”

— Michael Negash, Deputy Chief Party of T/WASH

Promoting self-managed care like Self-testing and Self-Sampling

By: Dr Karin Hatzold, Associate Director HIV/TB/Hepatitis

Building upon the success and insights gained from our work with HIV self-testing (HIVST), PSI is actively applying this approach to better integrate self-care, more broadly, in the health system beginning with Hepatitis C and COVID-19. Self-testing has emerged as a powerful tool to increase access to integrated, differentiated, and decentralized health services, accelerating prevention, care, and treatment for various diseases, while also increasing health system resilience against COVID-19.

Here’s how we got there.

Seven years ago, the landscape of HIV self-testing lacked global guidelines, and only the U.S., the UK and France had policies in place that allowed for HIV self-testing. High disease burdened countries in low-and-middle-income-countries (LMICs) lacked evidence and guidance for HIVST despite major gaps in HIV diagnosis.

However, through the groundbreaking research from the Unitaid-funded HIV Self-Testing Africa (STAR) initiative led by PSI, we demonstrated that HIVST is not only safe and acceptable but also cost-effective for reaching populations at high risk with limited access to conventional HIV testing. This research played a pivotal role in informing the normative guidelines of the World Health Organization (WHO) and shaping policies at the country level. As a result, more than 108 countries globally now have reported HIVST policies, with an increasing number of countries implementing and scaling up HIVST to complement and partially replace conventional testing services. This became especially significant as nations tried to sustain HIV services amidst the disruptions caused by the COVID-19 pandemic.

By leveraging our expertise, PSI is conducting research to identify specific areas and populations where the adoption of Hepatitis C and COVID-19 self-testing could significantly enhance testing uptake and coverage. This research serves as the foundation for developing targeted strategies and interventions to expand access to self-testing, ensure that individuals have convenient and timely options for testing for these diseases, and are linked to care, treatment and prevention services through differentiated test and treat approaches.

simplifying consumers’ journey to care in Vietnam

By: Hoa Nguyen, Country Director, PSI Vietnam

In late 2022, with funding from the Patrick J. McGovern Foundation, PSI and Babylon partnered to pilot AIOI in Vietnam. By combining Babylon’s AI symptom checker with PSI’s health provider locator tool, this digital health solution analyzes symptoms, recommends the appropriate level of care, and points them to health providers in their local area. The main goal is to support people in low-income communities to make informed decisions about their health and efficiently navigate the healthcare system, while reducing the burden on the healthcare workforce. The free 24/7 service saves people time and subsequent loss of income from taking time off work and from having to pay unnecessary out-of-pocket expenses. Under our global partnership with Meta, PSI launched a digital campaign to put this innovative product in the hands of people in Vietnam. By the end of June 2023 (in the nine months since product launch), 210 thousand people accessed the AIOI platform; 2.4 thousand people created personal accounts on the AIOI website, 4.8 thousand triages to Symptom Checker and linked 2.2 thousand people to health facilities.

Babylon’s AI symptom checker and PSI’s health provider locator tool captures real-time, quality data that supports health systems to plan, monitor and respond to consumer and provider needs. But for this data to be effective and useable, it needs to be available across the health system. Fast Healthcare Interoperability Resources (FHIR) standard provides a common, open standard that enables this data exchange.
PSI’s first consumer-facing implementation of FHIR was launched in September 2022 as part of the Babylon Symptom Checker project in Vietnam, enabling rapid alignment between PSI and Babylon’s FHIR-enabled client records systems. PSI already has several other consumer health FHIR implementations under active development in 2023, including PSI’s collaboration with the Kenya MOH to launch a FHIR-enabled WhatsApp national health line for COVID-19 health information. PSI will also look to adopt and scale health workforce-facing FHIR-enabled tools, such as OpenSRP2, which will be piloted in an SRH-HIV prevention project in eSwatini in partnership with Ona by the end of 2023.

— Martin Dale, Director, Digital Health and Monitoring, PSI

Engaging the private sector for disease surveillance in Myanmar

By: Dr. Zayar Kyaw, Head of Health Security & Innovation, PSI Myanmar

Under a three-year investment from the Indo-Pacific Center for Health Security under Australia’s Department for Foreign Affairs and Trade (DFAT), PSI is enhancing disease outbreak surveillance and public health emergency preparedness and response capacities in Myanmar, Cambodia, Laos, and Vietnam. When PSI conducted a review of existing disease surveillance systems in Myanmar, it identified several gaps: although the Ministry of Health had systems in place for HIV, tuberculosis, malaria and other communicable diseases, they were fragmented, with different reporting formats and reliance on paper-based reporting. In addition, private sector case surveillance data were not routinely captured, yet private clinics and pharmacies are the dominant health service delivery channel in the country. This hindered effective disease prevention and control efforts.

Building on our extensive private sector malaria surveillance work under the BMGF-funded GEMS project in the Greater Mekong Subregion, PSI implemented a case-based disease notification system using social media channels to overcome the limitations of paper-based and custom-built mobile reporting tools. These chatbots, accessible through popular social media platforms like Facebook Messenger and Viber, proved to be user-friendly and required minimal training, maintenance, and troubleshooting. The system was implemented in more than 550 clinics of the Sun Quality Health social franchise network as well as nearly 470 pharmacies. The captured information flows to a DHIS2 database used for real-time monitoring and analysis, enabling rapid detection of potential outbreaks. Local health authorities receive instant automated SMS notifications, enabling them to promptly perform case investigation and outbreak response.

In 2022, private clinics reported 1,440 malaria cases through the social media chatbots, while community mobilizers working with 475 private providers and community-based malaria volunteers reported more than 5,500 cases, leading to the detection of two local malaria outbreaks. Local health authorities were instantly notified, allowing them to take action to contain these surges in malaria transmission. During the same time, pharmacies referred 1,630 presumptive tuberculosis cases for confirmatory testing – a third of which were diagnosed as tuberculosis and enrolled into treatment programs.

Training health workers in Angola

By: Anya Fedorova, Country Representative, PSI Angola

The shortage of skilled health workers is widely acknowledged as a significant barrier to achieving Universal Health Coverage. To address this challenge, PSI supported ministries of health to develop a digital ecosystem that brings together stewardship, learning, and performance management (SLPM). The ecosystem enhances training, data-driven decision-making, and the efficiency of healthcare delivery.

Here’s what it looks like in practice.

In July 2020, PSI Angola, alongside the Angolan digital innovation company Appy People, launched Kassai, an eLearning platform that targets public sector health workers in Angola. Through funding from USAID and the President’s Malaria Initiatve (PMI), Kassai features 16 courses in malaria, family planning, and maternal and child health – with plans to expand learning topic areas through funding from ExxonMobil Foundation and private sector companies. A partnership with UNITEL, the largest telecommunication provider in Angola, provides all public health providers in Angola free internet access to use Kassai.

Kassai’s analytics system to follow learners’ success rate and to adjust the course content to learners’ performance and needs. Kassai analytics are integrated with DHIS2 – the Health Management Information System (HMIS) of Angolan MOH, to be able to link learners’ knowledge and performance with the health outcomes in the health facilities. The analytics track learners’ performance by course and gives visibility by health provider, health facility, municipality, and province. Each course has pre-and post-evaluation tests to track progress of learning, too.

By the end of 2022, there were 6,600 unique users on the Kassai platform and 31,000 course enrollments. PSI Angola’s partnership with UNITEL, the largest telecommunication provider in Angola, allows for free internet access to learn on the Kassai for all public health providers in Angola. Building on its success for malaria training, Kassai now also provides courses in family planning, COVID-19, and maternal and child health. This reduces training silos and provides cross-cutting benefits beyond a single disease.

Implementing the SLPM digital ecosystem brings numerous benefits to health systems. It allows for more strategic and efficient workforce training and performance management, enabling ministries of health to track changes in health workers’ knowledge, quality of care, service utilization, and health outcomes in real time. The ecosystem also supports better stewardship of mixed health systems by facilitating engagement with the private sector, aligning training programs and standards of care, and integrating private sector data into national HMIS. Furthermore, it enables the integration of community health workers into the broader health system, maximizing their impact and contribution to improving health outcomes and strengthening primary healthcare.

Evaluating Program Effectiveness: Key Concepts and How to Use Coarsened Exact Matching

Stay Updated

Your Investment, Real Impact

International Tax-Deductible Gifts

Qualified Charitable Distributions

ACCELERATING UPTAKE OF HEALTH INSURANCE

Explore our resources

Building Resilient, Consumer-Powered Health Systems

CAN DIGITAL LOCATOR TOOLS IMPROVE ACCESS TO HIGH-QUALITY HEALTH SERVICES AND PRODUCTS IN LOW-RESOURCE SETTINGS?

Explore our resources

listen to the podcast

Better data for stronger health systems

Explore our resources

View our short interviews

Scaling Digital Solutions for Disease Surveillance

Misinformation and Vaccine Hesitancy

The Frontline of Epidemic Preparedness and Response

HOW COULD PRIVATE SECTOR PHARMACIES AND DRUG SHOPS ADVANCE PROGRESS TOWARDS UNIVERSAL HEALTH COVERAGE?

Explore our resources

listen to the podcast

The Consumer as CEO

Digitalizing contraceptive counseling to reach rural women and girls in Ethiopia

Building community health worker capacity to deliver malaria care

Taking a market-based approach to scale sanitation in Ethiopia

Promoting self-managed care like Self-testing and Self-Sampling

Using peer coaches to counter HIV stigma in South Africa

simplifying consumers’ journey to care in Vietnam

Engaging the private sector for disease surveillance in Myanmar

Training health workers in Angola

WHISTLEBLOWER AND ANTI-RETALIATION

Global Code of Business Conduct And Ethics

The Future of Work

Meaningful Youth Engagement

Zero Tolerance for Modern-Day Slavery and Human Trafficking

UNITED NATIONS GLOBAL COMPACT

Environmental Sustainability

Affirmative Action and Equal Employment Opportunity

Zero Tolerance for Discrimination and Harassment

Diversity and Inclusion

Gender Equality

People & Communities

Scaling Digital Health Innovations

Supporting People to be Active Agents in their Healthcare

Cover

01 #PeoplePowered

02 Breaking Taboos

03 Moving Care Closer to Consumers

04 Innovating on Investments

ICFP Q&A: Let's Talk About Sex

Donor Advised Funds

Employer Match

Stay
Updated

Your Investment,
Real Impact

ICFP Q&A:
Let's Talk About Sex