This post originally appeared on Impact magazine
Impact magazine interviews Ellen Starbird, Director of the Office of Population and Reproductive Health, USAID.
IMPACT: How does USAID assess the effectiveness of its health investments?
ELLEN STARBIRD: USAID assesses the effectiveness of its health interventions by looking at trend data in health indicators that are related to the programmatic interventions that we support. For our family planning and reproductive health programs, contraceptive prevalence, improvements in birth spacing and increasing age at marriage are all measured by surveys, including the Demographic and Health Survey. Changes in these indicators can be related to our investments. USAID uses evaluation findings to inform decisions, improve program effectiveness, be accountable to stakeholders, and support organizational learning. Research tests the effectiveness of possible interventions and is used to identify high-impact practices for our family planning and reproductive health programs. Pilot studies and introduction studies test the effectiveness of interventions in specific contexts or countries. Those interventions that best “fit” a particular context (i.e., level of program development, epidemiological context, resources available, etc.) are selected.
IMPACT: USAID has a long history of using a “logical framework of results” to monitor health programs. Could you describe this framework and how it is used to facilitate decision-making?
ES: The logical framework is an important part of project design, as it identifies and briefly describes the problem the project intends to address and the expected outcomes of the project. The framework includes inputs, outputs, outcomes and impact. USAID uses Project Monitoring Plans to monitor at each step in this process. These plans examine answers to questions such as: Are inputs being delivered as planned? Are inputs leading to the anticipated outputs? Are outputs leading to the desired outcomes? If not, is the problem failure to deliver the input, or is the problem that inputs are delivered but for some unanticipated reason are not leading to the expected outcome?
IMPACT: USAID recently conducted a thorough review of its evaluation practices and developed a new policy on evaluation to guide the organization. What does USAID want to learn through implementation of this policy, and what does this mean specifically for health programs?
ES: USAID conducted this review to ensure that effective evaluations were taking place and guiding programmatic decisions. There was a concern that over the last several years fewer evaluations were being done, and the agency wanted evaluations to play a more prominent role in program decision-making. By implementing the new policy, USAID hopes to get a better understanding of the success with which its programs are implemented (process evaluations) and the impact of those programs (impact evaluation). This means that our health programs will put more focus on the implementation and impact of its projects, and that this information will guide future programming decisions. Ultimately, this creates a quality-improvement process, capturing experience to develop increasingly effective programs.
IMPACT: Can you share a recent example of receiving surprising results from work our office has been supporting? How did these results shape the decisions you and your colleagues had to make?
ES: In recent years, results from the DHS, especially those from Africa, showed an unexpected level of interest in and demand for long-acting contraceptive methods. These findings led us to expand our efforts to make these methods more widely available in an acceptable, accessible and affordable ways. Another example is that survey and qualitative research have identified a substantial demand for contraceptive information and services among youth in developing countries. M-Health is providing access to information on methods and source of supply to youth via electronic communication. Information collected on these programs indicated that youth are interested in a wide variety of methods, including natural methods, injectables and longer-acting methods.
IMPACT: What are some challenges you anticipate in generating meaningful data for decision-making post-2015?
ES: As we continue to make progress, what and how we measure will also have to change. In the area of family planning and reproductive health, for example, we’ll need better measurement around costs, as well as better understanding of how to measure choice and rights. The current data collection mechanisms in place will need to be adapted for such advances, or new ones will need to be developed.
