Evaluation: The Process, Importance and How We Are Innovative in Our Model
Canada’s infant mortality rate (IMR; the number of infant deaths per 1,000 live births) and maternal mortality ratio (MMR; the number of maternal deaths per 100,000 live births) are some of the lowest in the world with 4 infant deaths/1,000 live births and 7 maternal deaths/100,000 live births recorded in 2015, respectively (World Bank, 2016). As a result of this, childbirth is not considered to be a risk to a mother’s or an infant’s life in Canada or in most other Western countries. On the other hand, “childbirth” and “a safe procedure” are not synonymous concepts in many underdeveloped countries of the world such as Uganda where delivery-related complications cost the lives of roughly 17 women per day (World Health Organization, 2013). Despite experiencing a slow decline in such statistics over the past few years (World Bank, 2016; Mbonye et al., 2007), Uganda still had some of the highest MMRs and IMRs in the world in 2015 with 343 maternal deaths/100,000 live births and 38 infant deaths/1,000 live births, respectively (World Bank, 2016). Rather than being a risk-free process that symbolizes the beginning of one’s life, in Uganda childbirth is more of a dangerous task that could cost the life of the mother, the child, or both.
With this in mind, an organization that is highly aware of the poor state of such health indicators in Uganda is FullSoul Canada, anon-profit social enterprise dedicated to reducing such alarming statistics within this African country. FullSoul aims to improve and protect the health and safety of delivering mothers, their infants, and the medical staff involved in the birthing process by providing necessary Ugandan hospitals with locally made, safe, and reusable medical kits. With each sterile kit consisting of a needle holder, a dissecting forceps, 3 artery forceps, 3 surgical scissors, and 2 kidney dishes, such medical equipment is believed to contribute towards significantly reducing the burdensome tertiary delays – delays in receiving adequate or timely care once present at a health facility due to a lack of available medical resources, personnel, and/or services – that are responsible for roughly 50% of Uganda’s high MMR (Thaddeus & Maine, 1994; AOGU, 2013).
With each medical kit costing $500 CND to produce, translating into an expense that is less than 5 cents for every baby delivered, each one is theoretically capable of assisting the birth of 2-3 babies per day for about 20 years thereby potentially safely delivering over 15,000 infants in its lifetime. With 15 kits having already been implemented in 3 different Ugandan hospitals, it is hypothesized that this will be able to statistically significantly improve the proportion of mothers and infants surviving childbirth in these locations. It is theorized that these medical kits will provide such hospitals with a larger quantity of safe medical resources allowing for more timely and efficient care to be provided which will lead to a significant reduction in the effects of tertiary delays on maternal mortality and therefore also significantly reduce the MMR in these respective Ugandan hospitals. However, in order to confirm that such medical kits are actually able to achieve these intended aims, their abilities must be measured both quantitatively and qualitatively; one way that this can be done is via a comprehensive program evaluation that systematically analyzes both the processes and outcomes of this initiative. The outcome assessment of such evaluations involves a step-by-step process that investigates how effectively a particular program is achieving its intended outcomes (Centers for Disease Control and Prevention, 2011). Alternatively, the process assessment is meant to specifically evaluate if a particular program is being implemented as planned as well as to deduce how the program is operating with respect to what tasks are being carried out, who is performing them, how these are being performed, and who is being affected as a result of such tasks in order to inform the continuous improvement of the program. Overall, both the process and outcome aspects are critical components of an evaluation because they determine not only if a program works and/or is sufficient for achieving the intended goals but also allow for exploring its strengths, weaknesses, and areas that can be potentially improved in order to better obtain the desired results (CDC, 2011). All of this is to be done while simultaneously and continuously incorporating the needs and desires of key stakeholders throughout the entire process with such findings then being communicated back to these individuals as a means of being accountable for the program’s effects on the projected goals.
With respect to the medical kits, their ability to significantly reduce the MMR will be evaluated in the three Ugandan hospitals where they have already been implemented. The overall objective of the outcome portion of this evaluation will be to determine if the MMRs in these three hospitals after the kits have been implemented are significantly lower than the MMRs in these same respective hospitals before the initiation of this program. Likewise, it will be assessed if the MMRs in these locations have been significantly reduced in comparison to Uganda’s average MMR to determine if this initiative improved the health of the mothers and their infants. Secondary data sources will be used to keep track of such rates at both the hospital and national levels in order to compare the pre- and post-treatment MMRs. For the process evaluation, how the medical kits are being implemented (i.e. if the medical personnel involved in the childbirth process are using the kits as they were instructed with respect to using them at the right time, applying the correct techniques, properly sanitizing the kits after each use, being the only qualified stakeholders to use such kits, etc.) will be assessed using site observations, interviews, focus groups, and case reviews with information being collected from the hospital staff as well as the mothers being cared for in order to evaluate if the program is being delivered according to plan and if the kits are being used as intended in order to achieve optimal effectiveness. This will also be used to detect if the delivering mothers are informed about and receptive to such medical kits by analyzing if more of them are choosing to have their babies in a hospital setting where such materials are available instead of attempting to give birth in private venues such as within their own homes. By evaluating the implementation of these medical kits in 3 distinct hospitals, the execution of this program within these different settings can be compared and contrasted in order to obtain information with respect to how the overall delivery of this program can be improved (i.e. by either having more kits, having more tools in each kit, developing better ways of teaching the medical personnel how to use the kits, etc.). Such a process can underline some potential barriers to the proper delivery of this program that may exist in one hospital and not in another whereas it may also point out if one hospital is doing something differently than the other two which could lead to a more successful delivery of this program.
All in all, implementing such an evaluation and disseminating its findings will inform the Ugandan mothers and hospital staff as well as the FullSoul team, the sponsors of this organization, and any other key stakeholders if the implementation of these medical kits is a sufficiently effective approach for significantly improving the safety of childbirth in such Ugandan hospitals. Similarly, this assessment will reveal to such stakeholders if this program is being carried out as planned with respect to the kits being used as intended and if they are reaching a sufficient number of mothers in need. Both components of the evaluation will be used to infer if this current approach is sufficiently effective and if it is ready to be implemented in more hospitals to achieve such goals or if more work needs to be done in order to develop this program into a more feasible and successful initiative before a significant impact on Uganda’s MMR can be attained.
Mbonye, A.K., Mutabazi, M.G., Asimwe, J.B., Sentumbwe, O., Kabarangira, J., Nanda, G., and V. Orinda. (2007). Declining maternal mortality ratio in Uganda: Priority interventions to achieve Millennium Development Goal. Journal of Gynecology and Obstetrics, 98, 285- 290.
Thaddeus, S. & Maine, D. (1994). Too far to walk: Maternal mortality in context. Social Science & Medicine, 38(8), 1091-1110. doi: 10.1016/0277-9536(94)90226-7
U.S. Department of Health and Human Services Centers for Disease Control and Prevention. (2011). Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Retrieved from: http://www.cdc.gov/eval/guide/CDCEvalManual.pdf
World Bank. (2016). Data: Maternal mortality ratio (modeled estimate, per 100,000 live births. Retrieved from http://data.worldbank.org/indicator/SH.STA.MMRT
World Bank. (2016). Data: Mortality rate, infant (per 1,000 live births). Retrieved from http://data.worldbank.org/indicator/SP.DYN.IMRT.IN
World Health Organization (2013). The Partnership for Maternal, Newborn & Child Health: Millennium Development Goals 4 and 5. Retrieved from http://www.who.int/pmnch/about/about_mdgs/en/index.html.
–Written by FullSoul Canada Volunteer, Alin Herciu