Figures
Abstract
Background
Health communication campaigns – involving mass media and interpersonal communication - have long been utilized by national family planning programs to create awareness about contraceptive methods, to shift social norms related to fertility control, and to promote specific behaviors, such as the use of condoms, injectable methods or permanent sterilization. However, demonstrating the effectiveness of these campaigns is often complicated because the infeasibility of experimental designs generally yields statistically non-equivalent samples of campaign-exposed and unexposed individuals.
Methods
Using data from a panel survey of reproductive age women in Egypt, we estimate the effects of the multimedia health communication campaign “Your Health, Your Wealth” (“Sahatek Sarwetek”) on precursors to contraceptive use (e.g., spousal communication, birth spacing attitudes) and on modern contraceptive use. Difference-in-differences and fixed effects estimators that exploit the panel nature of the data are employed to control for both observed and unobserved heterogeneity in the sample of women who self-report recall of the messages, thereby potentially improving upon methods that make no such controls or that rely solely on cross-sectional data.
Findings
All of the estimators find positive effects of the “Your Health, Your Wealth” campaign on reproductive health outcomes, though the magnitudes of those effects diverge, often considerably. Difference-in-differences estimators find that exposure to the campaign increases the likelihood of spousal discussions by 14.4 percentage points (pp.) (SE = .039, p<0.001) but has no effect on contraceptive use. In contrast, the fixed effects, instrumental variables estimator, controlling for unobserved heterogeneity, finds a large, statistically significant effect on modern contraceptive use (27.4 pp., SE = 0.135, p = 0.043).
Conclusions
The difficulties of evaluating family planning communication programs may be surmountable using panel data and analytic methods that address both observed and unobserved heterogeneity in exposure. Not controlling for such effects may lead to substantial underestimates of the effectiveness of such campaigns.
Citation: Hutchinson PL, Meekers D (2012) Estimating Causal Effects from Family Planning Health Communication Campaigns Using Panel Data: The “Your Health, Your Wealth” Campaign in Egypt. PLoS ONE 7(9): e46138. https://doi.org/10.1371/journal.pone.0046138
Editor: Thomas Eisele, Tulane University School of Public Health and Tropical Medicine, United States of America
Received: July 4, 2012; Accepted: August 28, 2012; Published: September 25, 2012
Copyright: © Hutchinson, Meekers. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was funded through the United States Agency for International Development (USAID) through the Health Communication Partnership and under the terms of Cooperative Agreement No. GHP-A-00-2-00008-00. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Health communication campaigns have long been integral components of national family planning programs. Mass media, counseling and other forms of interpersonal communication have been widely used to inform and create awareness about family planning methods and their availability, to entertain populations and establish influential role models, and to promote specific behaviors, such as the use of condoms, injectable methods or permanent sterilization [1]–[9]. However, evaluations of these programs have frequently been plagued by a number of difficulties, which we address by using data from a panel of reproductive age women in Egypt to estimate the effects of the “Your Health, Your Wealth” national multimedia campaign on contraceptive use and attitudes.
At the heart of the problem for the evaluation of many large-scale health communication interventions is the inability or impracticality of using experimental designs in which individuals are randomized into exposed treatment groups and unexposed control groups. The use of randomization and experimental research designs is the predominant mechanism for inferring causal relationships because they enable a set of control individuals – equivalent in all respects except exposure to an intervention – to represent the counterfactual outcome for treatment individuals had they not received the intervention. Causal impacts are therefore measured as the difference in mean outcomes between treatment and control individuals [10], [11]. But randomization is rarely employed in the evaluation of communication interventions because those interventions often cover entire countries, potentially exposing all targeted individuals, or because localized interventions risk contamination across geographic areas, or because ethical concerns proscribe limiting dissemination of health messages to a subset of potential beneficiaries [11]–[16].
In the absence of randomized control designs, evaluations of health communication programs have frequently adopted alternative methods to generate inferences surrounding causal relationships, generally using non-equivalent comparison groups and statistical methods that seek to achieve equivalence based on observed characteristics of exposed and unexposed individuals. In many cases, comparison groups can be generated because health communication programs – even those that attempt to target all members of a population - are likely to leave some sub-population unexposed to the intervention, as some individuals may be less regular consumers of media than others or may not recall having been exposed. Population surveys, such as the Demographic Health Surveys or more focused communication surveys, can be used to identify individuals who recall being exposed to campaign messages and individuals who do not, while also collecting information on their health behaviors and outcomes. A common choice for a measure of a communication intervention’s effect involves a comparison of average outcomes for those who recall being exposed to intervention messages relative to those who do not. This is the approach taken by numerous evaluations of health communication programs, which use single equation multivariate regression models with cross sectional data to measure the effect of exposure to a health communication program on a family planning outcome of interest, controlling for a limited set of observable characteristics of those individuals [17]–[23].
Such a measure, however, may contain very serious limitations, as the sample of unexposed individuals may be very different than the sample of exposed in ways that may also affect outcomes under study. As noted in other studies [15], [24], [25], exposed individuals likely differ from unexposed individuals in measurable (exogenous) ways, such as levels of education, income, age, or geographic location. But they may also differ in other less easily measured ways. For example, previous researchers have argued that family planning programs are often targeted to specific populations such as those that are hard-to-reach, those with higher fertility or those that have more traditional norms regarding family planning use. Such targeting – to the extent that researchers cannot quantify or control for it – may bias estimates of campaign effectiveness if the targeting is also related to program outcomes [26]–[29]. Alternatively, individuals exposed to program messages may be more media savvy, or they may simply have higher levels of motivation to control their fertility. As a result, they may represent a form of low-hanging fruit - audiences that are more receptive to family planning communication messages, implying that attempts to replicate program successes with alternative populations may be less effective. In short, the sample of individuals who self-report exposure to family planning messages may be very different from the sample of unexposed individuals in ways that may bias estimates of campaign effectiveness.
Other researchers have attempted to overcome the limitations of regression models [25], [30] using matching methods that draw a cohort of non-intervention individuals from some population – either the same population or a similar enough comparison group – who can be “matched” with intervention individuals of similar characteristics. Average outcomes for untreated matched groups serve as the relevant counterfactual to the missing outcomes of exposed program participants. Program effects, as in an experiment, are measured as the difference in average outcomes for these two groups. Nonetheless, while intuitively appealing, matching methods and multivariate regression models share the same conditional independence assumption - that selection into the treatment group is determined by observable characteristics while unobserved characteristics that may affect family planning outcomes are distributed randomly across exposed and unexposed individuals [31], [32]. Hence, they are likely to face similar biases.
The fundamental difficulty, in short, is that evaluations of health communication programs often rely upon measures of program exposure that are at least in part determined by the actions, choices and characteristics of the potential beneficiaries, which may therefore confound estimates of the health communication intervention’s effectiveness. In such cases, naïve estimators that assume exogenous exposure – or exogeneity conditional on a limited set of control variables - may be severely biased.
To address non-random program exposure, researchers have several alternatives, which have seldom been used in the evaluation of the family planning communication literature but which we explore here. These involve, for example, the use of instrumental variables approaches to create exogenous variation in exposure by identifying a set of variables that affect exposure to the intervention but not the outcome itself, thereby purging estimates of the program effect from the confounding effects of the determinants of exposure [15], [33], [34]. In previous analyses, such instruments have included measures of frequency of exposure to different media [24], [30]. Alternatively, fixed effects models with a panel of exposed and unexposed individuals allow researchers to collect information on potential program beneficiaries at a baseline, again at a post-intervention time point, and then to compare changes in outcomes for those individuals who were exposed to the intervention relative to those who were not exposed. Under the assumption that unobserved heterogeneity affecting program exposure is time invariant, evaluators can better attribute changes in family planning outcomes to the program rather than to confounding by observed or unobserved characteristics of respondents [35], [36].
In this paper, we make use of panel data to estimate the effects of the “Your Health, Your Wealth” (Sahatek, Sarwetek) health communication program in Egypt while controlling for unobserved heterogeneity between exposed and unexposed respondents. The “Your Health, Your Wealth” campaign was a component of the Communication for Healthy Living (CHL) project in Egypt, reflecting the cooperation of the Egyptian Ministry of Health and the the United States Agency for International Development (USAID). CHL in turn was one part of the Health Communication Partnership (HCP), a global health communication initiative funded by USAID. The CHL program supported activities at both the national and local level in the areas of family planning and reproductive health, maternal and child health, infectious diseases control, healthy lifestyle, household preventive health, and health maintenance practices. Messages have been disseminated via a national integrated health communication campaign using television, radio, press advertising and public affairs programming. The “Your Health, Your Wealth” campaign involved national multimedia and community-based interventions aimed at encouraging families to engage in healthy behaviors at different points in the life stage. A key component of the overall communication strategy was the “Mabrouk!” (Congratulations!) Initiative, targeted towards newlyweds as a strategic entry point for encouraging behaviors that promote healthy families, including important events such as pregnancy, labor and delivery, postpartum infant care, family planning, and overall family health [37], [38]. Specific family planning messages include the benefits of birth spacing and the need for post-partum resumption or initiation of family planning to avoid early pregnancy [39], [40].
We focus here on estimating the effects of the “Your Health, Your Wealth” (Sahatek, Sarwetek) health communication campaign on several family planning outcomes, including current contraceptive use, discussions with a spouse about family planning and the use of family planning for birth spacing, and agreement with statements about the benefits of family planning for birth spacing. Several estimation methods are used to attempt to develop and to compare estimates of the causal effect of exposure to the “Your Health, Your Wealth” national multimedia health communication campaign: (1) a single-equation cross sectional estimator (using endline data), (2) matching on the propensity score of exposure, (3) a difference-in-differences estimator, (4) a fixed effects estimator, (5) a fixed effects, instrumental variables estimator, and (6) a bivariate probit model that models simultaneously both the outcome and exposure equations and allows for correlation in unobservables across the two equations. We find that all estimators – including those relying solely on the post-only cross section and those assuming exogenous campaign exposure – yield positive effects of campaign exposure, though the effects diverge in magnitude and statistical significance. Statistical tests indicate that endogenous exposure may in fact be problematic, in which case the naïve estimators assuming exogeneity may underestimate actual campaign effects.
Data
Data Source
We use data from two waves of the Menya Village Health Surveys conducted in seven villages of Menya Governorate in Egypt in 2004 and 2005. All villages were exposed to the national “Your Health, Your Wealth” multimedia health communication campaign, which was broad on national television. However, five of these villages received more intensive community-based interventions from CHL, while two villages were used as comparison villages. The intensive community-based interventions were implemented through community development associations (CDAs), and included activities such as newlywed visits, pregnancy classes, safe delivery referrals, and postpartum home visits [Hess, Meekers, Storey 2012]. The home visits provided an opportunity to reinforce national media messages through interpersonal communication, and to encourage women to start using family planning within forty days after the delivery and to space their children three years apart. The surveys were funded by the United States Agency for International Development (USAID) as part of the external evaluation of the impact of the Health Communication Partnership (HCP). This evaluation – part of a multi-country study – was conducted by Tulane University’s Department for International Health and Development (Tulane/IHD), School of Public Health and Tropical Medicine. Collection of data was undertaken by El-Zanaty and Associates.
Our analysis focuses on ever-married women aged 15–49 years. For the 2005 sample, women who were interviewed in 2004 and completed 50 years by the date of the interview in 2005 were excluded. Only usual household residents were eligible for interview.
Two types of questionnaires were used in the data collection: (1) a household questionnaire which identified eligible respondents and collected information on household socioeconomic characteristics and living conditions and (2) eligible respondent questionnaires which focused on health knowledge, attitudes and behaviors, as well as detailed questions about exposure to different health communication messages and campaigns.
A multi-stage cluster sample design was used to identify respondents. At the first stage, five intervention villages receiving more intensive community support (Zohra, Saft El khamar El sharkia, Nazlet Hussein Ali, Monshaat El Maghalka, and Koloba) and two control villages (Toukh El khail and Ebshedat) were selected. At the second stage, each village was divided into segments of approximately 1000 households. Each village had 10 segments, except for Koloba (which had 11 segments) and Ebshedat (which had 13). One segment was then selected at random, and a household listing was conducted by El-Zanaty and Associates. At the third stage, approximately 35 households were systematically sampled at random from the household listing. The sampling interval was determined by dividing the total number of households in each segment by 35 [39], [40].
Ethics statement.
Verbal consent was obtained from all survey respondents. Verbal – rather than written - consent was obtained because of the low literacy levels of many of the respondents. Prior to commencing an interview, survey interviewers read a consent statement to potential respondents. The consent statement explained the reasons for the survey and the rights of respondents, informed respondents of the estimated length of time for an interview, described procedures to maintain confidentiality, and listed contact information for survey administrators if respondents had subsequent questions or concerns. Interviewers then documented with a check mark and a signature on the questionnaire to indicate whether or not consent had been given by a respondent. This consent process, as well as the study design, research protocol, and questionnaires were reviewed and approved by the Tulane University Biomedical Institutional Review Board (IRB) prior to the implementation of fieldwork in order to ensure that they met the qualifications and restrictions of the Tulane University Human Research Subject Protection Program.
Fieldwork for the 2004 MVHS was conducted over three weeks beginning in late July and ending in mid-August 2004. The 2005 MVHS was conducted over a two-week period in August and September 2005. For quality control, 5 percent of the sample was selected for re-interview using shorter versions of the original questionnaires. The re-interviews occurred following the main fieldwork and involved special teams that did not involve the original interviewers. During the re-interviews, teams also attempted to visit households or individuals whose interviews were not completed during the initial village visits.
Attrition across the two waves was negligible. In 2004, 2,316 households were selected for interview, and 2,298 households were interviewed. In 2005, 205 of the original households were no longer eligible, while 2,093 of the 2004 households were re-interviewed and 126 new households were added to the sample. A total of 2,240 ever-married women were interviewed in 2004 (response rate of 99.7 percent). By 2005, 2,073 of these women were still eligible, while an additional 86 youth were married and became eligible for interview and 201 women had a new husband and were also eligible. Of these women, a total of 2,284 were successfully interviewed (response rate of 96.8 percent).
Methodology
In this analysis, we focus on several outcomes related to family planning and use of family planning in Egypt. Specifically, we examine whether or not a woman is currently using modern family planning (i.e., oral contraceptives, IUD, injectables, implants, or diaphragm), whether or not she had a discussion with her spouse about family planning in the past 12 months, whether or not she has discussed the use of family planning for birth spacing, and whether or not she agrees with statements about the benefits of family planning for birth spacing. These outcomes are chosen because they reflect specific message themes from the “Your Health, Your Wealth” campaign.
As our measure of exposure to the “Your Health, Your Wealth” campaign, we use the variable from the 2005 wave of the survey indicating whether or not the respondent reported having seen “Your Health, Your Wealth” messages in the last 12 months and specifically mentioned that those messages pertained to either “birth spacing” or “family planning use in the 40 days following birth.” In our sample of 2,088 women in 2005, 378 (18.1%) recalled having seen either messages; 321 (15.4%) recalled the messages related to postpartum family planning use, and 151 (7.2%) recalled the messages related to birth spacing.
Estimation Assuming Exogenous Exposure
As a starting point, we focus on estimation methods that assume that exposure to the “Your Health, Your Wealth” campaign is exogenous, that is, once we have controlled for observed characteristics of respondents, there are no other factors that simultaneously determine both exposure to the campaign and our family planning outcomes. We estimate equations for each of the four family planning outcomes using a combined exposed-unexposed group sample and include variables representing self-reported exposure to the “Your Health, Your Wealth” campaign and a set of exogenous controls. We estimate models separately for each of the two cross-sectional waves of the surveys. Using the first wave of data allows us to test for initial differences in the samples of exposed and unexposed respondents prior to the implementation of the communication campaign, while estimation with the second wave provides us with an estimate of the association between exposure and family planning outcomes after the campaign had been in operation for 18 months.(1)
In this specification, Yit is the outcome of interest (e.g., current use of modern family planning) for i = 1…N individuals in the sample at time t = 0 (2004) or t = 1 (2005). The Xit represent a vector of exogenous control variables (e.g., wealth, education, village characteristics), Dit represents self-reported exposure to the “Your Health, Your Wealth” program, and uit is a measure of unobservables associated with the outcome Yit and assumed in this specification to be uncorrelated with the program exposure variable Dit. The parameter β2 represents a measure of the cross-sectional association of program exposure with the outcome Yit, controlling for the exogenous control variables Xit.
Each of the outcome variables is binary, i.e., Yit = 1 if the individual engages in the behavior (e.g. is currently using modern family planning) and Yit = 0 otherwise. We make a similar assumption about exposure to the health communication program: a woman either recalls hearing or seeing the “Your Health, Your Wealth” messages (Dit = 1) or not (Dit = 0). We model the response probability Pr[Yit = 1|Xit, Dit] as a logit model.
Key explanatory variables in these multivariate models include a categorical variable for a woman’s level of education (none, primary or secondary/university), a categorical variable for a woman’s age (in 5- or 10-year increments), a categorical variable of household wealth constructed from a principal components analysis of household ownership of a set of consumer durables, a continuous variable for the number of children ever born to a woman, the presence in the village of a recognizable leader, and residence in a program village (relative to a non-program village).
This approach has two key limitations. First, it assumes that exposure to “Your Health, Your Wealth” messages is exogenous once socio-demographic control variables are included in the model. However, as previously mentioned, exposed respondents may differ from unexposed respondents in many important unmeasured ways – differing fertility experiences, differing norms related to family planning, or differing motivations to control their fertility (e.g., gender preferences for children). Such unmeasured factors can influence both campaign exposure and family planning, leading to estimates of β2 that are seriously under- or over-estimated. Further, the single equation cross-sectional model is unable to assess how exposure to the campaign is associated with changes across time in the family planning indicators.
Differences-in-differences (DID) estimator.
In contrast to the previous model, the difference-in-differences estimator uses the full panel sample of baseline and endline observations to estimate the effect of the campaign for intervention “exposed” individuals relative to comparison “unexposed” individuals. The measure of the causal effect is represented by the coefficient on the interaction term β4 in a regression of the family planning outcome on a year dummy variable Tt, self-reported exposure to the “Your Health, Your Wealth” program Dit, their interaction , and a set of controls.
(2)
As with the cross sectional model, we again model the response probability Pr[Yit = 1|Xit, Dit, Tt] as a logit.
While the DID estimator has the advantage that it addresses how exposure to the program may be associated with changes in family planning outcomes, it is unable to confront possible correlation between unobserved factors (e.g., motivations, family planning experience) and family planning outcomes that may bias estimates of the campaign’s effects.
Propensity score matching (PSM).
A method with a similar motivation involves matching individuals with similar likelihoods of exposure to the campaign at baseline and then comparing average outcomes at endline. Matching methods reduce bias from non-random treatment assignment by balancing on observed covariates [31], [41]–[43]. A central assumption of matching methods is that treatment assignment is strongly ignorable, i.e., that assignment and outcomes are independent conditional upon measured characteristics of survey respondents [44], an assumption shared by multivariate regression models [32].
The propensity score is estimated as a function of a set of pre-determined (baseline) characteristics of respondents hypothesized to be independent of the ultimate outcomes: age, education, wealth, presence of a recognizable village leader, being in a program village. In this analysis, the propensity score is constructed and tests of covariate balance are performed using the STATA 12.0 command pscore [41]. We estimate the average treatment on the treated (ATT) effect using kernel matching with the STATA 12.0 command psmatch2 [45]. The kernel matching procedure uses a weighted average of all controls, where the weights are inversely proportional to the distance between the propensity score of treated individuals and control individuals [41]. We restrict our matching to the area of common support between exposed and unexposed respondents. Overall, all 378 exposed respondents were matched.
Estimation Controlling for Endogenous Exposure
In addition to the above methods assuming exogenous program exposure, we also use the following estimators that assume and test for endogenous exposure:
Fixed effects logit.
As with the DID and PSM estimators, we make use of the panel nature of the data to estimate a fixed effects logit model using conditional maximum likelihood. In other words, information at the 2005 endline for our sample of 2,088 women is linked with their 2004 baseline information. In the fixed effects model, the error term in equation (1) can be expanded to include both a time-invariant individual-specific effect αi and time-varying component uit. A time period specific effect Tt is also included as in the DID model.(3)
In the fixed effects model, the correlation between the exposure variable Dit and the error term is assumed to be with the time invariant component αi. In other words, exposure may be higher among people with certain types of fertility experiences, motivations to use family planning, or norms surrounding the use of family planning but such relationships are constant across time. In this framework, parameters are estimated using the conditional likelihood function constructed from observations in which Yit varies from time period 1 to 2. For these observations, the individual effects αi can be shown to drop out of the probability density in the likelihood function, thereby removing possible confounding by time invariant motivations, experiences and norms and yielding a consistent estimate of the conditional maximum likelihood estimator of β2. An important drawback of this estimation method, however, is that the effects of variables that do not vary across time cannot be determined [46], [47]. Further, bias in parameter estimates may still result if correlation between exposure and the unobservables rests with the time varying error term .
Fixed effects, instrumental variables panel data model.
This model extends the previous fixed effects model to include the possibility of endogenous regressors. In the previous model, the correlation between exposure and the unobservables is assumed to be with the time invariant unobservable term αi.representing attitudes, motivations, or experiences that are fixed across time. This is a potentially problematic assumption as fertility experiences may alter a woman’s motivations or attitudes towards family planning in ways that may also be associated with exposure to the campaign. As a consequence, we employ an instrumental variable approach, which involves estimating an equation for program exposure:(4)where the Xit represent an overlapping vector of exogenous control variables that also affect the outcome Yit, the Zit are a non-overlapping vector of variables that are correlated with Dit but not Yit, and vit is a measure of unobservables associated with Dit but also potentially correlated with uit in equation (1). Because of this correlation, estimation of the parameter β2 in equation (3) – as well as the other parameters of the model - may be biased by some measure of the degree of correlation in the unobservables affecting both Dit and Yit. Estimation of equation (4) is then conducted simultaneously with the family planning outcome equation (3) via the method of Limited Information Maximum Likelihood (LIML) [48]. In theory, instrumental variables estimators can provide consistent estimates of program effects, assuming that valid exclusion restrictions for the instrumental variables can be identified. In reality, however, identifying suitable instruments is not without some degree of difficulty [15], [33], [34].
To achieve identification in this model, we include variables hypothesized to be statistically associated with exposure to “Your Health, Your Wealth” (Dit) but not with the outcomes under study (Yit). These include variables for whether or not a woman lived in a designated program village, whether or not a woman was willing to participate in community-organized activities to improve family health, and whether or not a woman had ever heard of community gatherings to discuss health and family planning. These variables were chosen as proxies for informal communication about family planning. Estimation involves use of the full panel of treatment and control individuals [44], [49] and is performed in Stata 12.0 using the xtivreg2 estimator. Model identification was determined by constructing the Kleibergen-Paap rk Wald F statistic [48]. These values are then compared with the Stock-Yogo critical values [50].
Bivariate probit.
As a final alternative, we estimate both equations (1) and (4) simultaneously using a bivariate probit model for two binary outcomes. The model is motivated using a continuous underlying latent variable specification for both exposure to “Your Health, Your Wealth” and the family planning outcome, whose discrete realizations are given as above by Dit and Yit respectively. In equations (1) and (4), the disturbance terms ui and vi are assumed to be joint normal with means of 0 and variances of 1. The likelihood function is constructed as the product of the four mutually exclusive outcomes – (Yit = 1, Dit = 1), (Yit = 1, Dit = 0), (Yit = 0, Dit = 1), and (Yit = 0, Dit = 0). Importantly, this specification allows for correlation ρ between the unobservables in the two equations, thereby controlling for unobserved heterogeneity in the samples of self-reported exposed and unexposed individuals. Standard statistical software provide an estimate of ρ as part of standard regression output. We estimate the bivariate probit model using the Stata 12.0 command cmp developed by Roodman [51] for conditional recursive mixed-process estimators. We use the Likelihood Ratio test proposed by Buis [52] to test for the exogeneity of exposure.
Results
Descriptive Results
Characteristics of the samples of exposed and unexposed women for 2005 are shown in Table 1. On average, women who reported having seen “Your Health, Your Wealth” were approximately 3 years younger (29.76 years versus 32.80 years, p<0.001), had fewer children (3.40 versus 4.09, p<0.001), were more likely to report that there was a leader in their community (24.3% versus 16.5%, p<0.001), and were less likely to live in a treatment village (62.7% versus 70.9%, p = 0.002). Exposed women were more educated; 29.9% had a secondary or higher level of education versus 19.0% of unexposed women. They were also wealthier on average; 25.7% of exposed women were in the highest wealth quintile versus 20.3% of unexposed women.
Table 2 presents outcomes for 2004 and 2005 for those who recalled exposure to the “Your Health, Your Wealth” messages (in 2005) as well as for those who did not. At the baseline, few statistically significant differences in family planning outcomes were observed. For example, contraceptive use was nearly identical - 39.2 percent of exposed ever married women versus 40.3 percent of unexposed women. The only statistically significant difference was for the variable “Discussed birth spacing in the last 6 months” –36.8 percent for exposed women versus 31.1 percent for unexposed women (p = .031).
At the endline, statistically significant differences were observed for three of the four outcomes. The lone exception was for modern contraceptive use –47.6 percent for the exposed relative to 44.2 percent for the unexposed (p = .228). Nonetheless, the changes across time for the exposed relative to the unexposed – equivalent to the difference-in-differences model without controls - were statistically significant for each of the outcomes. For example, the difference-in-differences estimate of the program effect on modern contraceptive use was 4.5 percentage points (p = 0.148), while the estimates for “Discussed birth spacing in the last 6 months” and “Agrees that spacing improves child health” were 9.3 (p = 0.010) and 8.3 (p = 0.049) percentage points respectively. The largest effect was for “Discussed FP with partner in the last 6 months” which showed a 15 percentage point difference (p<0.001) between treatment and comparison, due entirely to a significant decrease in the prevalence of discussion for the comparison group.
Estimation Results
Table 3 summarizes the marginal effects for each of the estimation methods. Full regression results for each outcome and estimation method are shown in Tables 4, 5, 6, 7, 8. As noted above, all estimators – with the exception of the 2004 cross sectional estimator - yielded positive effects of campaign exposure, though the effects diverged in magnitude and statistical significance. We show via several statistical tests that endogenous exposure may in fact be problematic, in which case the naïve estimators assuming exogeneity may underestimate actual campaign effects.
As a starting point, we ran a single-equation logit estimation of each family planning outcome on the exposure variable and a set of control variables using the pre-intervention 2004 sample only (Table 3). For none of the outcomes was the program exposure variable statistically significant, providing a general indication that – conditional on the controls – there was little baseline evidence of unobserved heterogeneity.
Two further tests for endogenous exposure were also conducted. First, in the bivariate probit estimations (Table 3 and Table 7), an exogeneity test of ρ = 0, representing the correlation in the unobservables across the outcome and exposure equations, was conducted. We rejected the null of exogeneity for three out of four outcomes, the lone exception being for the outcome “agree that spacing is healthy.” In contrast, we failed to reject the null of exogeneity for any of the outcomes in the fixed effects, instrumental variables estimations, although in two cases p<0.10.
The measures of campaign effects derived by both the 2005 cross sectional estimates (Tables 3 and 4) and propensity score matching (PSM) (Table 3) were roughly similar. For example, the marginal effect of exposure to the “Your Health, Your Wealth” on modern contraceptive use was 3.9 percentage points by the 2005 cross sectional estimator, as compared with 4.0 percentage points for PSM, though in neither case were the results statistically significant. For the outcome “discuss birth spacing with spouse,” PSM and the 2005 cross sectional estimate were also similar –9.4 and 10.8 percentage points respectively.
The difference-in-differences models with control variables showed the effects of changes in family planning outcomes from 2004 to 2005 for exposed respondents relative to unexposed respondents. For all outcomes, the estimates were less than in the difference-in-differences models without controls. In two cases – discuss family planning with spouse and discuss birth spacing with spouse – exposure to the “Your Health, Your Wealth” messaging yielded positive marginal effects of 0.144 (p<0.001) and 0.086 (p = .021) respectively. For a third outcome (“Agree that spacing is healthy”), results were suggestive of a positive effect (marginal effect = 0.076; p = 0.068). For two of the four outcomes, the estimates were smaller than those estimated by PSM and the 2005 control function estimator.
The fixed effects logit estimator – incorporating individual-level fixed effects and using only time-varying characteristics of individuals - provided estimates of exposure that did not widely diverge from the simpler models. Again, exposure to the “Your Health, Your Wealth” campaign was shown to yield a 10.6 percentage point increase (p = 0.005) in the probability of discussing family planning with a spouse, lower than the 15.4 percentage point difference for PSM and 14.4 percentage point difference for the DID with controls. For the outcome “discussed spacing with spouse,” the marginal effect from the fixed effects model was 14.9 percentage points (p = 0.001), larger than the estimates of 9.4 percentage points from the PSM model and 8.6 percentage points from the DID model.
With the exception of the fixed effects logit models, the other methods controlling for possibly endogenous exposure tended to have considerably larger estimates of programmatic effects, providing support for the possibility that those most likely to be exposed to the “Your Health, Your Wealth” messages were those less likely to be influenced by the program. After controlling for both measured and unmeasured factors affecting non-random exposure, exposure to “Your Health, Your Wealth” was associated with an increase in modern contraceptive use by 23.1 percentage points (p = 0.002) in the bivariate probit model and by 27.4 percentage points (p = 0.043) in the fixed effects IV model. A similar pattern was evident for the outcome “discuss FP with one’s spouse” –42.5 percentage points (p<0.001) for the bivariate probit model and 61.4 percentage points (p = 0.002) for the fixed effects, IV estimator. Each of these estimates is many times larger than for the simpler models assuming exogenous exposure and even larger than the fixed effects logit estimator. The remaining two outcomes – “discussed birth spacing with spouse” and “agree that spacing is healthy for the baby” – were not statistically significantly related to exposure in the fixed effects IV models but they were for the bivariate probit models. In each case, the effects were several times larger than in the simpler models.
The considerable discrepancies between the simpler estimation methods and those addressing both the panel nature of the data and the possibility of endogenous exposure warrant careful examination. As noted by numerous researchers [49], [53], [54], instrumental variables estimators can yield widely misleading results if instruments are not carefully chosen. In order to be acceptable, these instrumental variables must be strongly associated with exposure but only minimally associated with our family planning outcomes (except indirectly through exposure). In this case, we have used several key variables that are hypothesized to affect exposure but not our family planning outcomes. These include self-reported watching of television every day, attendance at community gatherings, an index of other family planning campaigns that the respondent recalls, and a measure of local community social capital. By all appearances, these instrumental variables appear to meet the technical criteria for acceptability as instruments. The Kleibergen-Paap rk LM statistic of 33.666 (Table 8) allows us to reject the null hypothesis that our model is under-identified (p<0.001) [48]. We further reject the null hypothesis that we have weak instruments, as the estimated Kleibergen-Paap rk Wald F statistic is 16.668– slightly less than the Stock-Yogo critical value of 16.85 for 5% maximal IV relative bias but better than the critical value of 10.27 for 10% maximal IV relative bias [50]. As a result, there is some reason for confidence in the validity of our chosen instruments.
On the other hand, the fixed effects estimators rely on variation across time within individuals. In our data, only 378 out of 2,088 women reported being exposed to the “Your Health, Your Wealth” campaign over the time period. Further, within individual variation in many of the covariates appears minimal. Variables such as educational attainment could be included only by interacting with the time dummy. In light of these possible limitations, these results require some degree of caution.
Discussion
This paper assesses the effects of exposure to a family planning health communication program – the “Your Health, Your Wealth” national multimedia campaign in Egypt – on a set of family planning outcomes, including current use of modern contraception, measures of interpersonal communication regarding family planning, and attitudes towards birth spacing. The aim of the paper is in large part methodological – to control appropriately for non-random (self-reported) exposure to the program in order to obtain more accurate measures of the program’s effects.
We make use of an atypically robust set of data – panel data with data collection occurring pre- and post-campaign and involving very low levels of attrition from the sample. The advantage of this data is that they allow for the use of estimation strategies that are not generally permitted by pooled cross-sectional data, a key limitation of many previous analyses of family planning communication efforts. Because the interventions are individualized – only through self-reported recall can researchers identify who is exposed and then trace that back to their pre-intervention outcomes and characteristics – cross-sectional models cannot identify changes across time in the treatment “exposed” group relative to the comparison “unexposed” group. Cross-sectional methods assuming exogenous exposure therefore assume that treatment and comparison individuals are statistically equivalent at baseline conditional on a limited set of control variables. Absent panel data, this assumption cannot be tested for time varying variables. Further, cross-sectional methods that attempt to control for endogenous exposure – instrumental variables methods or simultaneous equations models assuming a specific parametric distribution for the relationship between outcomes and exposure – must confront difficult issues of model identification or assume that identification is attained through assumptions about the parametric distribution.
By using panel data, in contrast, we can examine changes across time among a set of individuals who recall having been exposed to the campaign by the endline relative to individuals who do not recall such exposure. This allows for both difference-in-differences estimation and fixed effects estimation that can address changes across time or difference out unobserved heterogeneity affecting exposure and family planning outcomes. It also allows for the use of matching methods in which matching is determined by baseline characteristics of respondents rather than concurrently measured characteristics which may be more susceptible to underlying, unobserved heterogeneity.
The results, in this case, provide evidence that the “Your Health, Your Wealth” succeeded during the short time period of this study to achieve change in family planning outcomes. The models that made use of the panel nature of the data set – difference-in-differences, propensity score matching and fixed effects logit - provided similar results in terms of direction and levels of statistical significance but the magnitudes of effects often diverged widely. However, we find that estimates of program effects based on the perhaps naïve assumption that exposure is largely random once a limited set of observed covariates are controlled for may substantially underestimate program effects. In fact, we find that the magnitude of the underestimates could be on the order of three- to five-fold. For example, for two common methods assuming exogenous exposure – difference-in-differences models and propensity score matching – exposure to the “Your Health, Your Wealth” campaign was associated with an increase in modern contraceptive use of 2.9 and 4.0 percentage points respectively. In contrast, for the methods controlling for endogenous exposure – the bivariate probit and fixed effects IV estimator – the effect sizes were 23.1 and 27.4 percentage points respectively.
This is an important finding. If, for example, we applied the marginal effects from exposure to the “Your Health, Your Wealth” to a hypothetical population of 100,000 women exposed to the program, a doubling of the effect of the program – as was observed for the fixed effects IV model (ME = 0.270) relative to the difference-in-differences model (ME = 0.076) – would increase the number of women who agreed with the statement that “birth spacing is healthy” by approximately (27,000−7,600 = ) 19,400 women, an important programmatic result. Simpler methods such as the single equation cross-sectional estimator, which attribute to the program an increase in contraceptive use of 3.9 percentage points relative to the fixed effects IV estimate of only 27.4 percentage points, might similarly understate program effects.
In short, while the results from the different estimation methods are similar in direction and levels of statistical significance, the overall effects when applied at the population level can substantially alter conclusions about program success. Analysts and program managers who increasingly rely on estimates of program effects – particularly in estimates of cost-effectiveness - need to be cognizant of the limitations of their methods, particularly those based on cross sectional data.
Acknowledgments
The Menya Village Health Surveys were implemented by El-Zanaty & Associates. Special thanks go to Fatma El-Zanaty, Ron Hess, and Doug Storey.
References
- 1.
Bertrand JT, Kincaid DL (1996) Evaluating Information-Education-Communication (IEC) Programs for Family Planning and Reproductive Health: Final Report of the IEC Working Group. Chapel Hill, North Carolina. WG-IEC-03 WG-IEC-03.
- 2. Jato MN, Simbakalia C, Tarasevich JM, Awasum DN, Kihinga CNB, et al. (1999) The impact of multimedia family planning promotion on the contraceptive behavior of women in Tanzania. International Family Planning Perspectives 25: 60–67.
- 3.
Kincaid DL, Yun SH, Piotrow PT, Yaser Y (1993) Turkey’s Mass Media Family Planning Campaign. In: Backer TE, Rogers EM, editors. Organizational Aspects of Health Communications Campaigns: Sage Publications.
- 4.
Piotrow PT, Kincaid DL, Rimon II J, Rinehart W (1997) Health Communication: Lessons from Family Planning and Reproductive Health. Westport, CT: Praeger.
- 5. Babalola S, Vonrasek C (2005) Communication, ideation and contraceptive use in Burkina Faso: an application of the propensity score matching method. J Fam Plann Reprod Health Care 31: 207–212.
- 6.
Rogers EM (1973) Communication Strategies for Family Planning. New York: Free Press.
- 7.
Rogers EM (2003) Diffusion of Innovations. New York: Free Press.
- 8. Montgomery MR, Casterline JB (1993) The diffusion of fertility control in Taiwan: evidence from pooled cross-section time-series models. Popul Stud (Camb) 47: 457–479.
- 9. Agha S, Beaudoin CE (2012) Assessing a thematic condom advertising campaign on condom use in urban Pakistan. Journal of Health Communication 17: 601–623.
- 10. Holland PW (1986) Statistics and Causal Inference. Journal of the American Statistical Association 81: 945–960.
- 11.
Bertrand JT, Babalola S, Skinner J (2012) The Impact of Health Communcation Programs. In: Waisbord S, Obregon R, editors. Handbook of Global Health Communication: JWS Wiley.
- 12. Noar SM, Palmgreen P, Chabot M, Dobransky N, Zimmerman RS (2009) A 10-Year Systematic Review of HIV/AIDS Mass Communication Campaigns: Have We Made Progress? Journal of Health Communication 14: 15–42.
- 13. Noar MD (2009) The evolution and the natural selection process in the stenting of malignant bile duct obstruction: size does matter! Gastrointestinal Endoscopy. 70: 454–456.
- 14. Guilkey DK, Hutchinson PL (2011) Overcoming Methodological Challenges in Evaluating Health Communication Campaigns: Evidence from Rural Bangladesh. Studies in Family Planning 42: 93–106.
- 15. Guilkey DK, Hutchinson P, Lance P (2006) Cost-effectiveness analysis for health communication programs. Journal of Health Communication 11: 47–67.
- 16. Mwaikambo L, Speizer IS, Schurmann A, Morgan G, Fikree F (2011) What works in family planning interventions: a systematic review. Stud Fam Plann 42: 67–82.
- 17. Agha S, Van Rossem R (2002) Impact of mass media campaigns on intentions to use the female condom in Tanzania. International Family Planning Perspectives 28: 151–158.
- 18.
Agha S (2002) A quasi-experimental study to assess the impact of four adolescent sexual health interventions in sub-Saharan Africa. International Family Planning Perspectives 28: 67−+.
- 19. Van Rossem R, Meekers D (2000) An evaluation of the effectiveness of targeted social marketing to promote adolescent and young adult reproductive health in Cameroon. Aids Education and Prevention 12: 383–404.
- 20. Gupta N, Katende C, Bessinger R (2003) Associations of mass media exposure with family planning attitudes and practices in Uganda. Studies in Family Planning 34: 19–31.
- 21. Gupta N, Katende C, Bessinger R (2004) An evaluation of post-campaign knowledge and practices of exclusive breastfeeding in Uganda. Journal of Health Population and Nutrition 22: 429–439.
- 22. Bessinger R, Katende C, Gupta N (2004) Multi-media campaign exposure effects on knowledge and use of condoms for STI and HIV/AIDS prevention in Uganda. Evaluation and Program Planning 27: 397–407.
- 23. Kincaid DL, Merritt AP, Nickerson L, deCastroBuffington S, deCastro MPP, et al. (1996) Impact of a mass media vasectomy promotion campaign in Brazil. International Family Planning Perspectives 22: 169–175.
- 24. Hutchinson P, Wheeler J (2006) Advanced methods for evaluating the impact of family planning communication programs: evidence from Tanzania and Nepal. Stud Fam Plann 37: 169–186.
- 25. Kincaid DL, Do MP (2006) Multivariate causal attribution and cost-effectiveness of a national mass media campaign in the Philippines. Journal of Health Communication 11: 69–90.
- 26. Angeles G, Guilkey DK, Mroz TA (1998) Purposive program placement and the estimation of family planning program effects in Tanzania. Journal of the American Statistical Association 93: 884–899.
- 27. Angeles G, Guilkey DK, Mroz TA (2005) The determinants of fertility in rural Peru: Program effects in the early years of the national family planning program. Journal of Population Economics 18: 367–389.
- 28. Molyneaux JW, Gertler PJ (2000) The impact of targeted family planning programs in Indonesia. Population and Development Review 26: 61–85.
- 29. Pitt MM, Rosenzweig MR, Gibbons DM (1993) The Determinants and Consequences of the Placement of Government Programs in Indonesia. World Bank Economic Review 7: 319–348.
- 30. Do MP, Kincaid DL (2006) Impact of an entertainment-education television drama on health knowledge and behavior in Bangladesh: An application of propensity score matching. Journal of Health Communication 11: 301–325.
- 31. Rosenbaum PR, Rubin DB (1983) The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika 70: 41–55.
- 32.
Angrist JD, Pischke JS (2009) Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton: Princeton University Press.
- 33. Angrist JD (2001) Estimation of limited dependent variable models with dummy endogenous regressors: Simple strategies for empirical practice. Journal of Business & Economic Statistics 19: 2–16.
- 34. Abadie A, Angrist J, Imbens G (2002) Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica 70: 91–117.
- 35. Imbens GW, Wooldridge JM (2009) Recent Developments in the Econometrics of Program Evaluation. Journal of Economic Literature 47: 5–86.
- 36.
Wooldridge JM (2002) Econometric Analysis of Cross Section and Panel Data. Cambridge: MIT Press.
- 37.
Johns Hopkins University/Center for Communication Programs (2006) Communication For Healthy Living Health Communication Partnership Year Three Progress Report for the period of October 1, 2005– September 30, 2006. Baltimore, MD.
- 38.
Hess R, Meekers D, Storey D (2012) Egypt’s Mabrouk! Initiative. A Communication Strategy for Maternal/Child Health and Family Planning Intergration. In: Obregon R, Waisbord S, editors. The Handbook of Global Health Communication: Wiley and Blackwell, Ltd. 374–407.
- 39.
El-Zanaty F, Meekers D, Armanious D, El-Ghazaly N (2004) Menya Villages Health Survey 2004. Cairo, Egypt.
- 40.
El-Zanaty F, Meekers D, El-Ghazaly N, El-Said MM (2005) Follow-up Demographic and Health Profile of Some villages of Upper Egypt (Menya Governorate). Cairo, Egypt.
- 41. Becker SO, Ichino A (2002) Estimation of Average Treatment Effects Based on Propensity Scores. The Stata Journal 2: 358–377.
- 42. Rosenbaum PR, Rubin DB (1985) Constructing a Control-Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score. American Statistician 39: 33–38.
- 43. Ho DE, Imai K, King G, Stuart EA (2007) Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis 15: 199–236.
- 44. Imbens GW (2004) Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics 86: 4–29.
- 45.
Leuven E, Sianesi B (2003) PSMATCH2: Stata Module to Perform Full Mahalanobis and Propensity Score Matching, Common Support Graphing and Covariate Imbalance Testing.
- 46.
Arellano M, Honore BE (2001) Panel Data Models: Some Recent Developments. In: Heckman JJ, Leamer E, editors. Handbookd of Econometrics. Amsterdam: North-Holland. 3229–3296.
- 47. Chamberlain G (1980) Analysis of Covariance with Qualitative Data. Review of Economic Studies 47: 225–238.
- 48. Baum CF, Schaffer ME, Stillman S (2007) Enhanced routines for instrumental variables/generalized method of moments estimation and testing. The Stata Journal 7: 465–506.
- 49. Angrist JD, Imbens GW, Rubin DB (1996) Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91: 444–455.
- 50.
Stock JH, Yogo M (2005) Testing for weak instruments in linear IV regression. In: Andrews DWK, Stock JH, editors. Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg. Cambridge: Cambridge University Press. 80–108.
- 51.
Roodman D (2009) Estimating Fully Observed Recursive Mixed-Process Models with cmp. Washington, D.C.: Center for Global Development.
- 52. Buis ML (2011) Stata tip 97: Getting at rho’s and sigma’s. Stata Journal 11: 315–317.
- 53.
Staiger H, Stock JH (1997) Instrumental variables regression with weak instruments. Econometrica 65.
- 54. Stock JH, Wright JH, Yogo M (2002) A survey of weak instruments and weak identification in generalized method of moments. Journal of Business & Economic Statistics 20: 518–529.