Today's Agenda

• Review Sections 3.1, 3.4.2 of QSS Chapter 3

  • Role of randomization in survey sampling

  • Non-response and other sources of bias

Problem

• Most surveys/polls only interview several hundred people/voters

• Goal: infer what 200 million adults are thinking

The 1936 Literary Digest Poll

• Source: automobile registration, telephone books, etc.

• Initial sample size: over 10 million straw vote ballots

• Final sample size: over 2.3 million returned

• The young Gallup used 50,000 respondents

The 1948 Election

The Polling Disaster

• Quota sampling!

• fixed quota of certain respondents for each interviewer

• sample resembles the population with respect to these characteristics

• potential unobserved confounding ⇝ selection bias

Sample Surveys

• Probability sampling to ensure representativeness

• Definition: every unit in the population has a known non-zero probability of being selected

• Simple random sampling: every unit has an equal selection probability

Civilian Attitudes and War against Insurgency

• Conventional war: military against another military

• Counter-insurgency war: military against insurgents

  • From Vietnam war to war in Afghanistan

  • Key to victory: winning hearts and minds of civilians

  • aid provision, information campaign, minimizing civilian casualties

Non-Response and Other Sources of Bias

• Item non-response, like unit non-response, can create bias

• More violent areas ⇝ more non-response

Missing Values by Region: Support for the Taliban

Missing Values by Region: Support for the ISAF

• Sensitive questions ⇝ non-response, social desirability bias

• racial prejudice, corruption, even turnout

List Experiment

• Script for control group:

  I'm going to read you a list with the names of different groups and individuals on it. After I read the entire list, I'd like you to tell me how many of these groups and individuals you broadly support, meaning that you generally agree with the goals and policies of the group or individual. Please don't tell me which ones you generally agree with; only tell me how many groups or individuals you broadly support.

• Karzai Government; National Solidarity Program; Local Farmers

List Experiment

• Script for treatment group:

  I'm going to read you a list with the names of different groups and individuals on it. After I read the entire list, I'd like you to tell me how many of these groups and individuals you broadly support, meaning that you generally agree with the goals and policies of the group or individual. Please don't tell me which ones you generally agree with; only tell me how many groups or individuals you broadly support.

• Karzai Government; National Solidarity Program; Local Farmers; ISAF (Taliban)

Analysis of List Experiment

Proportion of those who support ISAF=ˉY(1)−ˉY(0)=0.049

• Problems with list experiments: floor and ceiling effects

Your Turn

Professor Hurt-and-Confused just read their student evaluations. Eight out of 22 students filled out student course evaluations for course last Spring, whereas 7 out of 27 students filled out their course evals for the same course taught last Fall. The average scores for the two courses (taught exactly the same) are 4.5 and 3.5.

• Apply the concepts from Slide 9 to explain potential causes of bias in course evaluation scores. Given this discussion, what could explain the difference in scores?

• What are some ways to improve the use of student evaluations for assessing teaching effectiveness?

What You Learned

Importance of probability sampling:

1. ensures representativeness

2. enables the calculation of uncertainty (QSS Chapter 7)

Sources of bias in survey sampling

1. unit non-response

2. item non-response

3. social desirability bias

4. differential item functioning (e.g., culturally-driven diff in GRE scores)

Survey methods for sensitive questions

1. list experiment

2. randomize response method

Political Efficacy in China and Mexico

• Goal: explore research tools for correcting survey measurement bias

• In 2002, the World Health Organization conducted a survey of two provinces in China and three provinces in Mexico. This exercise is based on:

• Gary King, Christopher J. L. Murray, Joshua A. Salomon, and Ajay Tandon. (2004). 'Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research.' American Political Science Review, 98:1 (February), pp.191-207.

Self-Assessment Question

How much say do you have in getting the government to address issues that interest you?

(5) Unlimited say, (4) A lot of say, (3) Some say, (2) Little say, (1) No say at all.

Data Exercise

• Download and open the data.

• Compute the mean response for each country.

• Plot the distribution of responses separately for China and Mexico using barplots, where the vertical axis is the proportion of respondents.

vig_chi <- subset(vig, china == 1)
vig_mex <- subset(vig, china == 0)
# China
chi_table <- prop.table(table(vig_chi$self))
barplot(chi_table, main = "China")
# Mexico
mex_table <- prop.table(table(vig_mex$self))
barplot(mex_table, main = "Mexico")

How Much Say Do You Have?

Data Exercise Cont'd

• Which country's respondents seem to have a higher degree of political efficacy? Does this seem plausible when Mexican citizens voted out the Institutional Revolutionary Party (PRI) after more than 70 years or rule and Chinese citizens have not been able to vote in a fair election to date?

Vignettes

After the self-assessment question, three vignette questions were asked.

• [Alison] lacks clean drinking water. She and her neighbors are supporting an opposition candidate in the forthcoming elections that has promised to address the issue. It appears that so many people in her area feel the same way that the opposition candidate will defeat the incumbent representative.

• [Jane] lacks clean drinking water because the government is pursuing an industrial development plan. In the campaign for an upcoming election, an opposition party has promised to address the issue, but she feels it would be futile to vote for the opposition since the government is certain to win.

• [Moses] lacks clean drinking water. He would like to change this, but he can't vote, and feels that no one in the government cares about this issue. So he suffers in silence, hoping something will be done in the future.

The respondent was asked to assess each vignette in the same manner as the self-assessment question.

• How much say does ['name'] in getting the government to address issues that interest [him/her]?

(5) Unlimited say, (4) A lot of say, (3) Some say, (2) Little say, (1) No say at all.

Your Turn

• Compute the proportion of respondents, again separately for China and Mexico, who ranks themselves (according to the self-assessment question) as having less say in the government’s decisions than Moses (the last vignette).

• How many respondents in China think they have less say than Moses?

• How many respondents in Mexico think they have less say than Moses?

• What does this tell you?

Challenge Yourself

• Restrict the data to survey respondents who ranked these three vignettes in the expected order (i.e., Alison ≥ Jane ≥ Moses).

• Now create a variable that represents how respondents rank themselves relative to each vignette. This variable should be equal to 1 if a respondent ranks themselves lower than Moses, 2 if ranked the same as Moses or higher than Moses but lower than Jane, 3 if ranked the same as Jane or higher than Jane but lower than Alison, and 4 if ranked as high as Alison or higher.

• Create the barplots of this new variable as in Question 1. The vertical axis should represent the proportion of respondents for each response category. Also, compute the mean value of this new variable separately for China and Mexico. Compare these results with those obtained in Question 1.