17 Statistical Processes

learn by watching

The links below will launch the video lessons in YouTube

Try Exercises #1-11

Try Exercises #12-13

Complete the Reflection: The Importance of a Representative Sample

Try Exercises #14-15

Try Exercises #16-22

Try Exercises #23-28 (review from previous concepts)

learn by reading

In the last lesson we learned to summarize and display data.  In this lesson we will look at HOW to collect the data in the first place.  Although this may seem like a fairly non-mathematical way to finish up the course, it is critical to realize that if your numbers were not collected in a thoughtful and meaningful way, the resulting statistical summaries don’t mean anything.  We will consider different sampling methods and common types of bias to avoid, as well as different scientific methods to collect data.

statistical vocabulary

There are some key terms that are important to understand and identify before collecting and analyzing your data. Properly identifying these values is an important beginning to a well-designed study.

Statistical Vocabulary

First decide what you want to study:
POPULATION: This is the large group that you are interested in finding out about
PARAMETER: This is the “true value” that you are trying to find that represents the entire population.
Finding a parameter is rarely done, since it is generally time-consuming, costly, or just downright impossible to include every single member of your population.
SAMPLE: This is a subgroup of the population — the group you actually talk to or include in your study
STATISTIC: This is a calculation based on the data gathered from a sample
We can use our statistic as a “best guess” to estimate the true value of the parameter

 

Example 1: You are interested in finding out how many kindergartners’ favorite color in your school district is red.  The district has 2500 kindergartners.  You stop by several of the elementary schools and and interview 53 kindergarten children.  15 said their favorite color was red,  20 said blue,  8 said purple, 5 said green,  3 said yellow, and 2 said pink. Identify the following key terms in this study:

  • population
  • parameter
  • sample
  • statistic

Use your statistic to predict the number of kindergarten students in the school district that like red

Population: All 2500 kindergartners in my school district

Parameter: number of kindergartners with a favorite color of red

Sample: The 53 children I talked to

Statistic: In my sample, I found that 15/53 students like the color red best.  15/53 = 0.283 = 28.3% of the students in my sample prefer red.

The statistic of 28.3% is based on my sample.  Since my sample gives me the best guess of the color preferences of all the district kindergartners, I would predict that 28.3% of the 2500 kindergartners would also like red best: 0.283 * 2500 = 707.5.  So I would expect 707 or 708 kindergartners prefer red.

 

Example 2: Suppose you are interested in the average distance that a student at LWTech travels to get to campus.  There are 3300 students currently enrolled at LWTech.  You talk to 40 students and find that their average commute distance is 11.7 miles. Identify the following key terms in this study:

  • population
  • parameter
  • sample
  • statistic

Population: All 3300 students at LWTech

Parameter: the average commute distance of LWTech

Sample: The 40 LWTech students that I talk to

Statistic: the average commute distance of my sample was 11.7 miles

In this last example, my best guess of LWTech students’ average commute time would be 11.7 miles (my guess is based on my sample survey statistic).  However, if someone else asked a different 40 students, chances are their average would be a different from mine.  This means that I should not be completely confident in my result. We call this sampling bias.

All surveys are subject to sampling bias — there are always going to be different results based on which group I end up selecting to talk talk to.  To account for this we often report our population’s parameter using what we call a margin of error.  The margin of error creates a bubble of accuracy around my statistic.  The size of the margin of error depends on several factors (sample size, standard deviation, and confidence required), and we will not get into calculating them in this lesson (take a statistics class if you are interested!). This would be information that you are given.

Suppose in my commute example, it was determined that I could be 95% confident that other samples of LWTech students will be within 1.2 miles of my statistic.  I would say this as “my margin of error was ±1.2 miles”.  We would report our “average commute distance of LWTech students to be 11.7±1.2 miles”.  We can also write this as a range of values where we believe that statistic will fall.  We call this a confidence interval.  In this case, I would report my confidence interval to be a commute distance between 10.5 miles and 12.9 miles.  We usually write confidence intervals in this format: (10.5,12.9) miles

Accounting for Sampling Bias

We use a margin of error to allow for natural variation between sampling groups.  We center the margin of error around our calculated statistic.

We can declare a confidence interval to be a range about which we expect future sampling groups to lie.  This makes for a good estimate of the parameter we are trying to find.  We find a confidence interval to be: (“statistic” – “margin of error”  , “statistic” + “margin of error”).  Our statistic will always be located half-way between this lower and upper bound.

Exercises: Try These!

  1. Describe the difference between a sample and a population.

  2. Describe the difference between a statistic and a parameter.

  3. The ASPCC randomly selects 200 students from UW Bothell campus to participate in a childcare survey in order to determine the demand for additional childcare options for all UW students.

    1. Who is the intended population?

    2. What is the sample?

    3. Is the collected data representative of the intended population? Why or why not?

  4. A local research firm randomly selects 1200 homes in King County to determine support for adding compost pick up to residents’ regular garbage service.

    1. Who is the intended population?

    2. What is the sample?

    3. Is the collected data representative of the intended population? Why or why not?

  5. A political scientist surveys 28 of the current 106 representatives in a state’s congress. Of them, 14 said they were supporting a new education bill, 12 said there were not supporting the bill, and 2 were undecided.

    1. Who is the population of this survey?

    2. What is the size of the population?

    3. What is the size of the sample?

    4. Give the statistic for the percentage of representatives surveyed who said they were supporting the education bill.

    5. If the margin of error was 5%, give the confidence interval for the percentage of representatives we might we expect to support the education bill and explain what the confidence interval tells us.

  6. The city of Raleigh has 9,500 registered voters. There are two candidates for city council in an upcoming election: Brown and Feliz. The day before the election, a telephone poll of 350 randomly selected registered voters was conducted. 112 said they’d vote for Brown, 207 said they’d vote for Feliz, and 31 were undecided.

    1. Who is the population of this survey?

    2. What is the size of the population?

    3. What is the size of the sample?

    4. Give the statistic for the percentage of voters surveyed who said they’d vote for Brown.

    5. If the margin of error was 3.5%, give the confidence interval for the percentage of voters surveyed that we might we expect to vote for Brown and explain what the confidence interval tells us.

  7. To determine the average length of trout in a lake, researchers catch 20 fish and measure them. Describe the population and sample of this study.
  8. To determine the average diameter of evergreen trees in a forested park, researchers randomly tag 45 specimens and measure their diameter. Describe the population and sample of this study.
  9. A college reports that the average age of their students is 28 years old. Is this a parameter or a statistic?

  10. A local newspaper reports that among a sample of 250 subscribers, 45% are over the age of 50. Is this a parameter or a statistic?

  11. A recent survey reported that 64% of respondents were in favor of expanding the BIKETOWN bike share system to the greater Portland area. Is this a parameter or a statistic?

sampling methods

At this point, it’s important to talk about the ways that we can decide on our sample group. To get a good sample, the more random the better!  Despite our best attempts, people do not choose randomly — studies have shown people prefer certain numbers, are more likely to choose people that look like them, or (if they are aware) might overcompensate when they feel they are not being “random enough.

Being random needs to be intentional, and using a “non-person”-ed approach to selecting the sample group for a study. There are many, many ways to approach picking a sampling group.  Here are a few of them.

Sampling Methods

Some commonly used sampling methods include:

  • SIMPLE RANDOM SAMPLE — Every member in the population is listed and has an equal chance of being selected during a randomized process. This is the “gold standard” of choosing a sampling group. However, it is often difficult or impossible to have access to a full list of the population to choose from.
  • STRATIFIED SAMPLE — The population is split into distinct demographic groups.  Then random selections are made from each group.  This is often used when you have key groups that you would like to ensure are adequately represented in your study (for example: certain age groups or certain genders or certain political affiliations)
  • CLUSTER SAMPLE — The population is divided into many different subgroups.  Then some of these subgroups are randomly selected. ALL members of the selected subgroups are included in the sample.
  • SYSTEMATIC SAMPLE — The sample is chosen by selecting every “nth” member of the population.  Although this isn’t exactly random, it does take the researcher’s influence out of the selection process.
  • CONVENIENCE SAMPLE — By far the worst choice for a study, and a method to be avoided. In this process, the sample is chosen by the researcher from a convenient group of participants available.

 

Example 3: Identify an example of each type of sampling method described.

SIMPLE RANDOM SAMPLE: You want to conduct a study to determine if students in your major are willing to take classes during summer quarter.  You work with the registrar to get a list of all the students in your program.  Using a random number generator, you identify 20 specific students from your list.  Then you contact those 20 people to participate in your study.

STRATIFIED SAMPLE: You want to determine if the potato chip bags coming out of your factory match the weight listed on the bag.  Too light and customers are mad.  Too full and your company is losing money.  You are running 3 different machines.  You decide to use a sample of 30 boxes to test, and randomly select 10 boxes in the warehouse that were produced from each machine.

CLUSTER SAMPLE: You want to determine if students at your college eat breakfast.  You use randomized methods to select 5 classrooms at the college.  Then you interview every student in each of those 5 classrooms.

SYSTEMATIC SAMPLE: You want to check on customer satisfaction at your store’s location.  You talk to every 5th customer that approaches the counter that day.

CONVENIENCE SAMPLE: Your teacher assigns you to conduct a survey with a minimum of 50 responses.  You decide to put your question up on your social media account and select the first 50 replies.

 

Exercises: Try These!

12) Which sampling method is being described?

    1. In a study, the sample is chosen by separating all cars by size and selecting 10 of each size grouping.

    2. In a study, the sample is chosen by writing everyone’s name on a playing card, shuffling the deck, then choosing the top 20 cards.

    3. Every 4th person on the class roster was selected.

13) Which sampling method is being described?

      1. A sample was selected to contain 25 people aged 18-34 and 30 people aged 35-70.

      2. Viewers of a new show are asked to respond to a poll on the show’s website.

      3. To survey voters in a town, a polling company randomly selects 100 addresses from a database and interviews those residents.

The importance of a representative sample

While all sample data inherently has a bit of variability based on the group we choose (we call this sampling variability), sometimes the variability is caused by a flaw in the design of how the groups for a study (we call this selection bias).  Anytime that we end up with a sample group that has unique characteristics which might be different from the population in general, we are suffering from selection bias.  To see the importance of this, let’s consider the make-up of the United States Congress – a “sample” of the United States population charged with creating laws for the country.

Reflection Assignment: The Importance of a Representative Sample

The current Congress of the United States has been lauded as the most diverse to date.  While this is certainly the case (if you compare the make-up of Congress from just a decade or two ago), how “representative” is our current governing body?  Let’s take a look at some statistics to compare.

There were 534 voting lawmakers in the 118th Congress (House of Representatives and Senate), in March 2023.

Fill in the following table, and then answer the questions that follow.

Characteristics % of US Population Expected # in Congress Actual # in Congress % of Congress
Black/African American 13.6% 61
American Indian/Alaska Native 1.3% 5
Asian/Native Hawaiian/Pacific Islander 6.4% 18
Multiracial 2.9% 4
Hispanic/Latino 18.9% 59
White, not Hispanic/Latino 59.3% 401
Married 52% 440
Women 50.5% 150
Openly Lesbian/Gay/Bisexual 6.5% 13
Foreign-Born 13.6% 18
Identify as Christian 63% 469

[Note: Congress characteristics were gathered from the Congressional Research Service Report; Population statistics from US census reports ; sexual orientation  and religious affiliation from recent national Pew poll surveys.]

1) What is your general impression about the current make-up of Congress?  Did anything surprise you? Would you consider this a representative sample of the US population? Why or why not?

2) Why do you think the Congressional make-up is what it is (you may want to consider a historical perspective)?  What does that say about power and privilege in our current society?  How do you see things changing (or not changing) in the future?

3) What effect do you think having a difference in representation might have on issues that arise in Congress?  Give examples of some issues that might not receive adequate attention and/or reflect the opinions of the general US population.  Think about the under-represented groups in the chart above — what issues might be more pressing or important to these groups?

If your sampling method suffers from bias, all results and conclusions from that study are suspect.  YOUR SELECTED SAMPLE MUST BE A REFLECTION OF THE POPULATION IN GENERAL OR YOUR DATA IS USELESS!  Randomly selecting your sample from the entire population is the best way to avoid bias and get a representative sample.

SELECTION BIAS

Bias is a result of choosing a sample that is not representative of the general population. Sometimes the way you conduct a survey may unintentionally keep particular groups from being selected, and their opinions might be very different from the population in general.

Consider collecting surveys at a shopping mall to learn about voting preferences in the city.  If you conduct this survey during the daytime, you will likely get stay-at-home parents, remote workers, and/or swing-shift workers.  By design, you are leaving entire groups out that may have unique priorities and concerns that would affect your overall results.

In the early 2000s, poll researchers used to choose survey participants by calling randomly-selected phone numbers.  The methods were intentionally randomized, but because of the way the survey was distributed, a powerful group of voters was unintentionally left out of selection.  See, the researchers were using land-lines to conduct all their surveys.  Although this had been an effective method in the past, at this time, many college-age students had discontinued land-line use altogether and were exclusively using their cellular phones.  Because the poll selection process dis-included a large number of young voters (who generally vote more progressively than their older counterparts), poll results incorrectly predicted the result of a presidential election.

There are some bias types that are fairly common.  As a reader of statistical analysis, you need to watch for these flaws in the design of a study.  If the data collection process is flawed, then the reported results cannot be trusted.

Common Selection Bias Types

VOLUNTARY RESPONSE BIAS — participants in the survey have to actively go out of their way to participate in a survey (call in to a phone number, mail-in a survey form, go to a specific website or loacation, etc.).  In cases like these you are likely to get low response rates.  And the people who DO respond tend to have more extreme views that they want to voice.  Those with more “middle” views often can’t be bothered to participate in the survey.

PERCEIVED LACK OF ANONYMITY — participants in a survey are unlikely to answer truthfully in a survey if they fear their responses might have negative consequences.

LEADING QUESTIONS — questions are worded in an emotionally-charged way, trying to get the participant to choose their side when they respond — these are often about controversial topics.

SELF-INTEREST STUDY — the study is sponsored by a company or individual who will benefit financially from certain responses

This is just a small collection of ways selecting a sample can result in bad data.  Others you may have heard of include: confirmation bias, cognitive bias, optimism bias, anchoring bias, etc. For the purposes of our lesson, we will stick to the four described above. If none of these categories apply, then just use the more general “selection bias” along with an explanation of why the design of the study was flawed.

 

Example 4: Identify an example of each of the bias types listed above.

VOLUNTARY RESPONSE BIAS: While watching a television show, a commercial asks users to go to their company’s website to participate in an important survey.  [users have to go out of their way to stop what they are doing and go to the website]

PERCEIVED LACK OF ANONYMITY BIAS: A police officer is conducting a study of how fast cars are going down a stretch of road.  They are standing outside in uniform, using their radar gun to collect the data.  [people would intentionally slow down, afraid that they would get a ticket]

LEADING QUESTIONS: “A lot of Americans believe we need stronger gun laws to protect our innocent children while they are at school. Do you agree?” [wording pushes the participant towards a particular reponse]

SELF-INTEREST STUDY: Nintendo sponsors a survey about gaming console preferences. [the company would benefit financially from being able to promote a particular type of answer in their survey]

 

Exercises: Try These!

14) Identify the most relevant source of bias in each situation.

    1. A survey asks the following: Should the mall prohibit loud and annoying rock music in clothing stores catering to teenagers?

    2. To determine opinions on voter support for a downtown renovation project, a surveyor randomly questions people working in downtown businesses.

    3. A survey asks people to report their actual income and the income they reported on their IRS tax form.

    4. A survey randomly calls people from the phone book and asks them to answer a long series of questions.

    5. The Beef Council releases a study stating that consuming red meat poses little cardiovascular risk.

    6. A poll asks, “Do you support a new transportation tax, or would you prefer to see our public transportation system fall apart?”

15) Identify the most relevant source of bias in each situation.

  1. A survey asks the following: Should the death penalty be permitted if innocent people might die?

  2. A study seeks to investigate whether a new pain medication is safe to market to the public. They test by randomly selecting 300 people who identify as men from a set of volunteers.

  3. A survey asks how many sexual partners a person has had in the last year.

  4. A radio station asks listeners to phone in their response to a daily poll.

  5. A substitute teacher wants to know how students in the class did on their last test. The teacher asks the 10 students sitting in the front row to state their latest test score.

  6. High school students are asked if they have consumed alcohol in the last two weeks.

Observations vs. experiments

There are 2 primary approaches that can be used to collect data.

RESEARCH DESIGN METHODS FOR COLLECTING DATA

Observation: The data collected is observed or measured, then recorded.  The role of the researcher is passive. Researchers do not control the conditions or intervene in the events

Experiment: The researcher assigns a “treatment” to a particular group, then records the results.  The role of the researcher is active, manipulating the variables in a carefully controlled environment to observe the effects of a treatment.

In an observation, the researcher is taking the world as it is.  They do not assign variables, but seek for situations where the topic of interest already is defined.  This allows a much wider range of possible topics of study.  However, as a result, observations cannot explain cause-and-effect, because other factors may be at play in the background (we call these confounding variables).  Observational studies can only establish relationships (correlation) between things that are being studied.

In an experiment, the researcher works to carefully control the variables at play.  Well-designed experiments will contain a control group — a randomly selected group of participants who behave “normally” so that the effect of the treatment can be directly compared and measured.  In such a study it is possible to establish cause-and-effect relationships because the application of the treatment is the only difference between the two groups.

Observations allow for a lot of studies that would be time-consuming, expensive, and/or unethical to study as an experiment.

Take, for example, a desire to study the connection between smoking and lung cancer.  Consider what would be involved in creating an experiment to establish cause and effect.  You would have to randomly assign 2 groups of young people who have never smoked.  For the first group, you are going to tell them to start smoking a pack of cigarettes a day for the next 40 years. For the second group, you would have to direct them not to smoke for the next 40 years. Then you would have to seek out the original participants of the study to see if they had developed lung cancer or not.  It would be a very expensive and extensive study to track the groups over a long period of time.  It also would be pretty unethical to assign one group to smoke for 40 years — especially if you think there are negative impacts to smoking.

An observational study, on the other hand, would be relatively inexpensive to run.  You could contact people who already have lung cancer and see if they smoked in the past or not.  Alternatively, you could collect a group of smokers and a group of non-smokers and check-in after a couple decades to see if they had developed lung cancer.  Either way, you are not assigning anyone to smoke.  However, at the end, all you can do is say that there is a correlation between people who smoke and people who develop lung cancer.  We cannot establish cause and effect.

Specialized Types of Experiments

CONTROLLED EXPERIMENT: an experiment that has a control group and a treatment group.   The treatment group is assigned a task, while the control group behaves normally.

PLACEBO-CONTROLLED EXPERIMENT: Medical experiments have shown that people who think they are getting help with a problem often improve, even if they are given something that just looks like the medication.  To account for this phenomenon, a placebo-controlled experiment is one where both the treatment and the control groups are given something that looks like the treatment, so all participants feel that they are getting help.  Only one group, however, has a pill with the key ingredient(s) being tested.  The other pill has no medicinal effects.

BLIND EXPERIMENT: This is an experiment where the participants do not know whether they are part of the treatment group or not.  A placebo-controlled experiment is one example of a type of blind experiment.

DOUBLE-BLIND EXPERIMENT: This is an experiment where the participants don’t know which group they are in, but ALSO the people collecting and recording the data don’t know what group the participant is in.  This gets a more objective recording of data (a doctor may inadvertently look more closely for improvements in a patient using a new treatment they feel is effective).  Of course, the researcher on the back end that analyzes the data has to know which person is which group when they compile results.

 

Exercises: Try These!

  1. Identify whether each situation describes an observational study or an experiment.

    1. The temperature on randomly selected days throughout the year was measured.

    2. One group of students listened to music and another group did not while they took a test and their scores were recorded.

    3. The weights of 30 randomly selected people are measured.

  2. Identify whether each situation describes an observational study or an experiment.

    1. Subjects are asked to do 20 jumping jacks, and then their heart rates are measured.

    2. Twenty coffee drinkers and twenty tea drinkers are given a concentration test.

    3. The weights of potato chip bags are weighed on the production line before they are put into boxes.

  3. A team of researchers is testing the effectiveness of a new vaccine for human papilloma virus (HPV). They randomly divide the subjects into two groups. Group 1 receives new HPV vaccine, and Group 2 receives the existing HPV vaccine. The patients in the study do not know which group they are in.

    1. Which is the treatment group?

    2. Which is the control group (if there is one)?

    3. Is this study blind, double-blind, or neither?

    4. Is this best described as an experiment, a controlled experiment, or a placebo-controlled experiment?

  4. Studies are often done by pharmaceutical companies to determine the effectiveness of a treatment. Suppose that a new cancer treatment is under study. Of interest is the average length of time in months patients live once starting the treatment. Two researchers each follow a different set of 40 cancer patients throughout this new treatment.

    1. What is the population of this study?

    2. Would you expect the data from the two researchers to be identical? Why or why not?

    3. If the first researcher collected their data by randomly selecting 10 nearby ZIP codes, then selecting 4 people from each, which sampling method did they use?

    4. If the second researcher collected their data by choosing 40 patients they knew, what sampling method did they use? What concerns would you have about this data set, based upon the data collection method?

  5. For the clinical trials of a weight loss drug containing Garcinia Cambogia the subjects were randomly divided into two groups. The first received an inert pill along with an exercise and diet plan, while the second received the test medicine along with the same exercise and diet plan. The patients do not know which group they are in, nor do the fitness and nutrition advisors.

    1. Which is the treatment group?

    2. Which is the control group (if there is one)?

    3. Is this study blind, double-blind, or neither?

    4. Is this best described as an experiment, a controlled experiment, or a placebo-controlled experiment?

  6. A study is conducted to determine whether people learn better with routine or crammed studying. Subjects volunteer from an introductory psychology class. At the beginning of the semester 12 subjects volunteer and are assigned to the routine studying group. At the end of the semester 12 subjects volunteer and are assigned to the crammed studying group.

    1. Identify the target population and the sample.

    2. Is this an observational study or an experiment?

    3. This study involves two kinds of non-random sampling: 1. Subjects are not randomly sampled from a specified population and 2. Subjects are not randomly assigned to groups. Which problem is more serious? What effect on the results does each have?

  7. To test a new lie detector, two groups of subjects are given the new test. One group is asked to answer all the questions truthfully. The second group is asked to tell the truth on the first half of the questions and lie on the second half. The person administering the lie detector test does not know what group each subject is in. Does this experiment have a control group? Is it blind, double-blind, or neither? Explain.

  8. A poll found that 30%, plus or minus 5% of college freshmen prefer morning classes to afternoon classes.

    1. What is the margin of error?

    2. Write the survey results as a confidence interval.

    3. Explain what the confidence interval tells us about the percentage of college freshmen who prefer morning classes?

  9. A poll found that 38% of U.S. employees are engaged at work, plus or minus 3.5%.

    1. What is the margin of error?

    2. Write the survey results as a confidence interval.

    3. Explain what the confidence interval tells us about the percentage of U.S. employees who are engaged at work?

  10. A recent study reported a confidence interval of (24%, 36%) for the percentage of U.S. adults who plan to purchase an electric car in the next 5 years.

    1. What is the statistic from this study?

    2. What is the margin of error?

  11. A recent study reported a confidence interval of (44%, 52%) for the percentage of two-year college students who are food insecure.

    1. What is the statistic from this study?

    2. What is the margin of error?

  12. A farmer believes that playing Barry Manilow songs to his peas will increase their yield. Describe a controlled experiment the farmer could use to test his theory.
  13. A sports psychologist believes that people are more likely to be extroverted as an adult if they played team sports as a child. Describe two possible studies to test this theory. Design one as an observational study and the other as an experiment. Which is more practical?

 

License

Icon for the Creative Commons Attribution-NonCommercial 4.0 International License

Math in Society from a Diversity and Social Justice Lens Copyright © by Sherry-Anne McLean is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.

Share This Book