By Luke Allison
Senior Project Analyst
That’s right–confidence levels are chosen, not created. Statisticians and surveyors make a conscious decision to set the confidence level for a survey. And that should happen before the data is even collected.
Before? Yes, before. That’s because a confidence level strikes a balance between precision and error-risk.
In survey-land, the more precise you want to be, the more you open yourself up to potential error (missing the mark). The less error-risk you are, the less precise your data will be.
A baseball analogy: baseball hitters have “sweet spots” inside the strike zone, where they get their best hits. That means they also have “cold spots” inside the strike zone where they usually don’t get hits. Hitters have to balance swinging at pitches in their “sweet spot” (precision) while still swinging at strikes in their “cold spots” so they don’t strike out (error).
We’re trying to do the same thing by choosing a confidence level. Be as precise as we can be, while managing our error-risk.
How does this play out? It’s based on the key idea that all results from surveys are ranges, not single values.
For example, we ask fixed route customers their satisfaction with on-time performance. In a sample of 400 customers, let’s say that 300 are satisfied with OTP. That makes our sample percentage 75% (300/400). We shouldn’t look at that number as simply 75%. Instead, we should look at OTP satisfaction as somewhere between 70.2% and 79.8%.
For the most statistically astute reading this, you probably figured out that I just employed the Margin of Error. With a sample of 400, the Margin of Error is ±4.8%. So, we add and subtract 4.8% to and from 75% to get our all important range.
How does this connect to the confidence level? The confidence level is one of the two main factors that affect the Margin of Error. The higher the confidence level, the wider the Margin of Error. The lower the confidence level, the narrower the Margin of Error.
Back to our OTP example for explanation. If we had chosen a 90% confidence level, instead of a 95% confidence level, then our Margin of Error would be 4.0% (compared to 4.8%). So our OTP satisfaction range would be 71% to 79%. Do you see how the lower confidence level narrowed our range? We just became more precise, but also just took a greater error-risk.
It’s time to talk about the error-risk we are taking. We risk not capturing the true population percentage within our range. That’s a protein-packed sentence that you just read. It may be worth reading again, and it’s definitely worth unpacking.
Surveys utilize a sample of your population to estimate what your whole population thinks. If we could ask every single one of your customers how they feel about on-time performance, we would get the true population satisfaction percentage.
Let’s say that you have 30,000 customers. We ask all of them about OTP. 23,760 of them are satisfied. That makes your OTP satisfaction true population percentage 79.2% (23,760/30,000). That’s a really useful number to know. The problem is that it’s almost impossible to get. Or at least VERY time-consuming and costly.
So instead of asking all 30,000 customers, we ask a random 400 and use that sample range to try and capture the true population percentage.
In our example above, our sample (300/400) yielded a range for OTP satisfaction from 70.2% to 79.8% when we used a 95% confidence level. Did we capture the true population percentage for OTP satisfaction (79.2%)? Yes! 79.2% is inside our range.
When we used a 90% confidence level, our range was 71% to 79%. Did we capture the true population percentage (79.2%)? No! 79.2% is outside the range.
It’s time to reveal what 95% and 90% actually mean. They denote the percentage of the time that we capture the true population proportion. With a 95% confidence level, we should capture the true population proportion 95% of the time. So, if we conducted 100 samples, 95 of them should capture the true population percentage. That means that we’re missing the true population percentage 5% of the time. If we downshift to 90% confidence, we open ourselves up to missing 10% of the time (or 10 out of 100).
So, what do you want for your surveys? How do you want to balance precision and error-risk? Maybe most importantly, how will you view your survey results now? Can you see the range? Do you see the true population percentage lurking in the background? And do you see the importance of multiple waves of surveying? The more waves, the more likely we are to identify the ranges that missed and confirm the ones that hit.
Thanks for taking a stroll down statistics lane. Now, when you watch Family Feud or a baseball game, you can wow your friends with insightful talk about confidence levels.
