Methodology Matters

By James Rubin, Ph.D. – 

Transit agency decision-makers receive detailed information on customer perceptions from a variety of sources, including traditional and social media, customer service interactions, and customer experience survey results. All but one of these sources, customer experience surveys, are subject to biased reporting, and even surveys, if not designed and administered properly, will lead to erroneous results. Transit agency leaders need an accurate read on what their customers think of service to make changes to support the positive, improve the negative, and inspire behavior change among agency staff.

Media sources distort. Traditional reporters cite a handful of customers and use that tiny qualitative sample to claim general perceptions. And those customer perceptions are likely hand-picked from a larger sample, whose opinions aren’t included as they don’t support the article. Worse, social media amplifies the latest and loudest, leaving transit agency officials to determine if a massive wave of social media concern really reflects the opinion of most customers or just a few vocal ones.

Though customer service interactions or customer relationship management systems (CRMs) help an agency determine specific conditions that need immediate action, they don’t help at all in determining typical customer perceptions. Approximately 90% of customer service interactions are complaints and very few are commendations. Using this data would lead to a massively inaccurate negative perception of service.

The only way to accurately determine the perceptions of transit customers is to go into the system and measure them by conducting random intercept surveys. It can be very tempting for an agency to conduct surveys online, but the same problem persists in that survey respondents are more motivated to complete a survey when they are dissatisfied than when they are satisfied. We try to minimize the bias by offering incentives to customers who complete the survey, but that also introduces a bias into the results. Often, the incentive is free rides, and if I’m a customer who doesn’t need that incentive, how am I being incentivized to complete the survey?

Another problem is the population of customers who respond to online surveys tends to be limited and insular. When I led customer survey efforts at MTA, we would send hundreds of thousands of invitations for a survey via email to customers who had previously provided us with an email address through a customer service interaction, at a public meeting, on a past survey, or as part of our fare payment system. We would also advertise widely throughout the system with digital and audio assets. When the results came in, about 95% of respondents came from invitations and only 5% from system advertisements. This creates the impression of a random sample survey, but really functions more as a panel survey.

Though typically more expensive and time-consuming, sending teams into the transit system to administer carefully-designed questionnaires is the gold standard for delivering unbiased customer opinions to transit agency decision-makers.

Intercept surveys are the only methodology that produces a truly random sample of customers but cost the most to administer.

Facebook
Twitter
LinkedIn
News Article

Methodology Matters

By James Rubin, Ph.D. –  Transit agency decision-makers receive detailed information on customer perceptions from a variety of sources, including

Read More »

Want to learn more about TransDASH?

Fill out this form below.