Improving survey methodologies for cessation studies

PDA is passionate about survey methodology, constantly learning and applying new methods to yield the most accurate survey results.

Background

A primary outcome measure for our tobacco cessation evaluation clients is quitting success measured seven months after program registration. Program and funding decisions are informed by this important measure. In any study that uses self-reported survey results, the representativeness of the sample affects the accuracy and generalizability of the results. In studies that ask about the success of a program (such as quitting tobacco), those who do not respond to the survey are less likely to be successful.

Graph related to survey methodology

Unfortunately, the past decade has seen response rates for most population surveys decreasing, and quit studies are no exception. PDA has been working for the past two decades to conduct studies with ample response rates, to address issues of representativeness, and to provide cost-effective survey solutions to our clients. 

Approach

We are always honing our craft. We attend conferences such as the American Association of Public Opinion Researchers and comb the literature for effective methods. Some of our key survey strategies include:

Survey methodology modes

Experimenting with multi-mode survey designs. Ten years ago, our quit surveys were conducted exclusively by phone. We now reach out to participants via mail, email, and even text, depending on the population. We have conducted studies that show the web response option both reduces survey costs and reaches a population that might not answer the phone. We built our own web-based survey management tool that allows us to seamlessly conduct multi-mode surveys.

Testing different incentive strategies. We are well-versed in different incentive strategies, including both prepaid and promised incentives of varying amounts. We routinely conduct experiments on the effect of different incentive amounts on response rate and survey cost.

Modifying our survey communications. We painstakingly design pre-notification letters and email invitations in hopes of increasing their open rates by potential responders. We experiment with the timing and wording of our communications to determine which are most effective.

Tweaking our survey attempts. We have done studies to look at the impact of the number of attempts to reach a participant on representativeness. We also test out the best days and times to make survey contacts.

Exploring how to most accurately interpret survey data. The decline in both survey response and consent rates across the field has led us to conduct investigations into the impact of these declines on responder quit rates.  In 2015 we collaborated with clients in three different states to examine the impact of declining response rates by studying survey responders that were the most difficult to reach. We have also used Bayesian methods to conduct simulations of true quit rates after applying assumptions about those who did not respond. Finally, in 2018 we partnered with St. Olaf College math and statistics students to explore options for measuring error associated with quit rates.

Results

Our work makes our clients happy. Our tobacco cessation clients enjoy response rates that are consistently among the highest of their peers, and routinely above the target of 50% set for these studies. Because of our investigations and studies, we accurately estimate survey costs and consult our clients on which methods are cost saving and which increase representativeness.

Our work contributes to the field. Between 2001 and 2018, PDA employees have presented findings from our survey methodology studies at 18 conferences. Seven current PDA employees have presented at one or more of the following conferences:

  •       National Conference on Tobacco or Health
  •       American Association of Public Opinion Researchers
  •       American Evaluation Association
  •       North American Quitline Consortium
  •       Minnesota Health Services Research Conference
  •       Minnesota Evaluation Studies Institute

We have become known in the tobacco cessation field as experts in survey methodology and in 2009 and 2015 were asked by the North American Quitline Consortium to author an issue paper on best practices for calculating a quit rate. These NAQC papers provide recommendations for survey protocols, study eligibility, sample sizes, dealing with missing data, interview training, and data collection that are considered standards in the field. Our paper “The effect of survey nonresponse on quitline abstinence rates: Implications for practice” was published Nicotine and Tobacco Research in 2015

Related Highlights

Using cognitive interviews to improve quantitative survey data

PDA used cognitive interviews to assess the functionality of survey items related to second-hand smoke exposure.

Read More

Contributing to education research through survey and statistical support

PDA supported geoscience education researchers by administering a survey to over 10,000 instructors and providing survey design and statistical support.

Read More

Helping schools improve culture through advanced surveys and automated reporting

PDA’s custom-built survey administration and automated reporting engine helps bring an anti-bullying program to American schools.

Read More


See More Highlights

Our Team

Our dedicated and talented team is united by our common desire to improve communities.

Polly DuRose

Polly DuRose

Survey Coordinator and Interviewer

Allison Wolf

Allison Wolf

Survey Interviewer

Mary Kim

Mary Kim

Survey Interviewer