St. Anthony Main
219 Main Street SE, Suite 302
Minneapolis, Minnesota 55414
Written by Melissa Chapman Haynes and Angie Ficek
We are the contracted evaluator for the Ohio Comprehensive Cancer Control Program (CCCP), which includes the cancer Program, the Ohio Cancer Plan, and the Partnership (the coalition), the Ohio Partners for Cancer Control (OPCC). The essence of CCCP work is on facilitating partnerships, increasing coordination and reducing duplication, and creating a value added for the individuals who volunteer their time to CCCP efforts. By design, the OPCC brings together a diverse array of stakeholders working on, affected by, or interested in reducing cancer incidence, mortality, and disparities. Given the wide range of activities, types of stakeholders, and key priorities, it was a formidable challenge to create and implement a responsive yet feasible evaluation and report of the OPCC’s work!
What we did
As with most coalitions, one way that OPCC collects information about the coalition’s processes is through an annual member survey. The findings are used internally to understand the strengths and areas in need of improvement. Further, the strengths inform promotional efforts of the OPCC in an effort to recruit interested stakeholders to join the coalition. A similar survey has been done for multiple years, and there are consistently positive findings every year, though we collaborated with ODH to include some open-ended items and some demographic items so we could better understand the meaning behind those numbers. To compliment and extend the survey data, we decided to conduct interviews as well to take a deeper dive into the coalition’s processes. There was a total of 12 interviews, and they were conducted with the OPCC executive chair, committee chairs, the CCCP staff at the Ohio Department of Health, and the Ohio Department of Health’s program cancer partners.
In the end, we produced a two-page summary in PowerPoint of the coalition survey findings and intertwined some related interview findings as well. We also produced several one pagers based on the interview findings that served as success stories and descriptions of emerging partnerships. The one-pagers helped to further showcase Ohio’s CCCP’s efforts.
What we learned
The succinct, visual nature of what we produced went over very well with our client. The lollipop chart distilled a lot of information that hasn’t changed much over time into one chart, and we felt it worked better than a bar chart because the it focuses your eyes on the end of the data points, which are all high. The inclusion of interview findings helped OPCC get a better sense of the context of the survey findings, and the interviews also highlighted the need for expanding OPCC membership, clarifying the resource limitations of the coalition, and coordinating the evaluation efforts across the various ODH partners, OPCC committee chairs, and internal and external evaluation efforts. We are continuing to incorporate interviews into the evaluation of the coalition, and also hope to continue to use this reporting style as we look to engage a broader array of stakeholders in partnership with ODH.
Written by Sara Richter and Angie Ficek
What it is
Our stats team recently partnered with five undergraduate students from St. Olaf College to determine if there is a better way to estimate the percent of participants from a tobacco cessation program who quit using tobacco, which we call a “quit rate”, and its variance. The project took place during St. Olaf’s J-term with juniors and seniors who are accepted into the math practicum course. The math practicum course divides the students into small groups, and each group works with a local organization to answer a specific stats-related question posed by the organization.
In our evaluations of tobacco cessation programs, we estimate a quit rate as the responder rate, or the percent of people who responded to the follow-up survey and who quit, and we apply a standard two-sided 95% confidence interval. This method poses a bit of a problem in that non-responders are more likely to still be using tobacco, so the responder rate can be an overestimate of the true quit rate. In the past, this has not been overly worrisome as the percent of people consenting to participate (the consent rate) was very high across all programs (>90%) and the percent of consenting participants who responded to the follow-up survey (the response rate) was also high (>50%). However, recently many of the tobacco cessation programs that we evaluate have experienced declining consent rates and declining response rates to the follow-up surveys. This can lead to inflated quit rates and confidence intervals that are inaccurate. We were preparing to investigate these trends and their impact on our quit rate calculations when St. Olaf approached us about the collaboration.
What we did
The students worked countless hours throughout January to get to the bottom of this dilemma. Using a statistical program, R, they created a simulation that allowed them to evaluate different methods of calculating the quit rate and its corresponding confidence interval under a variety of consent and response rates. We checked in with them periodically throughout January and provided feedback on their methods, but they ultimately did all of the legwork. In the end, they provided us with a report, complete with statistical code in R, and a presentation that provided an answer to our question.
Where we go from here
The students identified some interesting trends and came to some eye-opening conclusions that we plan to discuss further internally and publish. We may rethink our methods for calculating quit rates, which could impact how quit rates are calculated for tobacco control programs throughout the country.
We felt the collaboration with the St. Olaf students was a mutually beneficial endeavor. It provided them with a real-world project with messy data, and they contributed analyses and results that could potentially be used to make recommendations that have a nationwide impact on how quit rates are calculated. It allowed us to delegate a non-billable task that might have taken us six months to complete. It required some set-up time on our end to get the data organized and think through the problem and how to present it to them, but the amount and quality of the work that the students produced in the time given was well worth it.
We are excited to have Vince join our team as a Senior Business Analyst!
Vince has strengths in requirements elicitation, collaboration with partners, identifying solutions, and problem solving. He enjoys working closely with business partners and team members to deliver innovative process and technology based solutions that support and enhance the user experience.
Vince is focused on writing and organizing specifications for a large software system used by the Florida Department of Health. His primary focus is to ensure that we make strategic improvements to our system that provide the best value, as well as providing clear specifications for our development team to execute those improvements.
Prior to joining PDA, Vince worked as an Associate Product Owner at Renewal by Andersen where he led the strategic vision and specification writing for the primary Customer Relations Management tool. His background includes project management, business analysis, and application support.
Some of Vince’s primary interests outside of work are gaming (board games and video games), woodworking, cooking, and personal fitness. He particularly likes games that encourage competitive strategies and problem solving.
We added three new faces to our team recently – one in each of our three divisions. Say hello to Sam, Heather, and Jake!
Sam Carlson, Associate Analyst
Sam is a data-enthusiast with a background in tobacco-control policy and smoking cessation research. She enjoys working on community-oriented projects where she can see timely impact and real-world application of research findings. She also loves learning new statistical methods and SAS programming tricks to better meet the needs of our clients. Sam will support the data processing, sampling, and analysis for many of the state cessation programs that we evaluate. She will also conduct analysis and provide data support for other statistics and evaluation projects as needed.
Prior to PDA, Sam worked at the Association for Nonsmokers-MN, supporting local tobacco policy research and implementation efforts. During graduate school at the University of Minnesota (UMN), she worked as a research assistant for the School of Public Health Tobacco Policy Research Group, as well as the UMN’s Tobacco Research Programs. There, she was a study coordinator for a clinical trial on using exercise to promote self-initiated quit attempts among smokers. After graduate school, she worked as a biostatistician for the UMN Medical School, Department of Family Medicine & Community Health. During her time at the UMN, she co-authored 11 articles published in peer-reviewed journals and 20 abstracts for professional meetings.
Sam received her Master’s Degree in Public Health (Epidemiology) with a minor in Biostatistics from the University of Minnesota School of Public Health in the spring of 2016. Her Bachelor’s Degree is in Spanish Culture & Literature with a minor in Community Health Education from the University of St. Thomas.
In her free time, Sam volunteers as an advisor and co-founder of a nonprofit food-rescue organization called TC Food Justice. She also enjoys long-distance running, yoga, and exploring local restaurants and breweries with her soon-to-be husband.
Heather Moore, Evaluation Specialist
Heather has extensive experience using evaluation data to create tailored programs and services to meet the unique needs of intended populations. She enjoys seeing how the results of process and formative evaluations can be applied to improve programs and services for recipients. Heather brings an empathetic and intuitive approach to her work. She is especially passionate about disparities in healthcare and the way in which thoughtful and well-designed evaluations can identify gaps in services and improve outcomes for priority populations. Heather will focus on the evaluations of a statewide comprehensive cancer control program and initiative to improve healthy eating and active living in 23 Ohio counties as well as a tobacco cessation program in Hawaii.
Prior to joining PDA, she directed the evaluations of patient and family services at a Minneapolis-based healthcare non-profit organization. In her role, she led the strategic planning efforts, conducted systems thinking to implement a customized evaluation model, and guided the work of multidisciplinary teams to ensure effective program and evaluation planning for patient and family services. She also managed the evaluation activities under grant-funded projects, and led the development of patient-friendly consent forms and educational materials for a clinical trials network. Heather is experienced in disseminating evaluation results through presentations at national conferences and publications in peer-reviewed journals. She has been a member of the American Evaluation Association since 2007.
Heather received her Master of Public Health degree from the University of Minnesota in 2007. She also holds a Bachelor of Arts degree in Psychology from St. Catherine’s University and a professional certification in health education.
In her spare time, she likes to travel, bike, practice yoga, read, and bake. Despite her love for international travel and hopes to live abroad someday, Heather is a devout Minnesotan and even enjoys the winter…through a window, anyway.
Jake Mathias, Software Developer
Jake is a software developer with strengths in the .Net framework, including ASP.NET MVC, Windows Services, and WPF applications. He enjoys the process of designing and architecting clean solutions with an emphasis on data integrity.
Jake joined PDA in January of 2018 and is currently assisting in the continuing development and maintenance of a software system for Florida’s Department of Health.
He started his career as a Support Analyst, and worked his way into a developer position while also completing his degree. His previous development experience had a heavy focus on automated data management, including the transfer, mapping, and analysis of data.
Jake recently obtained his B.S. in Computer Science from Metropolitan State University.
Jake enjoys spending his free time attempting new recipes, strumming his acoustic guitar, and starting personal creative projects, such as designing video games and dabbling in 3D animation. He also loves staying active year-round, whether that be weight-lifting, jogging, or snowboarding during the winter.
We recently added a project manager to our team! Evan Shrader is an energetic project manager who excels at creating and enhancing structural efficiencies and promoting effective team strategies. Evan thrives on making projects easier for those he works with, on keeping a handle on complex priorities, and on pursuing projects that make life better for those impacted. Evan believes that getting any project to the end-zone requires a great team that can work well together while maintaining a positive and friendly atmosphere.
Evan will spend his time enhancing the current project management process at PDA, as well as managing the projects of PDA’s software development team. Evan is also involved in several of the Evaluations and Statistics projects at PDA.
Prior to joining PDA, Evan worked as an IT Project Manager at Provident Funding Associates, where he managed the proprietary cross-matrix software development projects of several key financial and business divisions. His background includes both Waterfall and Agile project management methodologies, helpdesk support, technical research and design, software user-testing, process innovation, budget analysis, and team management.
Outside of PDA, Evan loves spending time with his family. He mentors youth through his local church, and loves individualistic sports, including skiing, rock climbing, martial arts, and ice-skating. When the weather’s warm, he and his fiancé go camping as often as they can… although both being redheads, they often have to stay in the shade.
Written by Heather Zook
ClearWay MinnesotaSM awarded two-year grants to three health systems in Minnesota between 2014 and 2016. The purpose of these grants was to increase the health systems’ capacity to address tobacco use with patients. Each health system utilized multiple strategies to conduct systems change, and these strategies varied across grantees. Strategies included: utilizing a team-based approach; building system-level support; implementing tobacco screening, treatment, and referral protocols; modifying electronic health record (EHR) systems; training staff; and reporting and monitoring progress.
PDA conducted a process evaluation of the systems change grants. Evaluators held semi-structured, in-person interviews with key staff from each health system at three points in time (baseline, mid-point, and end of project). We also reviewed grantee documentation, including progress reports, meeting notes, and presentations. At the end of the two-year grant cycle, PDA developed a case study for each of the funded health systems (see links at end of post).
Include grantees in developing the final product
When developing a public-facing document for a health system, you will likely need to get final approval from multiple levels within the organization. This takes time, particularly if you include images such as an EHR screenshot or workflow that may be proprietary. It is important to include health system staff (in this case, the grant coordinator) in case study development to ensure that the wording accurately reflects the project activities. Along these lines, it is helpful to establish relationships with health system staff, especially when evaluating grant-funded projects, because staff may leave the organization once the grant ends.
Finalize all text in Word before moving to PowerPoint
This was PDA’s first time using PowerPoint to create case studies, and it was a great learning experience. We were inspired by Nancy Duarte’s Slidedoc, which is a document that falls somewhere in between a PowerPoint presentation and a report written in Word. First, we learned (the hard way) the importance of finalizing all text in a Word document before working in PowerPoint. This is especially key if you have multiple people reviewing and editing the text. Although you can add comments, PowerPoint does not currently have an easy method to track changes; therefore, it is difficult to see what edits other people have made. In addition, most edits to the text impact the formatting and spacing of other objects on the page. Thus, all formatting should be done after the text is finalized.
PowerPoint provides flexibility for designing case studies
On a positive note, we learned that PowerPoint is an excellent platform for designing case studies! It provides a lot of flexibility for adding images, shapes, and colors. PowerPoint makes it easy to add visual variation to each page, which is especially helpful when working with a lot of text. Even something as simple as a background image can draw in the reader and bring life to a text-heavy page.
Working in PowerPoint also encouraged us to focus the content of each page on a couple of key points, creating digestible chunks of information for our client. In Word, it’s easy for a section to bleed onto the next page or span across multiple pages. The format of PowerPoint helped us to focus our message and incorporate a visual to emphasize that point.
Links to case studies:
Hennepin County Medical Center
Mille Lacs Band of Ojibwe
We are participating in some upcoming conferences in the field of tobacco control!
The Society for Research on Nicotine and Tobacco’s annual meeting was in Florence, Italy March 8-11. Sadly, no PDA staff attended in-person, but we did co-author several posters.
The North American Quitline Consortium’s conference is March 20-21 in Austin, TX followed by the National Conference on Tobacco or Health, also in Austin, March 22-24. Several PDA staff will be in attendance at these two conferences. Austin isn’t Italy, but we’ll take it.
See below for a brief description of the posters and presentations that we are involved in. And say hi to Harlan, Julie, Becky, Amy, or Katherine if you see them there!
Society for Research on Nicotine and Tobacco
Expanding quitline reach: Minnesota’s and Oklahoma’s experience with a new service model
Barbara Schillo2, Laura Beebe3, Becky Lien1, Paula Keller2, Amy Kerr1, Paola Klein3, Stephen Gillaspy3, Randi Lachter2
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota; 3 University of Oklahoma Health Sciences Center
[Poster] Both Minnesota (in 2014) and Oklahoma (in 2015) changed cessation service models. The presenters share their experiences including marketing strategies. This research demonstrates that innovations in service delivery combined with statewide mass media campaigns have the potential to expand reach among tobacco users while achieving effective outcomes.
Offering choices engages more smokeless tobacco users in a quitline
Raymond Boyle2, Paula Keller2, Becky Lien1, Bruce Christiansen3
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota; 3 University of Wisconsin-Madison
[Poster] With the availability of NRT starter kits in 2014, ClearWay Minnesota saw an increase in the number and percent of smokeless tobacco users enrolling in QUITPLAN Services. This study uses Wisconsin quitline participants as a neighboring control to Minnesota and discusses the unintended impact of new services.
Strategic use of media modalities to maximize service volumes: Predictors of a statewide cessation program
Barbara Schillo2, Marietta Dreher2, Andrea Mowery2, Becky Lien1
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota
[Poster] This poster presents a study conducted to predict which media events were associated with an increase in inbound call volumes and web visits to QUITPLAN Services. Results demonstrate that a variety of media approaches work to achieve high levels of service volumes.
North American Quitline Consortium
Real-world application of reach ratios: A tool to monitor quitline reach among priority populations
Becky Lien1, Emily Subialka Nowariak1, Randi Lachter2
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota
[Presentation] The Reach Ratio is a simple metric that has not yet been widely adopted by quitlines. This presentation will provide practical information necessary to calculate the metric, and will encourage quitlines to consider assessing the reach of priority populations when they assess overall reach.
National Conference on Tobacco or Health
Treating tobacco dependence through systems change: A process evaluation of three Minnesota health systems
Megan Whittet2, Traci Capesius1, Heather Zook1, Paula Keller2
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota
[Poster] The purpose of this poster is to add to the evidence base regarding how health systems can make changes to routinely address and treat tobacco use dependence within their patient populations.
Maximizing scarce promotional dollars: Strategic analysis of media tactics to increase service volumes
Marietta Dreher2, Becky Lien1, Barbara Schillo2, Andrea Mowery2
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota
[Presentation] This presentation will describe a methodology that identifies the impact different media tactics have on service volumes and how they work to increase service volumes. Analyzing the effectiveness of media tactics can help managers maximize scarce promotional dollars to increase cessation services volumes.
Linking low-SES smokers to cessation services: Key strategies, successes, and lessons learned
Amy Kerr1, Carole Specktor2, Heather Zook1, Emma Ramsbottom Wagner1, Paula Keller2, Lija Greenseid3
1 Professional Data Analysts, Inc.; 2 ClearWay Minnesota; 3 Greenseid Consulting
[Poster] While overall tobacco use rates continue to drop, higher rates persist among people with low-socioeconomic status (SES). ClearWay Minnesota funded 13 Minnesota organizations that serve low-SES tobacco users to connect their clients to existing tobacco cessation services and build organizational capacity to do so. Through semi-structured interviews with key informants from grantee organizations and reviews of grantee reports, the evaluation explored key strategies to connect clients to cessation services, along with successes, challenges and reach. This poster will describe key findings, which can be used by tobacco control practitioners, community-based organizations, and evaluators as they determine the best methods to engage traditionally hard-to-reach populations.
A collaborative effort to implement a statewide bi-directional eReferral system: Practical experience from Tobacco Free Florida
Julie Rainey1, Jane Parker2, Colin McDonough3, Harlan Luxenberg1
1 Professional Data Analysts, Inc.; 2 Bureau of Tobacco Free Florida; 3 Optum
[Presentation] The presentation will discuss how the development of eReferral systems fit into the Bureau of Tobacco Free Florida’s strategic plans, the quitline vendor’s role in the implementation phase, elements of the bi-directional data exchange, how the evaluator extended the quitline referral system for use by the in-person cessation programs, and share evaluation results including the rate of conversion from referral to program enrollment.
Building a web-based application for managing, monitoring, and evaluating a multi-site tobacco cessation program in Florida
Harlan Luxenberg1, Julie Rainey1, Marilyn Leeds3, Jane Parker2
1 Professional Data Analysts, Inc.; 2 Bureau of Tobacco Free Florida; 3 Florida State University
[Poster] The Bureau of Tobacco Free Florida (BTFF) oversees a multi-site tobacco cessation program which provides face-to-face group counseling to over 20,000 tobacco users annually through Florida Area Health and Education Centers (AHECs). In 2014, BTFF asked the evaluator to design and develop a custom web-based program that would improve BTFF’s understanding of how funds were being spent, the programs’ ability to make data-based programmatic decisions, and the evaluators’ ability to provide timely reports. This poster will share how TFFACT has supported the standardization of the AHEC program; improved program operations, reach and efficiency; improved data quality; access to more data for the evaluation and BTFF.
Meeting tobacco users where they are: How Florida built a statewide referral system within employment centers to connect the unemployed with cessation services
Katherine Rehorst1, Elisia Norton2, Anne Betzner1, Julie Rainey1, Jane Parker2, Marilyn Leeds3
1 Professional Data Analysts, Inc.; 2 Bureau of Tobacco Free Florida; 3 Florida State University
[Poster] Florida institutionalized a fax referral process within CareerSource Florida to identify unemployed tobacco users and refer them to phone, online, and in-person cessation services. This session describes the development of the statewide CareerSource referral system, and shares lessons learned from the successes and challenges experienced during implementation. Results from a process and outcome evaluation document the number of CareerSource -generated referrals and program enrollments providing preliminary evidence that non-medical referral sources like CareerSource are competitive with traditional medical referral programs, and have the potential to reach under-served groups and improve access to services.
PDA is currently seeking a part-time program evaluation intern for the summer of 2017. It is a 10-week paid internship. We are looking for an upper level undergraduate student or masters-level graduate student who is interested in learning more about and getting more experience in the field of evaluation. For more information about this internship and how to apply, click here.
In November, Lara Rubinyi joined the PDA team as an Associate Evaluator. Lara received her Master of International Public Affairs degree from the University of Wisconsin-Madison. Before joining PDA, Lara worked with [rp] Collective in Kathmandu, Nepal to build healthy, community-focused urban plans. She has experience conducting large-scale research and evaluations on STEM education, child protection, dental services and opioid use, vector-borne disease, climate change, and migration. The common theme of her work has been creating healthier communities through evidence-based practices. She especially loves using both qualitative and quantitative methods to discover what is working and how to make programs work better.
At PDA, Lara will work on several projects including evaluations for the Florida Department of Health’s in-person cessation program and the Hawaii Department of Health’s community grant program.
In her free time, Lara enjoys camping, traveling, learning new instruments, and writing. Her new goal is to visit as many state parks in Minnesota as possible and maybe write a song about it.
Emma Ramsbottom Wagner, Associate Evaluator
In May, Emma joined the PDA team as an Associate Evaluator. Emma just received her Master of Arts degree in Evaluation Studies from the University of Minnesota. Prior to joining PDA, Emma served as an AmeriCorps VISTA member at the Hispanic Committee of Virginia, spent several years coordinating and developing non-profit programs in the Seattle area, and most recently carried out evaluation projects with Planned Parenthood of MN|ND|SD and Hamline University.
At PDA, Emma will primarily work on evaluating ClearWay MN’s community engagement grantee programs and processes and the Hawaii Department of Health’s statewide tobacco quitline and in-person tobacco cessation community grants program.
Emma is drawn to the facilitative aspects of evaluation work and enjoys collaborating with stakeholders to effectively surface, organize, and respond to their evolving ideas and needs. She is also enthusiastic about data visualization, and loves incorporating creative elements into her communications and reports. In her free time, Emma is an increasingly adventurous baker, pop culture nut, and very uncompetitive runner.
Colin Summers, Developer
Colin joined PDA in May as a developer. The main project he is working on is the construction and design of a large software system for the Florida Department of Health for tracking an in-person tobacco cessation program. His current technology focuses are .NET MVC, AngularJS and Microsoft SQL Server.
Colin received his Bachelor’s Degree in Computer Science from the University of Wisconsin – Parkside in 2013. He then moved to the twin cities to work as a Developer at Magenic where he gained experience with a variety of technologies including .NET MVC, Microsoft BizTalk and MongoDB. There he was recognized as Consultant of the Quarter for Q4 2015 for demonstrating excellent professional communication and technical skills and his ability for quickly mastering new technologies.
Outside of work, Colin enjoys spending time with his family and two dogs. He is busy getting ready for the second addition to the family, another son, expected this October. In his free time, he enjoys cruising in his Subaru WRX, fixing up his Fiat Spider, and playing video games.
Margaret Merrill, Stats Intern
In early June we welcomed Margaret, who has been serving as our Stats Intern this summer. Margaret just received her Bachelor of Arts degree in Economics and Statistics from St. Olaf College.
At PDA, Margaret will help with the analysis of data having to do with tobacco cessation and other public health programs. Margaret is interested in data visualization and is excited to spend her summer at PDA. Outside of work Margaret enjoys playing tennis and reading.