St. Anthony Main 219 Main Street SE, Suite 302 Minneapolis, Minnesota 55414 612.623.9110
(f) 612.623.8807 |
By
PDA_News
|
Published
November 14, 2011
PDA has been using LimeSurvey’s template design feature to improve web surveys. Specifically, this feature allows us to create web surveys using associated design – matching the survey design with the brand of the organization associated with the evaluation (the evaluand). Our experience in survey design has shown that survey participants are much more likely to complete a survey about an organization if they can visually associate the survey with the evaluand. The LimeSurvey template design feature allows PDA to create surveys that include recognizable color schemes, banners, logos and browser tab icons to help survey participants associate the survey with the program being evaluated.
LimeSurvey is a free, open source web survey software suite used by PDA for our web survey needs. For more information about LimeSurvey, visit their website, www.limesurvey.com, or watch for future posts about how PDA has been using LimeSurvey.
By
PDA_News
|
Published
November 4, 2011
On Saturday, November 5, a member of PDA’s staff will be presenting at the American Evaluation Association conference in Anaheim, CA. Dr. Melissa Chapman Haynes will present on how the evaluation team created logic models in collaboration with project staff in a community-based tobacco cessation program. Logic models are one key tool evaluators have to engage stakeholders and project leaders in determining values, and how those values relate to project activities and intended results. PDA specifically used logic models as a tool to better understand the broader context and scope of this project’s tobacco-cessation work, including in-person cessation classes, referrals to the tobacco Quitline, community partnerships, and training of health professionals related to tobacco cessation. PDA used the DoView program to create, share, and revise logic models with program stakeholders (http://doview.com/). The authors of this presentation are Dr. Melissa Chapman Haynes, Dr. Lija Greenseid, and Julie Rainey.
Also on Saturday, Dr. Lija Greenseid will give a demonstration on minimizing total survey error. Surveys are common tools for gathering evaluation information; however, they are only as useful as the quality of the data that are collected. This demonstration will provide a multi-faceted framework for thinking about survey quality that includes both the accuracy of the data and other important considerations such as usability, timeliness, and credibility. Dr. Greenseid will discuss advances in survey questionnaire design, sampling, and administration and share examples from her experience in evaluating tobacco control programs.
By
PDA_News
|
Published
October 31, 2011
At PDA we have begun using LimeSurvey for many of our online surveying needs. LimeSurvey is a free, open source web survey software suite that has an ever growing feature list and has allowed us to create more elaborate, complex surveys than what we were able to do with other paid online survey subscriptions. We are excited about the flexibility of LimeSurvey, as this provides us with expanded options to collect tailored data.
For more information about LimeSurvey, visit their website, www.limesurvey.com, or watch for updates about how PDA has been using LimeSurvey on future posts.
By
PDA_News
|
Published
October 24, 2011
The old joke goes … “A mathematician will add 2 plus 2 and get exactly 4. A physicist will say 2 plus 2 is approximately but not certainly 4. But a statistician will smile and say that 2 plus 2 is whatever you want it to be …”
Fortunately, the field of tobacco cessation and organizations like the North American Quitline Consortium (NAQC) are building agreement on methods for measuring quit rates. However, agreement on other elements of study design is less developed. A prime example of this is sampling strategy, or the process of systematically selecting a subset of participants to be included in a study. Program staff should examine the sampling strategies used in outcome studies to better understand what the resulting rates mean and how to use them.
Selecting participants by program intensity
Sampling criteria that are often considered include enrollment date, demographics (gender, race, etc.), and program-level variables (site, counselor, etc.). Program intensity is a sampling criterion that is often overlooked.
If a quitline offers several different interventions, you must decide which will be included in the sample. If a quitline offers an intensive multi-call program with free stop-smoking medications, as well as a single 10-minute session to answer questions about quitting, should both be included in the sample? This depends on the purpose and scope of the study. However, it is important to note that including participants enrolled in the brief intervention would result in a more conservative quit rate because evidence suggests that those who receive less intervention are less likely to quit.
A second facet of program intensity is the degree to which a participant completes the intervention. Some participants don’t complete intake before leaving the program or never speak with a counselor. Likewise, including these participants would result in a more conservative quit rate because they are less likely to have quit. On the other end of the spectrum, including only participants who completed two counseling sessions, for example, would result in more liberal or optimistic quit rates.
Choosing who to include in quit rates
Once the sample has been drawn and data is collected, program staff must decide who to include in the quit rates. A quit rate can include either the entire sample or only those who responded to the survey. Unfortunately, many participants who initially consent to take part in evaluations are later unreachable or refuse to participate when called for follow-up. Whether or not these participants are included in quit rate calculations can greatly influence the estimated quit rate and determine how conservative it is.
With the intention to treat rate, everyone in the survey sample is included in the quit rate, even those who did not respond to the survey. This is because the program intended to treat everyone, regardless of response status. However, no information is available on non-responders’ smoking status at follow-up. But because several studies
suggest that non-responders are more likely than responders to be using tobacco at follow-up, the intention to treat rate assumes all non-responders are still smoking. Because of this conservative assumption, this rate is considered to be the lower bound on the “true” quit rate.
A completer rate includes only those who completed the follow-up survey because this is the only group that reported their smoking status. Unfortunately, those who complete surveys are more likely to have quit than those who do not, so this rate may be somewhat inflated.
Best to report a range
Neither rate is “better” than the other. Calculating both completer and intention to treat rates provides an upper and lower bound on the “true” quit rate and provides a more complete picture of a cessation program’s success.
By
PDA_News
|
Published
October 17, 2011
Obtain accurate results and reduce costs: Put your confidence in statistical techniques
Social service providers are often caught in a bind. Their primary goal is to provide the highest quality services to the greatest number of participants. Many want to evaluate their programs to ensure that their clients’ satisfaction is high and to assess their outcomes. But financially it is difficult or impossible to reach all program participants for evaluation. One way that organizations can cut costs is to evaluate programming using a sample, or subset, of participants.
While sampling saves money, is it good practice? If used correctly, sampling participants and reporting results using confidence intervals contributes to a robust evaluation design. For example, a tobacco cessation organization may want to know the quit rate among its program participants. But staff might ask: how confident can we be that our quit rates would be the same if we surveyed all of our participants versus just a sample of them? And, how do I know that our sample will yield a quit rate that closely approximates the “true” quit rate?
First, the bad news is that each sample will likely give a slightly different result, and in a very small number of cases, you might see an extreme result much higher or lower than the “true” quit rate. The good news is that confidence intervals, a statistical calculation, allows you to be 95% confident that the quit rate estimate from your sample is an accurate estimate for all program participants within some range, even if the rate can’t be narrowed down to a single figure.
For example, a quit rate of 25% may have a confidence interval of plus or minus 5%. This means that you can be 95% confident the quit rate will fall between 20% and 30% (25% -5% = 20% and 25% + 5% = 30%). To say it in another way, in 95 out of 100 samples in which the same number of participants is randomly draw from a list of eligible participants, the quit rate would fall between 20% and 30%. This range in rates takes into consideration errors due to chance, that is, fluctuations in drawing one sample randomly from a large pool of participants.
Researchers and practitioners often overlook this essential component of confidence intervals—that the interval is only accurate if a sample is drawn randomly. Some examples of non-random samples where confidence intervals are not accurate include sampling participants who enroll during a promotional period or participants who enroll between 10am and 1pm. Errors created by these non-random sampling methods cannot be accounted for statistically and should, therefore, be avoided whenever possible.
However, while randomness is a priority, confidence intervals depend largely on sample size. The general rule being: the larger the sample size, the smaller the confidence interval. For example, a sample size of 100 results in a confidence interval of 10%, while a sample size of 1,000 results in a 3% confidence interval.
The implications of sampling and confidence intervals are substantial, as this example illustrates. Consider a telephone quitline that serves 2,000 people annually. An exhaustive survey could result in 2,000 completed surveys, while a sampled survey could result in 400 completed surveys. You would have 100% confidence in the rate produced by the exhaustive survey, and 95% confidence that the quit rate for the sample surveyed would be plus or minus 5%. Assuming the cost for each completed survey is $25, the survey cost for the exhaustive survey would be $50,000, as compared to the $10,000 cost of the sampled survey. Sampling produces a substantial cost savings in exchange for 95% confidence that the true rate lies within a range of quit rate values.
So, one way to make the most of evaluation dollars is to select a sample size that is within your evaluation budget, and draw a random sample of participants to take part in the evaluation. Quit rates should always be reported with the confidence intervals. For more information on how sampling and confidence intervals may be used in your program, please contact PDA.
By
PDA_News
|
Published
October 10, 2011
Sticky situations – Short of a crystal ball, evaluators cannot possibly anticipate every conflict that may occur during the course of an evaluation (or after). The Program Evaluation Standards can help evaluators and evaluation users navigate these situations by reflecting on:
1. What were the factors leading up to and relating to the conflict? Review the standards to be comprehensive and explicit in your review. Possible guidance from the Standards might include:
- Expanded stakeholder input or collaboration
- Issues of propriety, include formal agreements, conflict of interest, or transparency and disclosure
- A need to re-examine and possibly re-negotiate the purposes and priorities of the evaluation, or the project.
2. Given the context and the evaluator’s role and relationships with the stakeholders involved, which of those identified factors should be a priority to address? Which factors are feasible for the evaluator to address?, and
3. Given the context, negotiated purposes, and proprietary factors, should you intervene? How might you, as an evaluator, intervene? As a program leader? As a program participant or stakeholder?
When and How you Might Use the Program Evaluation Standards: Evaluation Planning
Evaluation planning – In preparation of meetings between evaluators and project leaders, both parties can reference the PES to form questions to elicit program theories (formal or informal – how and why do they think a program will work?), program context, identification of stakeholder groups and roles, a client’s understanding of the range of goals and procedures involved in evaluations, and the like. Some examples of questions using the standards as a guide might be:
- If this project were to be implemented exactly as you planned, what would a typical participant experience?
- Can you tell me who three or four key players are in Project X? To what extent are they involved in the project planning/ implementation/ data collection/ etc.?
- What, if any, barriers to you see in implementing Project X?
- What type of information has already been collected about your program (in the specific context(s) for the population(s) of interest? What do you already know?
- What type of information do you most need from the evaluation? What questions would you like the evaluation to address? What other questions do you anticipate Stakeholder x, y, z might want addressed?
By
PDA_News
|
Published
September 26, 2011
Improving Evaluation Use – Evaluation use starts from the initial conversations and interactions between a client and evaluator. It continues through the actual writing and dissemination of a report, and even beyond. Evaluation use might involve the following standards (at least):
Utility
- Evaluator Credibility – Have you established this with the stakeholder groups you intend to inform via the evaluation results?
- Attention to Stakeholders – To what extent have stakeholders been identified and involved? If you expect them to use your evaluation results, they need to be part of the evaluation process.
- Timely and Appropriate Communication and Reporting – who is your audience and what is the information need for each audience?
Feasibility
- Project Management – this incorporates two of the other standards – resource use and practical procedures.
- Contextual Viability – after identifying the needs and interests of key stakeholder groups, the key here is balancing those needs.
Propriety
- Responsive and inclusive orientation
- Human rights and respect
- Transparency and disclosure
Accuracy – nearly all of the accuracy standards are relevant to evaluation reporting, but these two are key:
- Justified Conclusions and Decisions
- Explicit Evaluation Reasoning
By
MChapman
|
Published
September 19, 2011
Evaluation demonstrates the importance of attending multiple counseling sessions.
QUITPLAN Centers® provide face to face counseling in clinic or hospital settings. The evaluation revealed:
- Most participants attend one or two sessions even though more are available
- Participants who attend more sessions are more likely to quit
Counselors begin sharing these findings with participants to improve retention and to encourage additional appointments
By
PDA_News
|
Published
September 12, 2011
PDA has joined the fight against climate change by offsetting our energy usage through NativeEnergy‘s unique Help Build carbon offsets program. By purchasing these offsets, our company is helping a specific carbon reduction project, the Wewoka Biogas Project. This project will trap and destroy polluting methane gas that otherwise would be emitted from a landfill and will also reduce the unsustainable use of fossil fuel.
Through NativeEnergy’s Help Build program, PDA is keeping a total of 63 tons of CO2 out of the air every month! PDA is proud to be a carbon neutral company, which helps further our commitment to protecting our environment and communities.
|
|
