This is the final post of a series of three that discusses our experience using tablet computers for data collection. In the first post we talked about why we chose to use them and in the second post we described how we used them. In this post we discuss the benefits and challenges of using them, what it cost, and what you should consider for your own projects.
The Benefits of Using an Electronic Device for Data Collection
To date we have used over 20 tablet computers for two evaluations, with a total of 17 organizations. Lots of great things have come out of it:
- We saved a bunch of paper in the first year and a half – up to 7,000 sheets! By the end of the evaluation we will have saved up to five trees, which certainly isn’t the size of a forest, but it’s a start. Replacing paper forms with electronic data collection is a greener solution, which is important to PDA as well as the sites we work with.
- The data collection process is more efficient because we no longer have to coordinate printing and sending forms and data don’t need to be sent, scanned, or entered. This in turn allows the sites and us to monitor their data in real-time.
- There are fewer security risks associated with entering data into an encrypted portal versus storing and transferring paper surveys across multiple sites. We have had no issues with lost or stolen data under the new system. As we said earlier, this was an important factor given the type of sensitive data we were working with.
- The data are cleaner. Following skip pattern instructions on paper forms was confusing for some sites and resulted in missing responses and data entry errors; however, having programmed skip patterns on the iPads made data entry much easier and resulted in better data integrity. So there’s certainly a benefit of electronic forms and being able to program question skips. There’s also the option of requiring answers on an electronic form. We don’t require an answer for every question on our forms, but there are a few key pieces of data that we set as required, which helps to get complete data.
- There is increased buy-in and use of the evaluation. For one, there is likely a novelty-effect of providing a tablet computer for data collection even with its restricted access. And with the portal, we were able to add some features that were helpful for the sites, like being able to monitor their data in real time and printing completed forms from the portal. Additionally, we think it may have in some ways further legitimized the evaluation as it shows the sites that the funder cares enough about the programs and their outcomes to invest in equipment. And for the sites involved with the first cycle when we used paper, it demonstrated that we responded to their feedback.
The Challenges of Using an Electronic System
One size does not fit all. We surveyed the most recent round of funded sites and only half of them reported using their iPad. Those who don’t use the iPad provided several reasons for not using it. Several sites responded that it worked better for how their intervention was set up to complete the surveys on paper and then enter the data into the portal later on. Another reason reported was that they simply found it easier or were more comfortable using paper or their desktop computers for data collection – they prefer a larger screen and keyboard. Finally, some sites are located in a more rural area and experienced a slower 3G connection on the iPad making the portal too slow to use.
We wondered if additional training would increase use of the iPad, but those sites who felt it was not easy to use also indicated they were not interested in additional training. We were kind of bummed to learn that only half of the grantees were using their iPad, but we were thrilled to see that the majority of the returning grantees prefer the new electronic data collection process and feel it is more efficient. So while we certainly wish for everyone to use the iPad or at least a computer, we understand that one size does not fit all and were glad that the evaluation could accommodate the various intervention settings.
Data management issues remain. While enforcing skip patterns and required responses are a clear benefit to an electronic system, there are other challenges that come with it. For example, it’s easier to misspell something when you’re typing, and we rely on the spellings of the participant’s name and address to find them in the system and add program data as well as to send a follow-up survey. The wrong name or address might mean we aren’t able to get a hold of them. We don’t believe this is a frequent issue, but it remains a challenge.
Another issue is the occurrence of the same form being entered twice. This has occurred when some sites record data on paper forms and then later data-enter the forms into the portal. Despite some data validation checks built into the system, sites sometimes inadvertently duplicate a record and then continue to update the duplicated record. This type of error can take a lot of time to clean up, so we ended up adding a data management feature to our electronic system that streamlined the data cleaning process (after spending many hours manually cleaning and merging data).
Another minor issue that sites have reported happening is the tablet or portal timing out during inactivity. Someone may be in the middle of completing a form, stop and have a lengthy discussion about something with a participant, and then get kicked out of the tablet or system, and have to start all over. This can result in blank and/or duplicated records. However, it’s important for us to put a time limit on how long a tablet and the website can be inactive before requiring a password to resume activity due to HIPAA-related concerns (i.e., in case an iPad is misplaced). If we ever build another portal, we would spend more time thinking about this limitation and brainstorming better ways to handle the timeout (i.e. ways to access forms that were started but not completed, which our system currently does not allow).
The costs can be divided into material costs and labor costs.
Tablet with 3G Enabled = $500 per iPad
Protective case = $25 per iPad
AppleCare = $90 per iPad
Data plan = $200/year per iPad
Labor costs. Labor costs vary greatly from project to project depending on how many iPads you’re dealing with, the technological capacity of your clients, the complexity of your survey, whether you create your own secure website for data entry versus just using an app or SurveyMonkey, etc. Since hourly rates may differ from our project to yours, I’m going to present the labor costs in terms of amount of staff time for our project.
First there’s the time we spent researching which tablet to purchase, figuring out how many each site needed, setting up the iPad agreements with each site, configuring the data plans with AT&T, figuring out what the heck a configuration profile was and how to make it work for us, and setting up each iPad with the configuration profile, restrictions, etc. That amounted to about a month’s worth of work time for one individual, which is quite a bit of time. Certainly if we did it again, it wouldn’t take nearly as much time.
Then there was the web programming. Programmers are not cheap, and the portal we created was not a simple task. We think it was totally worth it, but it also required about a month’s worth of work time (however this included design, development, implementation of the site, and other functionality besides just collecting data such as data management and administrative reporting). The hours will vary based on your specifications and needs. Using a custom built site allowed us to add functionality as we needed it, such as enhanced data management functions and more elaborate reporting.
As I mentioned previously, we still encounter some ongoing data quality issues but have made many changes to our data collection procedures to reduce these dramatically. We review the data on a quarterly basis and our technical assistance provider (see below) works with each site to address any data quality issues. Your data management needs may vary from ours, but don’t expect your data quality issues to be resolved because you moved from a paper to an electronic system.
The final labor cost is time spent providing technical assistance. The amount of time you spend on technical assistance could of course vary by how many iPads you have in the field and what people’s level of comfort and familiarity with the iPad is. In our first year this only amounted to about an hour per site in the first year. We were mostly troubleshooting iPad issues with staff and training any new staff.
Similar to the material costs, much of the labor costs were just a one-time gig (the iPad setup and web programming). The ongoing technical assistance is the only labor cost that continues throughout the project.
There are several key questions to ask yourself if you are thinking about using an electronic device for data collection:
- Users. Will the people who are going to be collecting data feel comfortable using the device (with some training)?
- Setting. Will the data be collected in a setting that is conducive to using an electronic device? Will the setting have access to Wi-Fi or a 3G/4G signal?
- Project Length. Is this a multi-year project? Is there a chance you’ll be doing this project again in the future?
- Survey/Project Complexity. Is your survey long? Does it have complex skip patterns? Do you need to monitor the data in real-time? Do you need a way to better manage the quality of your data?
- Cost. Is there budget for the needed material costs? Are staff available for planning, programming, ongoing data management, making updates, etc.?
- Security. Will you be storing sensitive data on the tablet itself? Will the tablets be safe in storage and when sites are using them?
- Technological/Data Capacity. Do you have the technological expertise to create and manage a data collection system that meets your needs?
- Technical Assistance. Are staff available to provide ongoing training and technical assistance to those collecting data?
For our evaluation, we decided that iPads would work best in the site settings. We knew this was a multi-year project with relatively complex data collection needs and the costs for startup were high but greatly decreased after that. Our company had the technology experience and infrastructure to build and support a portal, and we could contract with someone to provide technical assistance. Overall, we found that using tablet computers was more efficient, more environmentally friendly, and more secure than using paper forms; the use of tablets also improved data quality and increased evaluation buy-in.
I definitely encourage you to consider whether using an electronic device for data collection would alleviate any issues you’re experiencing with your own projects and surveys. Research the costs associated with that device and consider the feasibility of actually using it in the field. There are probably some pros and cons to both sides.
If you’re curious to hear more about our experience with this project, Angie Ficek is presenting this information at the next Twin Cities Research Group brown bag August 13 at noon at Wilder Research in St. Paul, MN. For more information and to RSVP to this event, go to: http://tcrgAUG13.eventbrite.com. And of course, feel reach out to us if you’d like any help planning your data collection efforts!