Why You Should Take Surveys (and What It Takes to Make One): Behind the Survey Scenes with a Social Scientist
By Brook McWherter
If you are like me, you are probably tired of being bothered by surveys – they are everywhere! There are surveys at the bottom of receipts, from people on the sidewalk, in the mail, and everywhere online. We are surrounded by surveys!
As a natural resource social scientist, I get a look behind the scenes of survey development and testing. You may be surprised to know that it is actually a lot of work. I am going to take you on a journey to Bolivia, to the first stage of my PhD research where I learned what it takes to get survey questions on your screen (or in your hand).
You may be asking yourself, why did she go all the way to Bolivia?
My answer is in two parts. First, we were trying to understand individuals’ and communities’ choices in and perceptions of a water conservation project in Bolivia. Second, because whom we decide to survey will impact the types of questions we can ask and how we ask them, testing in the community is vital to developing an effective survey.
With this objective in mind, we began our work in the town of Vallegrande (translated as large valley). Located in the southern part of Bolivia and nestled in the mountains, Vallegrande is a small and beautiful town filled with a diverse group of people who live both in the valley and further up into the mountains. It is surrounded by farmland, forest, ranch land. And, while it may have been summer in Indiana, when we traveled, in Bolivia it was winter, which meant many chilly 30₀F nights and with no heat!
Which leads us to our objective for the “summer”: to test 42 pages of questions with people who are similar to our study population to see how it is received in the local context, and if possible, make it a more reasonable length. For those of us in the social sciences this process is called piloting.
So, what does it take to pilot test a survey in the field?
To begin with, we had to assemble our team, as this was a large collaborative process. Our team included:
- My fellow PhD students from Purdue University and Utah State,
- Our professors who visited us throughout the process to provide guidance on piloting and communicated with us remotely and,
- Our local collaborator Natura Foundation Bolivia who provided us with technical assistance, assisted with transportation, and overall helped us in finding people to interview as well as making us feel welcome.
What did this look like on a daily basis?
Every day for 2.5 months, we would get up at 7am, get our breakfast, walk down to the market, buy lunch (always the same chicken with potatoes or rice), grab bottles of water, and then be ready to go by 8am when our Natura driver would pick us up. Oh! We also had to print off our survey sections for everyone. Then we would drive for as long as 2 hours away down a bumpy dirt road to a small community where we would go door to door asking if people had the time to talk to us.
When we arrived we would go house to house and strike up a conversation with local landowners to see if they would be willing to talk with us and take our survey.
If they agreed, we would ask them one of the questions from our survey. For each question we would observe:
- Did they understand the wording of the question?
- If we reworded the question in the moment, did it work?
- If we had to explain the question, what type of explanation went best?
- How long did it take us to ask the questions in each section?
During the interviews, we would listen and take notes on our observations and suggested changes. Then each night we would return to Vallegrande, grab a quick dinner at the market, and head to the hotel to tediously go over all our notes as a group. We would even call the rest of the team members scattered across the United States to ensure that our edits were not cutting out critical components to the survey.
This process was repeated throughout the day, week-by-week and month-by-month, often into the night reviewing everything…all in Spanish! Little by little, we began to make progress and our survey started to look trimmed and tidied up.
Along the way we met some amazing people who were kind enough to give us 30-60 minutes of their very busy days to talk with us, show us around, and more often than not, offer us food or drinks!
The last part of the “summer” consisted of us testing the entire survey, and again looking for clarity and making sure the length was acceptable. At this point, we had started dreaming about the survey and were editing it in our sleep! Eventually we had a solid survey that required only minor edits before being implemented in the field. Then, it was time to say goodbye to Bolivia to travel to Purdue University and start classes.
On a personal note, although we worked well into the late hours of the night, it was an amazing experience! We grew together as a team, learned valuable lessons about developing and pilot testing surveys, and were able to create strong relationships with Natura, who we will continue to work with this coming summer.
There you have it – now you know what pilot testing a survey is really like! It is long hours of rigorous note taking and discussion, amazing conversations, and if you are lucky, you can do it all in a beautiful location.
If I can leave you with one last thing, next time you see a survey on your screen or in the mail, think about what happened to get it to that point, and if you have time – take it!
