Abstract

Regular data collection with populations on the move is challenging for practitioners and researchers alike. With the support of IPA Colombia, the Immigration Policy Lab and Mercy Corps built and used a novel technology-assisted partially automated WhatsApp survey to interview Venezuelan migrants in Colombia and assess key indicators such as food security and well-being in a global pandemic. Researchers then conducted a randomized evaluation to assess the response rates of various low-cost data collection tools including WhatsApp, short message service (SMS), and Integrated Voice Response (IVR). WhatsApp surveys had the highest response rates compared to SMS and IVR, due to higher initial engagement and higher survey completion rates.

Policy Issue 

WhatsApp surveys are a relatively recent technology, and little is known about the empirical viability of this research method.[1] Traditional survey methods present limitations for researchers and organizations. This highlights the importance of timely and low-cost data collection tools for program design and adaptation. In-person and phone-based surveys can be time-intensive and costly to implement. It can also be challenging to retain respondents in a panel and stay in touch with them when there are changes in contact information or location.[2]

Researchers have used automated and semi-automated short message service (SMS) on mobile phones as a cheaper alternative to both collect survey data[3] as well as conduct randomized evaluations.[4] This research provides insight on testing WhatsApp as a low-cost, timely, and scalable data collection tool for program evaluation and tracking participant outcomes. Furthermore, its proof of concept is relevant to other contexts and research studies with mobile populations.

Context of the Evaluation 

In recent years, Venezuelan migrants have been displaced from their homes, with a majority of cross-border migrants arriving in Colombia. As of 2022, more than 2.5 million Venezuelans are estimated to be living in Colombia.[5] To help vulnerable Venezuelan migrants satisfy their basic needs, the International Rescue Committee, Mercy Corps, Save the Children, and World Vision implemented a cash assistance program called VenEsperanza.

Researchers from Mercy Corps and Immigration Policy Lab (IPL) were conducting an impact evaluation of the VenEsperanza program when the COVID-19 pandemic hit. Due to the pandemic’s impact, the number of vulnerable households increased significantly, making it impossible to draw conclusions about the impact of the VenEsperanza program. However, the researchers - with technical support from IPA Colombia - decided to use the evaluation to investigate whether new communication technologies can be used to conduct rigorous research among highly mobile populations.

Details of the Intervention 

Innovations for Poverty Action only provided technical support for this study.

The Immigration Policy Lab and Mercy Corps conducted a survey via WhatsApp to assess migrants’ wellbeing over a nine-month period. WhatsApp is the most widely used messaging application in Colombia and other countries in the region. Individuals who completed Mercy Corps’ vulnerability assessment for the VenEsperanza program from March to October 2020 were given the opportunity to opt-in to follow up surveys and continued research. For individuals who registered from March to May 2020, IPL and Mercy Corps sent three surveys at three-month intervals on WhatsApp to assess migrants’ regularization status, housing and food security, education access, psychosocial wellbeing, and integration. These surveys proved the feasibility of using this platform for recurring data collection.

Then, researchers conducted a randomized evaluation to assess the response rates of various low-cost data collection tools to compare WhatsApp surveys with those delivered via SMS or Integrated Voice Response (IVR). A different group of individuals who had completed the VenEsperanza screening and opted-in for additional research from July to October 2020 were randomly assigned into three groups to receive a single survey in July 2021. This 25-question survey included questions on employment, access to financial tools, and willingness to receive a COVID-19 vaccine, among others.

• WhatsApp group: Migrants in this group received an automated survey through WhatsApp.
• Text message group: Migrants in this group received the same survey through traditional text messages (SMS) on their mobile phones.
• Interactive Voice Response (IVR) group: Migrants in this group received a phone call built with IVR technology to answer the same survey by listening to pre-recorded questions and answering on the phone keypad.

For each completed survey, respondents received COP $5,000 pesos (USD $1.43) in phone credit. Innovations for Poverty Action provided technical support to: redesign the evaluation after the COVID-19 pandemic hit, support the development of WhatsApp surveys, conduct quality checks, support the drafting of a technical manual, and help with payments to participants.

Results and Policy Lessons 

WhatsApp surveys had the highest response rates compared to SMS and IVR, as well as the highest start rate and survey completion rate.

Response rate breakdown: The response rate for the WhatsApp surveys was 55 percent, 12 percentage points higher than the response rate for IVR and 27 percentage points higher than the response rate for SMS.

Sixty percent of WhatsApp survey respondents started the survey, compared to 55 percent for IVR respondents and 33 percent for SMS respondents. In addition, 92 percent of WhatsApp respondents who started the survey ended up completing it, whereas 77 percent of IVR respondents and 82 percent of SMS respondents completed the survey. For more details, please see the working paper here.

Demographics of respondents: The gender, income, region, and household size of WhatsApp and IVR respondents were overall representative of the migrant population observed at the start of the intervention. A majority of the SMS respondents were women.

This project offers a proof of concept that WhatsApp is a viable survey platform and a low-cost alternative to other data collection methods. One goal of this project was to create publicly available resources for other practitioners and researchers to replicate the low-cost, automated WhatsApp survey method. The research team developed a Technical Documentation and a How To Manual which has since been used by many additional research teams worldwide to replicate the survey approach. All the resources - including the manual, demo videos, and reusable code - can be found here.

Sources

[1] de Gruchy, Thea, Jo Vearey, Calvin Opiti, Langelihle Mlotshwa, Karima Manji, and Johanna Hanefeld. "Research on the move: exploring WhatsApp as a tool for understanding the intersections between migration, mobility, health and gender in South Africa." Globalization and health 17, no. 1 (2021): 1-13.

Ndashimye, Felix, Oumarou Hebie, and Jasper Tjaden. "Effectiveness of WhatsApp for measuring migration in follow-up phone surveys-Lessons from a mode experiment in two low-income countries during COVID contact restrictions." (2021).

[2] Fei, Jennifer, Jessica Wolff, Michael Hotard, Hannah Ingham, Saurabh Khanna, Duncan Lawrence, Beza Tesfaye, Jeremy M. Weinstein, Vasil I. Yasenov, and Jens Hainmueller. "Automated Chat Application Surveys Using Whatsapp: Evidence from Panel Surveys and a Mode Experiment." (2022).

[3] Tomlinson, Mark, Wesley Solomon, Yages Singh, Tanya Doherty, Mickey Chopra, Petrida Ijumba, Alexander C. Tsai, and Debra Jackson. "The use of mobile phones as a data collection tool: a report from a household survey in South Africa." BMC medical informatics and decision making 9, no. 1 (2009): 1-8.

Henderson, Savanna, and Michael Rosenbaum. "Remote surveying in a pandemic: research synthesis." Innovation for Poverty Action (2020).

[4] Buntaine, Mark T., Ryan Jablonski, Daniel L. Nielson, and Paula M. Pickering. "SMS texts on corruption help Ugandan voters hold elected councillors accountable at the polls." Proceedings of the National Academy of Sciences 115, no. 26 (2018): 6668-6673.

Hainmueller, Jens, Duncan Lawrence, Justin Gest, Michael Hotard, Rey Koslowski, and David D. Laitin. "A randomized controlled design reveals barriers to citizenship for low-income immigrants." Proceedings of the National Academy of Sciences 115, no. 5 (2018): 939-944.

[5] International Crisis Group, “Hard Times in a Safe Haven: Protecting Venezuelan Migrants in Colombia,” International Crisis Group, August 9, 2022, https://www.crisisgroup.org/latin-america-caribbean/andes/colombia-venezuela/hard-times-safe-haven-protecting-venezuelan#:~:text=09%20August%202022-,Hard%20Times%20in%20a%20Safe%20Haven%3A%20Protecting%20Venezuelan%20Migrants%20in,the%20clutches%20of%20organised%20crime.