This blog was written by Claire LeBlanc, Rebecca Adler and Matthew McConnachie at NIRAS-LTS International, who recently hosted a webinar with The Cloudburst Group on how the sector has adapted and improved its remote data collection, analysis and learning processes during a difficult year when project plans were disrupted.
The COVID-19 pandemic forced a change in how Monitoring, Evaluation and Learning (MEL) practitioners work. Scotland’s International Development Alliance, along with NIRAS-LTS International and Cloudburst, came together in a webinar to reflect on the challenges, complexities and unexpected positive practices that have emerged over the past year. In reviewing what has changed, been adapted or rethought entirely, we can take the learnings to bring MEL back better in the post-pandemic world.
Nearly half of webinar participants reported that the pandemic impacted not only their MEL activities, but also the scope of their MEL work. The implications of the pandemic on MEL for most practitioners meant transitioning to a remote practice, tapping into digital and remote resources, and relying more on local evaluators. It also brought the need to rethink evaluation design and approach to adapt to different needs and considerations, for example in data collection where in-person activities would no longer be possible, or for managing a team that included those with caring responsibilities.
For collecting quantitative data in remote or rural areas, evaluators were able to tap into digital resources and platforms, such as mobile phone surveys in place of in-person household surveys. This method required a strong in-country team already in place to implement. It also came with trade-offs in the form of reduced survey length, shorter interview times, and sampling challenges. These trade-offs needed to be addressed by changing the scope of the evaluation.
Other MEL practitioners utilised online qualitative moderated discussion platforms, such as LiveMinds, to hold online discussions and focus groups with interviewee groups. The platform allowed for flexibility for the interviewees who lived across geographies and time zones and gave the evaluators data-rich information that was cost-effective over traditional focus group methods. With leftover funds originally allocated for international travel, it also allowed the evaluators to use a data science company to produce focused web and document data analysis, which then produced a stronger final evaluation report.
Practitioners creatively reimagined evaluation design in response to COVID-19. Not being allowed to travel prompted teams to choose projects where they had historically been prohibited from doing in-person work, due to geographic location and remoteness, or security risks. With these former barriers removed, the evaluators were able to widen the scope of the study to capture a greater context and allow wider data collection. By moving evaluation activities online, the pandemic further forced project teams to become more organised, with clear meeting agendas and workshop deliverables to ensure effective online engagement. It also created space and a greater reliance on the in-country project team to make arrangements with project stakeholders and ID interviews.
But moving things online presents its own challenges; for example, online meeting fatigue needed to be considered and managed. Turning digital also meant that the quality of connection could not be developed as fully with informal conversations with local project teams, resulting in the loss of ability to pick up on the subtle nuances of the project work they were evaluating. Additionally, relying on digital and remote tools and methods such as mobile phones naturally introduce bias: those without a phone, or reception/connectivity are not able to be captured.
Despite these challenges, the pandemic has evolved MEL by creating more agile practices that evaluators are confident in utilising to produce robust data. It has encouraged more use of local evaluators, an environmentally conscious act from less international travel that also strengthens networks and capabilities in partner countries. These lessons and changes could make an important contribution to building MEL back better and support its important role in strengthening societal resilience to future shocks.
If you are an Alliance member you can watch the webinar here.