Monitoring and evaluation (M&E), data collection and management, including digitalization

• M&E for SBD should include monitoring process efficiency, monitoring commodity audit trails, and evaluating outcomes.
• RBM has defined population and household survey indicators that countries should track as part of their overall ITN M&E strategy, but more detailed programme-specific indicators would help inform the contribution of SBD to coverage targets and inform design decisions in future years.
• Routine monitoring may be supported by evaluations using household surveys to estimate relevant coverage indicators recommended by RBM.

The following activities may be useful to routinely monitor:

• Quality of trainings.
• Supply chain processes: Programmes may choose to fund an audit process to review proper adherence to processes and the quality of the commodity audit trail.
• Assessment of SBC activities: Ideally, health promotion in schools and communities is conducted throughout the year. Pre-distribution events may include class discussions, school assemblies, radio spots, community activities, etc. ITN care and maintenance will be an important focus.
  • Data on the strength of implementation should be routinely captured, e.g. number of activities conducted, target geographies/audience, communication channel(s) used, message(s) conveyed.
• Review of distribution data: Data on ITNs distributed to students should be collated and reported up the supervision chain (e.g. from school to cluster to sub-national and national levels). Key metrics to measure are:
  • Number of ITNs delivered
  • Percentage of targeted students recorded as receiving an ITN (by level, depending on the SBD design: grade, school, cluster, sub-national area, (e.g. district, region) and nationally)

• Convene the SBD coordination group to develop or review a M&E plan and tools. If members of the NMP and MoE M&E teams are not usually involved in the SBD coordination group, they should be invited to the meeting(s) to develop or review the M&E plan and tools.
• Consider suitable M&E systems: Determine whether data collection and reporting will occur using an existing data collection and reporting system or one created specifically for school-based ITN distribution.
  • Determine whether data collection and reporting will be paper-based or digital. Digital approaches are preferred and strongly recommended.
  • Data collection and reporting tools should be developed accordingly.
• Consider existing national health reporting platform(s): Where feasible, data should be integrated into national platforms such as DHIS2 to enhance interoperability and improve real-time monitoring.
• Budget accordingly: A data collection and reporting system specific to school-based ITN distribution may require additional training or procurement if appropriate systems are not already in place.
• Similarly, resources for digital data collection and reporting systems may require additional training or procurement of electronic devices.

Consider the following points:

• Data requirements for routine monitoring should focus on need-to-know data (based on defined indicators) and exclude nice-to-know data. Simplifying data requirements is important to improve the accuracy, completeness, and timeliness of information collected.
• Ideally, monitoring data should flow through existing systems. However, there may not be an existing health reporting structure which is appropriate for reporting on SBD, and therefore, it will need parallel reporting systems. Guidance can be drawn from campaign-style monitoring (see theAlliance for Malaria Prevention [AMP] toolkit, Chapter 8, for guidance).
• Monitoring activities can be time-consuming and costly. The results, however, are vital to understand how the programme is performing and where improvements can be made to better serve populations at risk of malaria. Creating a culture of quality data collection and data use for decision-making takes time but is an investment that should repay itself in increased transparency, efficiency and outcomes. Wherever possible, data review and decision-making should be integrated into existing health systems and activities, to avoid adding parallel or burdensome activities.

• Routine monitoring can be complemented by community surveys of coverage conducted with an appropriate delay after the SBD event.
• The timing of community surveys is critical:
  • Respondent recall bias will increase with increasing time since distribution. However, conducting a survey immediately following distribution may not provide sufficient time for households to integrate the SBD ITN into their household ITN practices.
• Surveys can be used to estimate:
  • Percentage of households with eligible schoolchildren that received an SBD ITN
  • Characteristics of households with eligible schoolchildren that did not receive SBD ITNs
  • Percentage of SBD ITNs retained by households
  • Reasons SBD ITNs were not retained by households
  • Household and population coverage of ITNs
  • Contribution of SBD ITNs to household and population ITN access (requires data on other ITNs in the household and household composition)
  • Use of SBD ITNs, by whom and how many people
  • Community member perceptions and experience of the SBD programme (e.g. reach and recall of SBC activities)

To improve SBD planning and implementation it is vital that M&E data are analysed and results are interpreted and used. Following analysis by the M&E team, the SBD coordination team should identify and document successes, challenges and lessons learned to inform future SBD activities.

• Plans made at the start of an SBD strategy are typically pragmatic but may be overly optimistic. Regular formal review of monitoring and supervision data, through regular management meetings at appropriate levels, will inform lessons learned and improve future performance.
• Modifications and improvements are very likely to be needed but they should be seen as welcome opportunities to strengthen the system, particularly in the early years of implementing a new SBD approach.
• The performance of an SBD system can vary over time and between contexts. Plans that worked well one time, and in one place, may need to be adjusted. Even after a system has been running for several years, ongoing monitoring, regular reviewing and adjusting will probably be needed.

Who should review data, for what, and at which levels?

Class teachers and head teachers
Review their class and school performance, considering the number of ITNs distributed versus the number expected to be distributed from the enrolment data and the number of children present on the day of the SBD.

Head teachers
Reflect on the process flow of SBD within the school, with input from class teachers, to inform future adjustments.

Sub-national-level structures
A key point for reviewing sub-national data, including identification of outlier schools or clusters with poorer-than-expected results. Sub-national MoE teams may have some leeway to modify the operational approaches used in their area, as appropriate, based on the findings. Some recommendations may be best reported up the supervision chain to a higher or national level to see amendments to the model nationally if similar findings are being reported throughout the country.

Hold (annual) process review meeting
With distribution typically occurring once per year, it is vital to ensure that a yearly post-distribution review meeting is scheduled to assess performance and allow for a timely revision of plans for the following year.

• The review should be informed by participants’ experiences and not just be numeric data, i.e. qualitative as well as quantitative. Stakeholders from students and communities, schools, sub-national divisions and the national level should be involved and include all departments or organizations involved.
• Training, monitoring, supervision and distribution data should be collected and analysed, and successes, challenges and lessons learned synthesized.
• Sub-national process review meetings should be held to disseminate findings and gather personal experiences from stakeholders. Insights from sub-national meetings can feed into a national meeting.
• A final report should be prepared and disseminated, including a clear list of recommendations and costed actions to be implemented for the next SBD, with realistic timelines and focal points for each action.

The following knowledge gaps around SBD may be of interest to programmes, implementing partners and funders of SBD.

• Destination of ITNs received by students: ITN quantification approaches assume that ITNs are redistributed within and outside of households. Understanding ITN redistribution is important to verify the quantification model. The information could also be useful to allow countries to understand whether their chosen mix of channels might leave some subgroups of the population without access to ITNs.
• Can schools have a role as a continuous ITN distribution point for households without school-age children, to increase access to ITNs for a larger percentage of the population?
• What is the feasibility of distributing ITNs continuously through schools, as opposed to during an annual or bi-annual distribution event?
• Researching the most effective strategy to combine different CD channels to reach the greatest number of people in the most equitable manner should be a priority topic for consideration.

• ITN SBD routine monitoring forms [Word] EN | FR | PT
• Protocol for evaluation of CD (which can be modified to suit the needs of SBD) is available here [Word]. The associated survey questionnaire is here [Word].

• Zambia NMEC [Word] SBD routine monitoring forms at multiple levels and supervision checklist.
• Zambia NMEC [Word] ITN School-Based Distribution Campaign, 2021, Final Report.