In PIALA-based mixed evaluation designs, we employ Quasi Experiments and Conventional Statistics alongside participatory methods to measure immediate effects of specific programme interventions or innovations and identify trends or changes in the impact distribution of complex programme configurations across large populations.

Quasi Experiments can help determine if a certain programme or policy intervention or a newly introduced mechanism has the intended effect that is considered necessary for achieving sustainable impact. It can be particularly useful for assessing the added value of innovations or of innovation packages within complex programme configurations. Examples are: a micro enterprise fund aimed at linking smallholders to commercial markets within a country-wide commodity chain development programme; or a participatory WASH governance model aimed at enhancing access to clean water and sanitation in new urban settlement communities.

Conventional Statistics play a crucial role in analyzing vast amounts of quantified effectiveness data collected through impact surveys and performance monitoring. It is most suitable for identifying trends or changes in the impact distribution of complex programme configurations across large populations. Examples are: trends in self-employment and entrepreneurship among youth in a programme aimed at enhancing their socio-economic empowerment; or distributed impact of smallholder-led commodity chain development on access to food and income in remote rural areas.

Use of these methods in PIALA-based evaluation

We typically combine these methods with Participatory Statistics in mixed-methods evaluations that are designed and implemented with our Participatory Impact Assessment and Learning Approach (PIALA). The purpose is to understand why specific interventions (or configurations of interventions) were more or less effective for certain groups in certain areas, and how this affects the inclusiveness and sustainability of impact-level changes.

Quasi Experiments and Conventional Statistics require sophisticated sampling techniques and reliable data collection tools. Depending on the context and purpose of the evaluation, we apply random, purposive and/or self selection techniques to sample cases and populations that receive the interventions and their best matching comparisons that do not receive any intervention (or are receiving different types of interventions). Furthermore, we develop survey tools in close consultation with key stakeholders and crosscheck them with other methods to optimise the construct validity of the findings.

The sampling techniques and data collection tools are used in ways that strictly adhere to our core value of empowerment and our voice, rights, ethics and safety policy. Survey tools are increasingly mobile, requiring safe/protected and reliable data storage, transfer and management systems.

A PIALA-based mixed design often requires a harmonisation of different impact definitions and constructs (e.g. attribution- and contribution-based). Through PIALA’s Configurational Analysis method, we seek to integrate all data analysis processes as the basis for assessing the more complex causation hypotheses as well as the direct and more predictable cause-effect relations within a complex programme configuration. Where feasible, this generates evidence-based in-depth insights for learning and improvement.

Making tradeoffs in ‘mixed designs’

Mixed designs can be quite demanding and may not always generate the desired value for money. It demands a thorough consideration and balance of aspirations related to rigour, inclusiveness and feasibility. This involves assisting evaluation commissioners and users in understanding the value for money and navigating the necessary tradeoffs.

When assessing the evaluability of interventions, we therefore look at not only the likelihood but also at the feasibility of uncovering credible evidence that can produce useful and actionable insights within the available time, resources and capacities.

Projects

Resources

• Holland J, et al. (2018). Getting the Most Out of Participatory Impact Assessment: Reflections from a Multi-Country Cash Transfer Impact Assessment. CDI Practice Papers. IDS.

• Peersman G, Guijt I and Pasanen T. (2015). Evaluability Assessment for Impact Evaluation. The Methods Lab. DFAT, ODI and BE. Stern E. (2015). Impact Evaluation: A Guide for Commissioners and Managers. Bond.

• Van Hemelrijck A. (2013). Insights and lessons learned from the reflections on the PIALA piloting in Vietnam. IFAD.