10 highlights from the Monitoring, Evaluation and Learning issue of the Gender & Development Journal


July 28th, 2014

WyI2MDB4NjAwIiwiY3JvcCJd

The Journal of Gender and Development has dedicated its latest issue to Monitoring, Evaluation & Learning (MEL). Below are quotes from across all the articles which reflect some of the current discussions and challenges practitioners face when reflecting on the systems they are using to capture change:

 

 1. MEL is viewed as a technical realm rather than a political one

‘Despite its immense importance, MEL is all-too-often viewed reductively as a technical realm, rather than the political one that it really is (‘MEL can render development policymakers, practitioners and researchers accountable to the individuals and groups they aim to support, as well as accountable to the funders and supporters of that work’). It may be dismissed as consisting of repetitious, time-consuming technical tasks, involving flow charts, logical frameworks and lists of ‘indicators’ which need to be generated in order to keep donors and managers happy and ‘feed the beast’. When we considered how we wanted to commission this issue of Gender & Development, a very important factor was the need to challenge this reductive view of an essential part of development and humanitarian work.’ (Introduction to Gender, Monitoring, Evaluation and Learning: Kimberly Bowman & Caroline Sweetman)

 

2. Putting a price on MEL

‘The cost of M&E systems should be noted. Many donors limit the total cost of M&E to around 5–7 per cent of the total project budget, but this is difficult to stipulate, given the very different nature and significance of development projects. Furthermore, the cost can be calculated in different ways, especially with regard to staffing costs. The question remains: are sophisticated M&E systems themselves ‘value for money’ in terms of the information that they provide – to the donor, to project staff and partners, and to the project beneficiaries?

Funding appeared to be lower for qualitative evaluations. A large number of those reviewed (or parts of them) had limited scope and reach, with small samples. This suggests a lack of value and funding attached to qualitative methods. (A review of approaches and methods to measure economic empowerment of women and girls, Paola Pereznieto, Georgia Taylor).

A risk is that if success is increasingly defined by quantitative indicators, then meaningful signs of change which are harder to quantify, such as an increase in the self-confidence of a survivor of violence, are not valued, making some projects less attractive to donors. Tackling this challenge requires both creative ways of expressing and summarising qualitative change, and a recognition of the value of measuring changes more suited towards qualitative data collection.’

 

3. The temptation to claim big to secure funds and support leaves MEL ‘numbers’ driven

‘Monitoring and evaluation of development projects is more of a priority than it ever has been, in a world of austerity in the global North, and complex crises facing both South and North, creating and perpetuating poverty and social and political marginalisation. Resources available for development initiatives in the global South fall far short of the need for them, and governments and development organisations are keener than ever to use their resources as efficiently as possible.’

Claiming big is a temptation if development workers are to secure the support and funds they need for the work from the powerful. They would like clear demonstrations of results. On the other hand, achieving change of the ambitious kind requires long-term work from development organisations, and partnerships with the international women’s movements who have long experience of struggles for political and social justice. This work takes place against a backdrop of wider changes which are beyond any organisation – or state – to control. The role of planned development interventions in achieving such vast goals as ‘gender equality’ can only ever be modest. In MEL, what many powerful stakeholders seek is sharp, clear, definite information which can be assessed, compared, and contrasted.’ (Introduction to Gender, Monitoring, Evaluation and Learning: Kimberly Bowman & Caroline Sweetman)

 

  4. Assumptions limit the level of analysis, leading to women and men being asked questions based on misconceptions

‘The reviews identified that evaluation teams and researchers often make underlying assumptions about gender roles and relations, and women’s activities, and use stereotypes. This can limit the level of analysis, for example leading to women and men being asked questions based on misconceptions.’

Type of expenditure: Women may only be asked about expenditure on household items or child-related expenditure, whereas men are asked about recreation and other activities… negating the possibility that men may also have a responsibility for household expenditure.

Type of decisions: Women are typically asked about who makes decisions about issues that are relevant to the division of household labour (that is, who decides how women’s and children’s time is allocated), but seldom asked about other critical areas of life such as decisions about who participates in community or civic events, how women’s time is allocated to activities outside the household (labour or social), and the like. Only rarely may questions be asked about joint decision-making, with researchers tending to frame their questions as ‘either/or’ (e.g. are household decisions made by men or women?). In many cases, joint decision-making is the norm in many decision-making spheres. Even when this is revealed it is rare to see an analysis that explores the relative power balance between men and women who are making decisions together. In general, this level of detail was missing from the evaluations.

Type of economic activities: Assumptions may be made about women being homebased, or economically active only in certain sectors, so that their involvement in other sectors is not considered. Considerations must also be made of unpaid care work or subsistence work which changes women’s workload dramatically and completely affects their engagement or non-engagement with paid work.

Involvement of men, boys, and young women and girls in the evaluation. Many of the evaluations reviewed did not conduct interviews, surveys, or focus group discussions with men and/or young men, unless the project was specifically targeted at men as well. It could be argued that it is impossible to measure women’s economic empowerment unless men’s attitudes and behaviours are also taken into account. It is also true that issues including the relative contribution and therefore significance of women’s income cannot be discerned – even though these issues are important to the question of empowerment within marriage and the family – without research into men’s economic role within the household. This can be questioned in interviews with women but triangulation and direct involvement of men in interviews is likely to improve the reliability of data. The engagement of men was seen as critical in gaining acceptance for the strategies chosen to improve women’s economic empowerment. Some of the evaluations had a general sample with men and women, but with no disaggregation of results, so women’s empowerment was not easy to identify. There were very few evaluations of interventions targeting girls’ or young people’s economic empowerment. Reports were excluded because they were either poor quality and/or lacked detail. In the other evaluations, lack of age-disaggregated data meant that impact on women and girls was not separated out, even though there are many young women and girls who are married and may form an important part of a sample. There was no evidence of differentiated approaches being used. (A review of approaches and methods to measure economic empowerment of women and girls, Paola Pereznieto, Georgia Taylor).

 

5. Development projects focusing on power, and on increasing participation in decision making have greater difficulty in fulfilling conventional donor MEL requirements which emphasise countable inputs and outputs.

We then consider whether or not empowerment as an aim has in fact been ‘lost in translation’ because of the systems we use. Our findings show that understanding empowerment requires mixed methods, that capturing empowerment as a holistic process requires equally holistic M&E systems capable of analysing all power’s dimensions (‘power-over’, ‘power-to’, ‘power-within’, and ‘power-with’), and that we should design the M&E of women’s empowerment to be itself an empowering, voice-enhancing, process.

Projects often seek to increase women’s empowerment in large part by providing economic opportunities specifically for women…While changes in income and decision-making are being measured, the effect that interventions have in adding to women’s existing roles and responsibilities is not sufficiently understood or measured. Economic interventions of the type described – whether aimed at a collective or more individual income generating activity – have related and generally quite intensive demands on the participant’s time and energy. It is fair to say that our evaluative work to date has not adequately assessed the knock-on effects of these demands on supported women’s time or responsibilities. (Learning about women’s empowerment in the context of development projects: do the figures tell us enough? Jane Carter Sarah Byrne, Kai Schrader, Humayun Kabir, Zenebe Bashaw Uraguchi, Bhanu Pandit, Badri Manandhar, Merita Barileva, Norbert Pijls & Pascal Fendrich).

 

6. Highlighting the importance of ‘Power within’

‘The four dimensions of power are referred to and defined as follows: 1. Power within: the knowledge, individual capabilities, sense of entitlement, self esteem, and self-belief to make changes in their lives, including learning skills to get a job or start an enterprise. 2. Power to: economic decision-making power within their household, community, and local economy (including markets), not just in areas that are traditionally regarded as women’s realm, but extending to areas that are traditionally regarded as men’s realm. 3. Power over: access to and control over financial, physical, and knowledge-based assets, including access to employment and income-generation activities. 4. Power with: the ability to organise with others to enhance economic activity and rights.’

‘At the time of writing the conclusion to this paper, one of us was struck to read the following comment made by a woman who had participated in a training session in Kyrgyzstan, which perfectly highlights the issue we have raised in this paper:

Question: Would you like other farmers to join such project activities?

Response: Yes, of course. Firstly, they can change and develop their self-awareness. Secondly, participating in meetings and trainings will help them to get useful information that they can use in practice. (Interview with Malika Djulaeva, Tolman village, Kyrgyzstan, 4 March 2014)

Malika Djulaeva identified the development of self-awareness, or ‘power-within’, as the first reason why farmers should participate in trainings about the efficient use of irrigation water. She prioritised this ahead of the practical information the training was ostensibly organised to provide. In short, it could be argued that we are measuring the wrong things because we are too focused on what can be readily counted and on measuring results according to a very limited list of objectives. This brings us back to the women with whom we work. If we take the issue of their empowerment seriously, not only with regard to the activities that we implement but also in how we generate knowledge and learning about their life experiences (related to project interventions), we need to reflect on our role in M&E processes as translator. Does it have an empowering, or a disempowering, effect?’ (Learning about women’s empowerment in the context of development projects: do the figures tell us enough? Jane Carter Sarah Byrne, Kai Schrader, Humayun Kabir, Zenebe Bashaw Uraguchi, Bhanu Pandit, Badri Manandhar, Merita Barileva, Norbert Pijls & Pascal Fendrich)

 

7. Advocating for a mixed methods approach

‘There is a growing body of literature advocating the use of mixed methods for M&E (see Stern et al. 2012, amongst others). Our own findings have shown that this would allow a much more nuanced view of outcomes in terms of women’s empowerment and, importantly, would leave space for women to identify their own indicators of change.

Quantitative methodologies tended to be used to demonstrate that: (1) change had taken place; (2) the intervention caused the change to take place (causality); and (3) the findings can be generalised across a population group (in the case of representative samples). Qualitative methodologies were most commonly used for: (1) context analysis to design sampling and tools; (2) conducting participatory activities to identify indicators of change; (3) establishing how the change takes place; (4) understanding why change happens (or does not happen); (5) researching how people understand and describe that change; and (6) identifying unintended changes or impacts. (A review of approaches and methods to measure economic empowerment of women and girls, Paola Pereznieto, Georgia Taylor).

Despite being characterised as yielding ‘soft’ data, research methods which involve open-ended discussion including focus group discussions and case studies yield a mix of ‘hard’ and ‘soft’ data. Hard data include shifts in decision-making within marriage – for example, on women’s right and freedom to travel, or to choose their own clothes.

Research methods associated with collecting qualitative data often actually reveal unexpected quantitative data, including changes in children’s school attendance, better nutrition, and so on. However, projects do not set out to monitor such a wide range of outcomes, due to the cost of collecting mountains of data and the challenge of attribution (e.g. linking women’s capacity building to children’s nutritional status). In contrast, ‘soft’ data can be characterised as perceptions and impressions of less tangible and material changes, which nevertheless have immense significance in women’s lives: for example, women’s sense of increased self-confidence, and perceptions of greater respect in the community. These outcomes are actually more readily attributable to project activities, but are not yet specifically monitored.We were struck as we explored the three projects that, as the following quotations show, and they are typical in this respect, the changes associated with ‘soft’ data appear to be what the women value most in projects.’ (Learning about women’s empowerment in the context of development projects: do the figures tell us enough? Jane Carter Sarah Byrne, Kai Schrader, Humayun Kabir, Zenebe Bashaw Uraguchi, Bhanu Pandit, Badri Manandhar, Merita Barileva, Norbert Pijls & Pascal Fendrich)

 

8. The importance of signposting the changes

Helen Lindley from WomanKind advocates the use of progress markers to ‘signpost’ the changes. Outcome journals are encouraged to record low, medium and high level changes. Why is it important to ‘signpost’ the changes?

It might take a long time to see changes

Programmes that cause change in the long term are often seen as failures in the short term e.g. ‘projects which seek to raise women’s knowledge of the existence of local service providers for cases of violence, and support women’s confidence and ability to access services if needed, may result in an increase in the number of women reporting cases of violence in the short term. Viewing these statistics without knowledge of the programme’s theory, what changes it expected to achieve, and when, would risk painting a picture of failure.’

Change is subjective: ‘Empowerment may vary according to a woman’s background. For an indigenous woman it may be seen in terms of having access to land, water, and resources; for a richer Peruvian woman a sign of empowerment may be being able to influence decision-making; and for a rural woman it may be not being beaten very often, or denouncing the violence they face.’

Changes might take place after the project: can these be linked back to signposts recorded earlier on?(Reflections on Womankind Worldwide’s experiences of tackling common challenges in monitoring and evaluating women’s rights programming, Helen Lindley).

 

9. Good-quality evaluations contain a good analysis of the social and economic context, including a strong gender analysis

‘If context analysis is carried out sufficiently well, it will enable a relevant theory of change and an evaluation methodology that takes into account the context. Good context analysis presented market and economic context, political and institutional context, gender analysis including data on the gender division of labour, roles and power relations, and information on wider processes of economic, social, and political change at household, community, and institutional levels. Gender analysis is key to good context analysis. This is essential for good-quality evaluations, and our findings bore this out. Fewer than 25 per cent of the reports reviewed included a strong gender analysis. Gender differences, gender-related norms and behaviours, gender roles, and gender relations differ in every context, and assumptions should not be made when designing an evaluation.

 

10. Advocating for approaches which assess contribution to change,   rather than a framework seeking to claim the entire credit for change

MEL can help to transform development and the world we live in by exploring and recording the experience and perceptions of women – in particular, women living in poverty – whose lives have in the past gone largely unrecorded, and whose experience has remained outside the realm of what is conventionally seen as knowledge. MEL in a development organisation which seeks to challenge unequal power, reduce economic and political inequality and support a changed global order which takes into account the interests of the poorest in our societies is a massive challenge. Women’s rights activists call for a more genuine commitment to making MEL a learning partnership rather than ‘mining’ for facts or putting partner organisations through a performance test, and for a balance between quantitative and qualitative techniques and a use of multiple methods, gender analysis tools and frameworks. They also call for approaches which assess development programmes’ contribution to change, rather than frameworks which focus solely on changes attributable to development programmes – ‘that seek to claim the entire credit for change’ (Batliwala 2011a, 4).

 

 

Leave a reply