The role(s) of evidence
The role(s) of evidence
The encounter with two of the people behind Tamkeen last week has stayed with me and reverberated. It has stimulated further thoughts, for example on the role(s) of evidence. It has gotten me to a point where I'm holding two seemingly contradictory truths around evidence.
There is the view that evidence is needed for us to show that we are spending our money well and that we are making a meaningful difference in the world together with our partners. This is the view of evaluation as a means of proving that change happened, of analysing our contribution to change, and of establishing value for money - are we doing those things that generate the greatest effect possible with the necessarily limited amount of money we have at our disposal? This is the thinking behind effective philanthropy. The field of evaluation has become really good at answering questions to this effect, and I'm particularly interested in how these relevant question can be answered while still seeing the world as complex and emergent rather than linear and predictable.
Yet, there is a part of my that asks, "what's the point of all that?" It feels that this way of defining the role of evidence and evaluation puts us as an organisation and our money at the centre of attention and even when we talk about us and our partners, in the end the question is always what our money has achieved. Whether we have spent our money to the best effect measured according to our values. Yet if we hold true what I described in the paragraph above, this is not a problem but an obvious consequence.
Looking from this first perspective on evidence, what irritated me about the conversation with Tamkeen last week was that we never much got to talk about what the organisation we fund has been doing as part of our shared project, and what change they observe as a consequence, and if they thought the contribution we are making is a meaningful one. We always got back to reflecting on the questions we are asking and why we were asking them. However, as I wrote last week, by focusing the reflection on the people in the room and becoming each other's mirror for reflection, we experienced the essence of Tamkeen itself and it did feel like a magical experience. We learned a lot about ourselves; and we learned a lot about the process by experiencing it ourselves. There was no lack of impressive anecdotes that showed how Tamkeen has an effect on people in the education sector in Morocco, including students, teachers, parents and people in the Ministry of Education. Still, my evaluator's mind asked for a more robust way to show the changes and say something about our contribution.
Based on the experience with Tamkeen and my Warm Data background, the second truth I hold about evidence is that the personal experience of Tamkeen and comparing that with what effect it has on the people who experience it in Morocco should be evidence enough that our partnership with Tamkeen is a meaningful endeavour, even without success indicators, KPIs, outreach measures, etc. This truth is rooted in an understanding of deep (trans-)contextuality of change and that the people who live in and experience a context and its dynamics on a daily basis are the best judges of how meaningful and beneficial something is for their lives. There are many more factors that determine if something is 'successful' than those we can know of, and even more than those we can objectively measure. The question is if anything we can measure would ever sufficiently be able to capture what is going on in a human system. Given the complexity of life, what is more important than measures is to find actors from within a context that found a way to relate to other people in that context in a generative way, in a way that aligns with our own values and principles, and to support them in doing what they do. No need for intricate methods to deduct causal pathways and contributions. If we do that across multiple contexts we are making a meaningful contribution by allowing life to live and thrive, particularly enabling the shoots that promote the same principles we hold dear, like in our case human rights or participation.
Maybe these two thruths are not incomensurable, though. Cathy Sharp, an experienced evaluator, challenges us in her award-winning essay: "to let go of a desire to attribute impact or isolate contributions, but to share credit, and be accountable for our learning, rather than for specific outcomes." (Sharp 2022)
Sharp's recommendation to use an action inquiry approach to evaluation in complexity might well be the way to bring the two truths together. To that end, the approach aims to overcome the "high expectations of evidence-based or informed practice" with its "embedded assumptions about what we can claim for our intervening, what is valid evidence, what transfers and how we go to scale" because these "act as barriers to evaluation in complexity." Instead, Sharp advocates that what we need are "conversations that discover, explore, and co-create (rather than manage) our mutual expectations and assumptions and track how these might themselves be influenced by the work as it unfolds." This sounds a lot like the conversation we experienced with Tamkeen.
Sharp addresses the same perceived dichotomy I'm alluding to: On the one hand, "funders and commissioners still want to be confident that they will achieve value for their investment or that public money will be used well" while on the other hand we need to live up to the "realities of evaluating in complexity where ‘nothing is clear, and everything keeps changing’. Sharp cautions that the former has lead to a situation where "fear of perceived failure and the real risk of loss of funding creates a culture of gaming and superficial evaluation at the expense of genuine learning that recognises good work and supports change."
What Sharp suggests is to adopt an action inquiry approach to evaluation, which invites our partners and wider group of stakeholders to "become an active participant rather than spectator of evaluation." This, again, reminds me of Tamkeen, where the process becomes its own meta-process to recognise and appreciate one's own experience in the process. Or in Sharp's words: "The idea of building inquiry into living systems, generating practical wisdom from the work that we are involved in, and becoming observers of experience, might help us address the reasons that so many change-efforts fail." The inquiry thereby is the intervention and the evaluation at the same time, as "Everything that you do in a system is an intervention... and everything you experience is data about the system" (Edgar Schein, quoted in Sharp 2018).
It is my aspiration to establish a systemic inquiry approach at Fondation Botnar. We are very early on in this process and have a lot to learn still. No doubt I'll keep writing about it. Please do let me know if you have any relevant experiences that you could share.
References:
-
Sharp, Cathy. 2022. “Be a Participant, Not a Spectator – New Territories for Evaluation.”The Evaluator.
-
Sharp, Cathy. 2018. “Collective Leadership: Where Nothing Is Clear and Everything Keeps Changing - Exploring New Territories for Evaluation.” Collective Leadership Scotland. http://tinyurl.com/y6vll2hp.
The Paper Museum
This is from a blog article of the European School of Governance on their approach to evaluating Tamkeen (emphasis in the original):
Addressing magic and the existentiality of love in the context of scientific research is a challenge. Even in the contexts of community development and human development, it feels unfamiliar as it includes the commonly excluded. Tamkeen was never at ease with being called magical, however, it addresses love with ease, advocating for the inseparability of science and love. Certainly, the Arabic language facilitates this ease, nevertheless, exploring and trusting our human potential and our humanity, be it in poetry or arts, philosophy or science, will eventually guide us to meet love and its existential unconditionality. The dislocated locus of value of the ubiquitous culture of achievement returns to the immanence of being and becoming. Addressing love reveals the limitations of contemporary research frameworks and the prevalent scientific language. Yet, accepting this challenge does not only allow embracing love in science, acknowledging its inseparability, but it also facilitates our understanding of research and science as expressions of our humanity. Or as Fatima’s father, one of the elders in the Zouitina neighbourhood, addressing the necessary coherence of experiences, says: “What you do is good if I feel in it the heart that I feel in my community.”
Why have I added this to my Paper Museum? It connects directly to my Paper Museum entry from last week by talking about emotion and science - two of the quadrants of the Medicine Wheel of Knowledge - and particularly love and science, and how they can both be included in an approach to evaluation. I particularly relate to the move from a focus on achievement to one on being and continuous becoming - from doing to being. Something I think about often. It also connects to my thoughts above about two views on evidence when quoting Fatima's father as his statement expresses a way of capturing truth that is not appreciated in traditional evaluation as it is not seen as objective or rigorous.
Photo
Étienne Léo pold Trouvelot (1827-1895): Total Eclipse of the Sun, May 29, 1878. Bibliothèque de l'Observatoire de Paris. I took this photo at the Museum Barberini in Potsdam.