The Community is Not the Problem

Overview

The Interview

Gladys Rowe: Tansi. Greetings. Join me and my guests as we open up our evaluation bundles to share what we’ve gathered in our journeys and bring them together into this space. What have we learned so far? My hope is that these interviews allow you to reflect on how to design, implement, learn from, and support evaluation by with and for Indigenous families, communities, organizations, and nations.

Welcome. I am so glad to be here with Aneta Cram, a doctoral candidate with the School of Health at the Victoria University of Wellington in Aotearoa. Her research explores what Indigenous evaluation frameworks currently exist, how they were developed, and what impact they’re having in the communities that they were developed for. This research will provide guidance to other Indigenous communities in developing their own community-specific evaluation frameworks. So excited to have you here today, Aneta. I’m wondering if you would like to introduce yourself into this space in any other way before we get started?

Aneta Cram: Yeah, thank you so much, Gladys. Ko Takitimu te waka, ko Mohaka te awa, ko Tawhirirangi te maunga, ko Te Huki te marae, ko Ngāti Kahungunu me Ngāti Pahauwera ngā iwi. Greetings everybody. I just shared how I connect with the land of Aotearoa New Zealand, connecting with the voyaging canoe Takitimu, the river Mohaka, the mountain Tawhirirangi, the meeting house Te Huki and my tribes Ngāti Kahungunu and Ngāti Pahauwera as well. Thank you.

Gladys: Thank you so much for sharing those relations that you bring with you into this space. I’ve been looking forward to this conversation for a long time, but I had a lot of patience because I know you’re working on your PhD right now. I’m wondering if maybe you can share a little bit about how long you’ve been working in the field of Indigenous evaluation and how you made your way into this space?

Aneta: Sure. I’ve been working in the field on and off for the past 10 years, and I actually came into the field through my aunt, Dr. Fiona Cram, a Māori researcher and evaluator here in Aotearoa. I had finished my undergrad in 2013 in an area that was absolutely unrelated to evaluation. I did my degree in philosophy and religious studies and I started working for Fiona. She invited me along to get a little bit of work experience and see what this whole evaluation thing was all about. And I got involved as a research assistant working with her on a Kaupapa Māori evaluation of a Māori boys’ after-school program, which was looking to connect young men to their roots. And I got a little taster of that and thought it was really interesting, then wanted to continue on from there. Yeah.

Gladys: So your aunt hooked you into this work? <laughs>

Aneta: Yeah.

Gladys: I’m assuming you’re still interested because you’re doing your PhD in this work. What drew you forward to say this is something that’s really important?

Aneta: I think part of it was the relational aspect, connecting with different groups of people that were looking to really make change for their community. And seeing that and hearing different stories – I don’t know if beautiful is the right word, but it was really motivating, and to also see the impact that evaluation can have, whether that’s just getting a little bit of funding or putting down onto paper people’s experiences, and the impact of that, if that makes sense. That someone can actually read that quote and think, Oh, that was my experience. So that was part of the drive. I’ve been in and out of different evaluation spaces – Kaupapa Māori, Indigenous, and working in other parts of the globe, as well as very Western spaces.

I did my master’s in evaluation in Australia a few years ago. And for me, that was getting a real taste of the foundations of Western evaluation and what’s expected. You know, learning about the philosophical foundations as well as the methods that are privileged in the evaluation field. My education there was very, very Western, with only small considerations being given to culture and really nothing that focused on indigeneity. To me, having started in the field with a Kaupapa Māori foundation and knowing the importance of culture in evaluation practice, Western evaluation practices felt very foreign. Yeah. I was able to see the difference and be like, Oh, actually what I’m being taught here is very different from what I know to be true. And I felt not so much anger as a drive to push for Indigenous thought and evaluation in the field. And that’s what led me to a PhD.

Gladys: Yeah, it sounds like those were two starkly different experiences. But you told me it gave you drive and motivation. What are some of the differences, or what stood out for you as you were in that Western evaluation space versus the experiences you had with Kaupapa Māori? What are the backgrounds and understandings of those ways of doing evaluation that are more in line with community priorities?

Aneta: Yeah, interesting question. I guess what stood out for me was that I was taught – not that this is wrong, but in the master’s I was taught to focus on the program, right? And the program logic, what was happening in the program, how they were using funding and what they were doing and what were the best sort of qualitative, quantitative tools to be able to assess what was going on in the program. And yes, of course that’s important, but what I had learned from my experience with Fiona and also other Māori evaluators in Aotearoa was also the importance of the process involved. You need to have those relationships. Relationships are so important in evaluation, as well as knowledge of culture, values and context, where hearing the experience of an Indigenous person going through a program might mean something completely different to a non-Indigenous evaluator versus an Indigenous evaluator who’s from that shared background to the person that they’re speaking with, if that makes sense.

Gladys: Yeah, that makes sense. With that shared background, there’s an underlying code of what you’re listening for, what you’re listening to learn about.

Aneta: Yeah, and there’s also that trust as well. And something that I’ve found is that people recognize that you get it, and that allows them to feel comfortable and open up a little bit more and entrust you. There’s a lot of trust that goes with people sharing their stories and it’s a privilege to hear it, as we know. So yeah, I think that was one of the biggest things, that what I was learning in this master’s course didn’t feel like I was getting a full education, especially being in Australia and on the land of other Indigenous peoples and that not being brought up at all during this master’s program.

Gladys: It sounds like that experience served as an impetus to push you to where you are today. And I’m wondering if you want to share a little bit about your PhD journey and what you’ve been thinking about and working on.

Aneta: I’m in my fourth year at the moment and looking at Indigenous evaluation frameworks. And part of why I wanted to dive down into this particular area was after having a few conversations – again, one with my aunt, she comes up a lot in my story – having conversations about where we are heading in terms of Kaupapa Māori evaluation and also Indigenous evaluation more broadly. It’s really this idea that we’re at a place where communities want to develop their own frameworks, getting more local, which is a really, really exciting place to be in. So I wanted to look at what’s currently out there, what frameworks have been developed, how they’ve been developed, who was involved, what knowledge was privileged, and what sort of contextual factors influenced the development of these frameworks. I started on this PhD journey a few years ago and ended up really focusing on four frameworks from different parts of the globe: the Ngaa bi nya framework in Australia^[1]; the Aloha framework from Hawaiʻi^[2]; and – I will not be able to pronounce this correctly, but I’m gonna give it a go – Na-gah mo Waasbishkizi Ojijaak Bimise Keetwaatino: Singing White Crane Flying North framework^[3], which as an author and developer you know a lot about, Gladys; and Te Korekoreka framework in Aotearoa.^[4][5] I wanted to look at how and why they were developed, the role of community in the development of the frameworks and how they are being used to support community in their evaluation endeavors.

Gladys: Thank you for sharing a little bit about your journey through your PhD and where you are in this moment. I’m excited to celebrate when you get to the point where you’re all done this work and drawing forward from this reading and reflecting on the learning. I’m wondering if you’re in a space where you might be able to reflect on what you have learned so far about building relationship with these four frameworks. What are you sitting with or thinking about in this moment right now, knowing that you’re not really at the end of your journey yet and things might shift and grow?

Aneta: I’m sitting with how different they are and how important that is, because they’re developed by different Indigenous peoples and each is really grounded in the cultural base that it was developed with, and some sort of reflecting on some of the learnings from that and what that might mean for Indigenous evaluation moving forward. And especially for other communities that might be looking to develop their own framework: what do we really need to consider when we think about evaluation within our own communities? Sure, of course it does need to consider Protocols and also how we conduct evaluation that’s aligned with the culture and priorities of our people. But also, what about these external factors like policies, programs that are being developed for us but not by us? What does that mean for evaluation if we want to do that right? These funding considerations: what does it mean when a lot of government agencies do not see the legitimacy in Indigenous evaluation approaches? How do we speak back to that or rather push back against that?

I also want to acknowledge that it is such a privilege to gain insight into the different lived realities of different groups of Indigenous peoples from across the world. And what a gift that is to have people share their stories so that we can learn from them and craft evaluation tools to better serve our people.

“…what about these external factors like policies, programs that are being developed for us but not by us? What does that mean for evaluation if we want to do that right? These funding considerations: what does it mean when a lot of government agencies do not see the legitimacy in Indigenous evaluation approaches? How do we speak back to that or rather, push back against that?”
_{– Aneta}

Gladys: So beautiful. Thank you for embracing my question when I know from my own experience that it probably is challenging to try and speak about your work when you’re so immersed in it right now.

Aneta: Thank you.

Gladys: And some of the points that you brought forward hint at the answers to this question, but I want to ask you explicitly, why is Indigenous evaluation important? From your experience and from all of the work and the wisdom that you carry forward to where you are right now, what are some of the pieces you would like to share?

Aneta: I think it’s important because it needs to happen, right? <laughs> We live in these Western colonial societies and have lived and are living by an imposed structure, imposed ways of thinking. And Indigenous communities and activists are doing great work pushing back against that and fighting for the right to speak their own language, to have Indigenous histories being taught in schools and added to university curriculum, to have programs being developed and undergirded by cultural practice and knowledge. And I think it’s just right that Indigenous evaluation – well, the field of evaluation – follows suit. We need to have evaluation approaches and evaluation tools that can help to tell the stories of these communities, of these organizations, of these programs in a way that is true to what is actually happening.

Because I think there’s so much that gets missed when the tools, the frameworks or the ideas do not align with what is actually happening. There’s also – I don’t know if the listeners have heard of this – but Nan Wehipeihana created a model looking at the impact that having evaluation that is done onto you can have on a community, on Indigenous communities.^[6] So having a spectrum where it’s from the evaluation that’s done onto you versus when it’s led by Indigenous communities. And when it’s on the other end of the scale, when it’s done onto communities, there is that greater potential for harm to be done, for stories to be misunderstood, which can lead to funding being cut or the process of evaluation leaving people not sitting well. So, I think it’s really important as a way of not only asserting Indigenous ways of thinking in the evaluation space, but also mitigating that continuous harm that can be caused by neo-colonial practices and living in these very colonial Western societies.

Gladys: Thank you for sharing those reflections.

And I think that directs us to this question of whether there’s value. What happens if funders or decision makers don’t understand or believe that Indigenous evaluation is as important? And so there’s a whole necessary undoing of mental models or ways of thinking about Indigenous knowledges, and the field of evaluation as you shared has an important role and accountability. Like in Canada there’s the Truth and Reconciliation Commission, and if we’re looking at reconciliation through evaluation practices, it’s about sovereignty over who is telling the story, the evaluation stories, who is providing the evidence, who is designing the programs in the first place. And it just kind of weaves together for me the paradigm shift that needs to happen in order for so much of our wellbeing and our self-determination and our sovereignty as Indigenous nations globally to be positively impacted. And evaluation has such a significant role in that. And that’s not really something that I understood when I started <laughs> as an undergrad or a master’s student. What are your thoughts on that?

Aneta: Yeah, I didn’t either. And the PhD has been a big, big learning journey, but I think it’s definitely – it feels a little empty sometimes when people state, “We’re going to honor Te Tiriti.” That’s the foundational document for Aotearoa that was signed between the British and Māori leaders back in 1840, which was around partnership. And when you make those statements that the university’s going to align with Te Tiriti and uphold it, and then their actions or their policies go against that, then it just feels like empty words and it’s just that reminder that the fight isn’t over. Even if you might have the Truth and Reconciliation Act, these other government agencies and organizations still need to be told how to do things better and also to step aside as well. There’s still that paternalistic idea that they know best: they’re going to implement policies and programs that will fix the problem that they caused. And yeah, I think it’s that need to recognize that they shouldn’t be leading this work and they need to really take direction from Native people.

Gladys: Mm-hmm. So much more depth needs to happen for sure. So, so much needs to change.

But also so much great work is being done in the realm of Indigenous evaluation. Ideally, what can an Indigenous evaluation look like from design to implementation to end? That’s a really big span, so you may want to jump in at any point there and share some of your reflections: what does this need to look like?

Aneta: Yeah, that’s a tough question because it’s so unique to different people in different places. But from what I’ve learned, if we’re talking about an evaluation that has been commissioned by a government agency, then ideally what that would look like is having the funder being hands-off, there being minimal strings attached to the evaluation, and there being flexibility around funding and time. Because as we know when working with our people sometimes it takes longer, because we need to do it right and it needs to be done in the right way and also, if at all possible, be led by an Indigenous person from the community that they’re working with. But you know, sometimes that’s not possible. So there being a little bit of a partnership approach could also be ideal, whether that’s with an advisory group or supporting evaluation capacity within the community. For instance, bringing people on to help with data collection. Because that’s also a way of putting some of that funding and learning again into the community.

And then, if we’re talking about the actual ideal of what an Indigenous evaluation could look like, it’s – and I think it’s very much an ideal, given the world that we live in – it’s when communities already have those resources and when they have internal evaluation knowledge and capacity or they can have access to an external evaluator that is Indigenous and of their community, where there are those relationships already in place and there’s an intimate knowledge of the place and culture. I think that’s a very important component of good Indigenous evaluation. I guess really an evaluation that is conducted and is grounded in cultural Protocols, is done with great respect, where there is a focus on relationship, and community priorities are privileged.

Gladys: Yes, absolutely. Thank you for tackling that really big <laughs> broad question and bringing it down into the need for local context, the need for resources. I love that you started off with the responsibility of the funder, thinking about how funding expectations and reporting and the strings attached also need to shift in order for Indigenous evaluation to be able to flourish.

Right now, what are you excited about in Indigenous evaluation?“I’m excited about communities developing their own tools for evaluation because it’s quite common now. Nowadays you’ve got communities starting to think, Actually, why don’t we just do this ourselves?”
_{– Aneta}

Aneta: Well, <laughs> this might be an unexpected answer, but I’m excited about frameworks! I’m excited about communities developing their own tools for evaluation because it’s quite common now. Nowadays you’ve got communities starting to think, Actually, why don’t we just do this ourselves? We can do this and we can add our own flavor. So I’m really excited about what has been developed, not only the four frameworks that I mentioned before, but also what isn’t available from a quick Google search, what people are keeping to themselves in terms of developing their own evaluation capacity. And, yeah, I’m excited to see what comes out of that and how our field evolves as we move forward. How we learn from each other and move from that whole, that global space, to more of a local space.

Gladys: Yeah. I’m wondering if you can speak a little bit more about that global to local movement. You know, at the beginning you talked about the importance of the move from the global to the local as Indigenous evaluation continues to evolve and become more honed by local context and languages and land. Do you have any examples of what these practices have looked like at a local level that you can share?

Aneta: Yeah, sure. So here in Aotearoa Indigenous evaluation looks like Kaupapa evaluation, so evaluation that is by Māori, for Māori, with Māori, and as Māori. Where Māori ways of thinking and doing are considered normal, and that’s how evaluation is framed. So that’s been sort of how we’ve thought about evaluation with Māori for at least 15 to 20 years now. And what we’re seeing here is iwi (tribal groups) picking up the mantle or working with evaluators to come up with their own local ways of doing evaluation. One example is Te Korekoreka, one of the frameworks that I’ve included in my research, a framework developed by an iwi group from down south, Kai Tahu group. And they have developed a model that is informed by one of their creation stories. It’s based on the idea that life emerged in three stages.

First it was te kore, which is the nothingness, and then te pō, which is the night, and then te ao, which is the world and life around us. They’ve taken that idea and it’s informed how they’re thinking about not only evaluation, but also program development, how their organization runs. And it’s those phases of learning where you move through te kore then te pō into te ao. You’re moving from these phases of sitting in the darkness and seeing what emerges, what comes forth as you move into te ao, into conceptualizing your thoughts and putting them down. So that’s one example. And this is also an example of a group where they do their own evaluation work as well as work with funders to do evaluation. So it’s really driven by their own priorities and what they want for themselves.

So that, again, it’s having that resource, flexibility of time and funding. What has come out of my research is that need for flexibility especially with deadlines has been really quite important when we are thinking about Indigenous evaluation. But Kai Tahu are also using this framework as a tool to push back against funders, and to educate them on what evaluation looks like within their communities. So for example, if a funder wants X, Y, Z, then Kai Tahu has conversations bringing in their framework and asking, Can we align these two? This is what we prioritize within our organization, and can that align with what you’re expecting from us? So that’s I think a really cool example.

Gladys: Thank you. I like how in the foundation of that framework the expectation is for that funder to enter into a different kind of relationship where there is that conversation about what is going to be measured. And I imagine that there is often some movement that needs to happen from the funder’s perspective about what’s important to measure and how the community might measure that in a way that’s meaningful. Measuring what matters to Indigenous communities can feel really different for funders and funding relationships, I think.

Aneta: Yeah, yeah, absolutely. And I’ve heard this a couple of times, where evaluators talk about how people aren’t talking to them; you know, it’s just, “What’s wrong with these communities? Why aren’t they giving to me? Why aren’t they responding to my tools?” And it’s like, The community is not the problem. It’s that you’re approaching this in the wrong way and you might not be asking the right question. Yeah, for sure.

Gladys: I love that you’re excited about frameworks, and that makes sense. That’s something that you are living within right now, and I’m glad that you’re still excited about the work that you are doing for your PhD.

I wonder if you have anything that you would like to share with emerging Indigenous evaluators? If you were to speak to them in this space right now, what would you love for them to know about their work, about things that you’ve maybe learned along the way?

Aneta: Something about my journey and that I’m very grateful for is that I came into the field through the mentorship and the guidance of my aunt who’s an already established Kaupapa Māori evaluator. So having that support and that guidance has been invaluable. And I know that that’s not the case for a lot of emerging Indigenous evaluators. I think what’s difficult is, especially if you’re coming through university, these Western institutions, it’s hard to unlearn these Western ways of thinking about evaluation. And I guess something that I would like to share with emerging Indigenous evaluators is to find your people. If you can’t, if that’s not other Indigenous evaluators, then your family or people in your community as that reference point, that post that you can always go back to and be like, Ah, right, this is what I’m doing it for. There’s a different way of doing things than what I’ve been taught. And even if you don’t have those people, then going to the literature, having these texts: Linda Tuhiwai Smith’s book Decolonizing Methodologies is always the one for me.^[7] But it’s just that reminder of what we’re doing it for; we’re doing this work for our people and we don’t have to follow these Western approaches in order to do that.

Gladys: Mm, thank you. And I love those words of wisdom: find your people. I know when I’m in a space where I don’t have to start from the beginning, explaining everything that has brought me to the moment of whatever I’m feeling, having a shared understanding or shared context or shared experience of doing work in Indigenous evaluation, and finding your people can feel like a breath of fresh air, can feel like a big hug, can feel like a warm cup of tea together, all of those lovely feelings. So I love that. Thank you for sharing.

Are there any other pieces that you were thinking about that you wanted to draw forward into this conversation? Is there anything else that you want to make sure to share into the space for listeners to think about Indigenous evaluation and this work moving forward into the future?

Aneta: Yeah, it’s something that I reflected over when I started doing my PhD studies. It’s that question: Am I just looking locally within Aotearoa, or do I want to expand and look at frameworks in other Indigenous spaces with other Indigenous communities? I’ve thought a lot about how to do that work, how to engage with different cultural groups that I don’t belong to, how I’m going to approach different people from different cultures, how I’m going to engage with the knowledge that they share with me, how I’m going to engage with the different way that these people see the world. So part of my methodology is called manuhiritanga, which in English means “the way of being a good guest.” And that was my approach to my research, using a Kaupapa Māori methodology.

So taking that a little bit further, really grounding down in who I am as a Māori person, what that means to me as Māori and also as an Indigenous person. And then taking these principles of maanakitanga, which is caring, and whanaungatanga, which is connecting and relationship building, and applying that to how I go about and engage with Indigenous peoples from different parts of the world. I don’t know if that’s enough detail on that. But for other Indigenous evaluators, especially if you find yourself working with cultures or communities that you don’t belong to, I think it is that point of reflecting on who you are as a person within your culture, and definitely also being mindful that you’re stepping into a space where you’re a guest and there are different Protocols to follow. There’s a different way of holding yourself that you need to be mindful of. But yeah, I’ll stop there. Thank you.

Gladys: Thank you for drawing that forward. I’m so glad that you included that in your reflection as well. I know you have a blog post on the American Evaluation Association about being a good guest in different cultural spaces, so I’ll make sure to link that. I’m so happy to have shared the space with you today and to have been able to hear a little bit more about your PhD work and how you came to this journey. Thank you so very much for spending time with me.

Aneta: Oh, thank you. It’s been a real privilege.

Gladys: Wonderful.

Listeners, I’m so glad you spent time with us today, too. I look forward to sharing the space with you again soon. Ekosani.

Footnotes

About the author

name: Aneta Cram, (Ngāti Kahungunu, Ngāti Pahauwera)

institution: Victoria University of Wellington

website: https://www.wgtn.ac.nz/health/research/research-highlights/aneta-cram

Aneta Cram is a doctoral candidate with the School of Health at the Victoria University of Wellington in Aotearoa. Her doctoral research explores what Indigenous evaluation frameworks currently exist, how they were developed and the impact that they are having with the communities that they were developed for in order to provide guidance to support other Indigenous communities in developing their own community-specific evaluation frameworks.