You Wouldn’t Pick That Up on the Questionnaire

Overview

The Interview

Gladys Rowe: Tansi, greetings. I’m so grateful you are here. What is Indigenous Evaluation? Who is doing this work? How are we doing this work? It is my hope that this podcast will feel like a deep breath, will feel like a space where you can listen and learn, where I invite you to grab a cozy beverage and settle in. Join me and my guests as we open up our evaluation bundles to share the gifts, knowledges and hopes that we’ve gathered in our journeys and bring them together in this space. I hope in these stories you will find resonance in the critical contributions that Indigenous evaluation can make as we work towards decolonial futures and strengthening Indigenous resurgence.

I am here today with Larry Bremner. He’s the president and partner at Proactive Information Services Inc., which he established in 1984 to provide social research services to the not-for-profit and public sectors. Larry’s worked throughout Canada, including northern and Indigenous communities, as well as in Europe and Mongolia. He’s recognized for his methodological expertise and his ability to synthesize learnings from his many experiences, bringing significant insights to each new project. Larry was elected national president of the Canadian Evaluation Society in 2012. Later he represented CES as past president on the international stage, as treasurer of the International Organization for Cooperation and Evaluation, and as member of EvalPartners Management Group. Larry was the driving force behind the creation of the global EvalPartners network, EvalIndigenous. He was given the Canadian Evaluation Society Service Award in 2017 and the Prestigious Contribution to Evaluation in Canada Award in 2018. He was inducted into the Fellowship of Canadian Evaluation Society in 2019. Recently Larry Bremner became the co-editor for the new section of the Canadian Journal of Program Evaluation, Roots and Relations. The purpose of Roots and Relations is “to honour our lineage, grow our kinship, and sustain our intergenerational legacies of Indigenous wisdom practices in and through evaluation.”^[1]

I’m so grateful to have you here with me today. Larry, welcome.

Larry Bremner: Thank you. Thank you very much.

Gladys: I’m wondering if you wanted to start off this conversation, beyond all of the accolades that I just shared, with an introduction, a little bit about who you are and where you come from.

Larry: Sure. First of all, thank you for having me. I really appreciate the opportunity. I live, work, and learn on the ceded and unceded traditional and ancestral land of Indigenous people of North America. Currently I’m on Treaty One territory on the shores of Lake Winnipeg and the ancestral land of the Red River Métis. I am Métis. I was born in Prince Albert, Saskatchewan. My great-grandmother Rose Boucher was born in St. Francois Xavier in what is now Manitoba, and she moved to Saskatchewan in 1883 and married my great-grandfather Moise Bremner.  In 1883, Moise and his father William were two of 30 Métis that signed a petition sent to the federal government protesting the giving of Métis land to the Prince Albert Colonization company. In 1885 the resistance at Batoche took place and Moise fought in one of Gabrielle Dumont’s 19 dizaines. When the battle didn’t go the way they had hoped, they moved to the United States for a time and came back to Saskatchewan in 1905, where they homesteaded. That’s a little bit about me.

Gladys: Wow. There’s so much that can be learned from the ways that the generations before us have resisted and fought for Indigenous rights and upheld relationship to the lands that we’re from. So thank you for sharing that story. I appreciate it. Indigenous evaluation connects to a lot of the themes you just shared around resistance and relationship to land and connection and belonging.

I wanted to ask, how long have you been working in this area of Indigenous evaluation? And when you think about Indigenous evaluation, what does that mean for you specifically? What does that look like?

Larry: You mentioned I started the company in 1984, and I started working in Indigenous evaluation in the late 1980s. From my perspective, Indigenous evaluation is about social, environmental, and economic justice for Indigenous people. I believe for too long the evaluation agenda has been set by external forces not taking into account the realities – the priorities, political realities and economic realities – of the communities in which the programs and evaluations are being implemented. What I would like evaluation to become in Indigenous communities is an exercise that is controlled by the community. The community sets the evaluation agenda, the evaluation addresses community priorities, and it is judged by community values and standards. The community is involved in the whole evaluation process, and it becomes not something that is done to us by outsiders, but is done by us, with us, and for us. I really believe it is important that Indigenous people start taking control of their own evaluation agenda.

“From my perspective, Indigenous evaluation is about social, environmental, and economic justice for Indigenous people.”
_{– Larry}

Gladys: Yeah, definitely. In being involved in Indigenous evaluation from the late 1980s until now, you must have seen some shifts in power, in funding, in the way Indigenous evaluation has been undertaken. Could you share some of the shifts you’ve seen?

Larry: One of the things is we were not talking about Indigenous evaluation in the late 1980s, early 1990s. Today, it is something that is talked about in most evaluation circles on a fairly regular basis. Lately, I believe we have started recognizing the importance of story as a legitimate evaluation method and as an authentic method in Indigenous communities. I believe this has happened fairly recently. Story was not an accepted methodological approach in the early 1980s or 1990s. So I think in some ways there is a recognition that the Euro-Western approaches are not particularly relevant in Indigenous communities, and in fact have done a lot of harm to Indigenous communities.

You might have heard me say this before, but I always argue there have been hundreds if not thousands of evaluation and research projects done in Indigenous communities that have been reliable and valid, and nothing has changed for the better in those communities.

I would rather have an approach that authentically reflects the community than one that is reliable and valid but has no chance of improving life on the ground in those communities.

Also there is a greater recognition now of the whole notion of reciprocity. Rather than evaluation and research just extracting knowledge from communities and returning nothing, there is a recognition that evaluation is a reciprocal process, and if in fact the community does not benefit from this process, the community should not be involved in the process. So I think these are examples of how things have started to change.

One of the other things I worry about is that, because people are becoming aware of the importance of Indigenous evaluation, they start changing their language to better fit in with the notion of Indigenous evaluation. However, while they have changed the language, they haven’t really changed their practice, they are just calling it something different.

Gladys: Definitely. You’ve touched on some really important points that I’ve also been seeing as excitement builds around this thing called Indigenous evaluation. With that excitement about research and evaluation, there’s an increase in funding and interest and resources. But the significant caution there, like you said, is that the language is changing, but the practices remain the same. And so it’s a re-colonizing experience that’s happening under the guise of Indigenous evaluation. There’s really a lot of questions that need to be asked as we think about Indigenous evaluation, as organizations think about implementing Indigenous evaluation.

What are the roots that you need to make sure this is truly evaluation that’s grounded in Indigenous ways of knowing, being, and doing, that centers the voices and the priorities of community? You talked about some of those values, such as reciprocity: What does that look like, when reciprocity is engaged as a value of Indigenous evaluation?

Larry: I’ll answer that, but one of the things is, I was working in a community once and one of the Elders said to me, “Larry, we don’t have a word for evaluation in our language.” Another time, I was working in a community that was supposed to be looking at individual skills, knowledge, and attitudes and a person said to me, “Larry, we don’t have a word in our language for attitudes.” It strikes me that we need to change the language of evaluation so that it accurately reflects the language of the community. The language being used in a lot of evaluations and research projects has no meaning to people in the community. People in the community are not really sure what you’re talking about.

In terms of reciprocity, one example is I did an evaluation a couple years ago on a land-based wellness camp in Yellowknife in northern Canada. And I think one of the reasons they hired me was because I had been involved in other evaluations in Indigenous communities and the people at the camp remembered me. The wellness camp had been in effect for about a year, and the client wanted to know if in fact they were achieving the kinds of things that they were hoping to. While they wanted some numbers, we also took into account the realities of the community. These were mainly homeless men with literacy challenges. So we developed some simple visuals that people could look at, but the important thing was I spent quite a bit of time on the ground in the camp becoming a face that was recognized and trusted and I became involved in the stories.

And as an Indigenous evaluator – I think it’s Shawn Wilson who says, You become part of the story. You do not remove yourself and just listen to somebody’s story. It is also your story and you have to tell part of it yourself.^[2] And through this evaluation, it gave back to the camp in terms of the camp got more resources and expanded. The whole evaluation, the whole notion of that evaluation, was for the camp, and it was actually done by the camp. I remember interviewing a government person at the time and he said, “This is fantastic! Usually we have to really work hard to get people to undertake evaluation. This is unique in that the camp itself wants to see how they’re doing and how they can improve what they’re doing for the people they’re serving.” So for me, it was a really lovely exercise and a really meaningful exercise.

Gladys: Thank you for sharing that story. It really illustrates the shift that happens when something is grounded and sprouts from community priorities and community interests. Then there’s some impetus and momentum there like that government official noticed. It was a very different qualitative experience in working with community in the way that you talked about.

Larry: One of the important things about that camp – there are a number of important things! – was its setting. Many of the people that came to the camp were originally from remote small communities. Being able to have a wellness camp set in the woods away from the road, they could hear the crunch of gravel when they walked to the camp. It brought them back to their community – so many of them told me how much they missed their home community and coming to the camp was sort of like coming back to part of their community.

I remember one day when an individual showed up with remnants of clothing from one of her children that had died the year before. The Elder started a fire and went with the person, and they burnt the clothes and prayed. The Elder was also a traditional healer. I didn’t go – that was not appropriate for me to go there – but I was sitting in the camp when it was finished, and the woman came back, saying because of this ceremony she finally had closure on her child’s death …. I saw this because I was at the camp. You wouldn’t pick that up on the questionnaire. So it’s really important for evaluators to understand that you need to be on the ground, you need to be in the camp, you need to be in the community. You need to walk around the community and become recognized as a face in that community that can be trusted.

So anyway, I’m sorry – I’m a storyteller and I start telling stories.

Gladys: <laughs> Beautiful, beautiful. Thank you. It connects to one of the points you made earlier about the use of story in Indigenous evaluation and how central that is to the way of showing up in spaces, the way of gathering knowledges about impact, and then also the way that you share, as the evaluator and as the community, the story of that program or the story of the transformation that’s taking place. You quoted Shawn Wilson, so I’m going to quote him too: If research doesn’t change you, then you’re doing it wrong. Also, I would posit: If evaluation doesn’t change you, then you’re doing it wrong.

That space sounds so powerful, and you’re there in community, in alignment with the vision of the work that they’re doing.

Larry: Exactly. One of the things that people have to understand is that you might have a wonderful evaluation framework that you drew up in the office. However, if you take the time to go to the community, you will come to some realization that the evaluation framework that made perfect sense in the office is completely inappropriate in the community. One of the keys is being able to be flexible when you go into the communities to adapt to the situations as they unfold.

You have to realize that while you are in community, you are taking people’s time, which they might not have at that particular moment to give you.

So you have to be willing to go in to community. A number of years ago we were looking at schools in the Yukon and we wanted to go to communities that would send their children into Whitehorse to go to school. So we flew to a remote community in the Arctic Circle and got off the plane and got on the quad. We talked to an Elder and said, We would like to interview some people about school and we would like to talk to people in the band office. How do we do that? He said, “Well, you walk in the band office and if the door’s open, you introduce yourself and they might talk to you and if their door’s closed, don’t bother knocking.” We agreed and we did that. We said, We would like to talk to some people in the community. He said, “Well, start walking around. You see people walking around, ask them if they’ll talk to you.” We did that, and we ended up going to a community feast. At that time they were taking a visiting group of exchange students, in boats, to their traditional home down the Porcupine River. So we went and talked to community students in the boats and at the fish fry. It’s becoming part of the community in a way that allows people a comfort with you and you a comfort with them. That is not something that you can really delineate in a clear sense in an office if you don’t have an understanding of the context in which this is happening.

“One of the keys is being able to be flexible when you go into the communities to adapt to the situations as they unfold.”
 _{–  Larry}Gladys: Yeah, application looks different once you get into those spaces.

I wanted to go back to your point around needing to change the language of evaluation. You said at the beginning, a study or evaluation could be deemed reliable or valid, but it doesn’t actually make a difference. And so, in shifting the language of evaluation, I’m wondering if you had a story or an example about what reliability or validity would look like from an Indigenous perspective.

Larry: We have been talking about authentic evaluation. We were in a project once working with some Indigenous communities and the non-Indigenous client wanted a theory of change and a logic model. The Indigenous people working with us said, “Larry, theory of change, what are you talking about? That’s not our language.” So what we came up with was something called a pathway. I think we have to take into account that culturally evaluation (as well as research) is based on this sort of linear process, and that isn’t really a process that is a reality in Indigenous communities. It is also a process that doesn’t take into account the importance of spirituality and regalia, right? I was thinking last night, watching some of the news on the Queen [Elizabeth II] being buried, about the combination of spirituality, tradition, and regalia that we saw on display through this whole process! And yet it is something that’s been completely ignored when researchers go into Indigenous communities. I found it so interesting in terms of how important that whole process was yesterday to the burial of the Queen. And yet when the evaluation process is developed and conceived from outside perspectives looking into Indigenous communities, the relationality of spirituality and regalia is completely underappreciated or disregarded.

Gladys: Yes, whose knowledges are prioritized? Whose knowledges are valued? Indigenous evaluation: It’s political, it’s about power, it’s about the right to self-determination and self-governance and all of those interconnected things that look at how we produce knowledge. How do we say what is actually knowledge? And that might be one of the biggest challenges in Indigenous evaluation, that push to build awareness and to build authenticity around these valid ways of knowing, being, and doing. And the insights that come from Indigenous evaluation are just as important as those Western scientific tools are in Western frameworks. And our ways of knowing, being and doing are central to the health and wellness of our communities.

Larry: I agree! I think there are a number of challenges right now. This goes back to the idea of cultural competency and I worry that we are going to have non-Indigenous individuals graduating from universities, having gone through some courses where they feel culturally competent and they have the language now and they go into communities and basically do a lot of the same things that have been done for years. They say they are “Indigenizing” the process or approach but basically evaluation and research continues to be a tool of colonization.

And one of the things that I really want is to have a panel at one of the Canadian Evaluation Society conferences on the difference between sharing and taking. Because there seems to be – I know a number of colleagues who are Indigenous who are continually approached by non-Indigenous people saying, “Could you give me an idea of what you’d do in a situation like this?” I don’t mind sharing, but sometimes people will use the approach or your suggestions, and then you don’t get credit for it or basically they have taken knowledge again. So I think it’s an interesting question: What is the difference between sharing knowledge and taking knowledge? I know a colleague of mine and I were asked to write a chapter in a book on ethics and evaluation, and we withdrew because a publisher wouldn’t give us the rights of ownership. <laughs> There is this conflict about being able to own what you produce in terms of knowledge sharing. I think publishers have to start understanding that from an Indigenous perspective: We can share knowledge through articles and through publications, but we also want to retain the ownership of that article, the knowledge that we have shared with people. So I think those are some of the other challenges right now.

Gladys: Yeah, when you offer insights or advice or potential approaches and someone uses it without the roots of those approaches or ways of knowing, it’s appropriation all over again, like you said.  And there is a danger in people being able to take training within educational institutions and then have this certificate that says, “Well, I’m culturally competent and so therefore I should be able to do these evaluations with Indigenous communities, organizations, peoples, et cetera.” So that leads me to the question of who should be doing this work and how do we support the people who should be doing it? How do we support emerging Indigenous evaluators and strengthen that field?

Larry: We cannot underestimate the importance of having allies. For a number of years now, myself and a colleague have been working with an ally who argues for, trains for, shares for the importance of understanding Indigenous approaches to evaluation and the importance of funding Indigenous evaluators, the importance of training Indigenous evaluators and communities. That person has a voice at the table that a lot of Indigenous evaluators wouldn’t have and don’t have. So I think having allies is really important to help ensure we get our seat at the table.

I had an interesting discussion the other day about having a seat at the table. Who is setting the table? The table in this case is either a federal, provincial, or territorial government department saying, “Okay, we are going to have consultations on this, and we want to have a couple people from the department and a couple people from the community. Oh yeah, we need a couple of Indigenous representatives, and we will talk about whatever the topic’s going to be.”

At some point, I argue, the table should be set by the community. I have worked with some communities and federal government departments about how to work together, nation to nation. I try to make the point that working together nation to nation is a starting point. For me, the endpoint is the Indigenous nation deciding what nation they’re going to work with, and it might be the federal government or it might be another Indigenous nation without the federal government. So I think it’s important right now in the federal government and in provincial and territorial governments to have allies that make sure that our voices are heard at the table and that advocate for the importance and inclusion of Indigenous approaches to evaluation when they’re looking at programs.

Gladys: That’s such a good point. I talk often about the relationships and the responsibilities that we hold, we being everyone in Canada, Indigenous and non-Indigenous peoples. What are the responsibilities that we hold? And the reality is a lot of Indigenous peoples aren’t at those decision-making tables, aren’t in those spaces. And so what are the roles of allies? I love the connection there: Whose table is it, who is setting this table? That’s really such an important shift in thinking that we need to work from.

Larry: Education is also important. We always talk about the importance – in education circles, we always talk about the importance of learning on the land, education where the learner is meaningfully involved in his or her own education. And I wonder sometimes, if we’re talking about Indigenous approaches and post-secondary education training in evaluation, if there should be mentorship working in communities for part of the learning experience. If it’s strictly based in the classroom at university or college, there’s an important aspect – if we’re talking about Indigenous evaluation, there’s a very, very important aspect – that’s missing, and that is the community and the importance of community in the whole process. You can talk about it in an office or on Zoom, but until you are in the community, you do not understand the different protocols and the different realities and the whole vibrancy of the community and how it operates. That has a huge impact on decisions regarding what is appropriate in terms of approach and involvement.

Gladys: Yeah, one of the things that I’ve been thinking about is, How do we as Indigenous evaluators learn to do this work? I have a masters of social work, and I did take one evaluation course in my program that was focused on policy and administration. So that gave me some insight into the “typical” way of doing evaluation. But how I learned to do this work was in community, with Knowledge Keepers, with mentors, with Elders, with community leaders and organizations who are working in a way that actually was in resistance to all of the things that I learned in my social work degree! <laughs> And so I really love that point that you make around the active participation in community. From my perspective, that’s one of the most valuable learnings over my years.

There’s a lot of excitement, like you said, about Indigenous evaluation, and there’s so many requests that come to all of the different people I know who are doing this work, and it just feels like there’s not enough of us. Do you have any insights that you’d like to share with people who are new to doing this work?

Larry: One of the realities we deal with is artificial timelines and artificial funding cycles. There is a reality to government departments and to NGOs that have to apply for funding, and to funders themselves. Emerging evaluators have to understand that, if we are talking about Indigenous evaluation, good evaluation takes time. It takes time that needs to be spent in the community. It takes time to listen. It takes time to be understanding and respectful, those are things that you hear about in university and college. While it is hard, it takes time to let people think about an answer when you are asking them a question about a program in their community. While the program is top of mind for you, it may not be for them. It is hard sometimes to wait, to wait, to listen for the answer. So, my advice to emerging evaluators would be going back to something you said about coming out of school with tools. Some of those tools are inappropriate for the work that you are going to do if you want to do it in a meaningful and authentic way. You have to keep your mind open to the fact that when you go into community this is a journey of learning for you as well. What you learn might be that a lot of the things that you thought you knew because of your education, you don’t understand, and you do not know. Your real learning is going to take place through this journey called evaluation.

“Emerging evaluators have to understand that, if we are talking about Indigenous evaluation, good evaluation takes time.”
 _{–  Larry}

Gladys: I was sitting here laughing in my head thinking, Geez, yeah, that brings me back to being fresh in community! And still today I learn so much from the communities that I work with, and I don’t know what I don’t know. And sometimes that becomes really clear and then I pick up that learning and I carry it forward.

Larry: I always say that as evaluators, the tools in our toolbox are only limited by our imagination. So you have to go there and you have to be in community, and you have to get excited and say, Hey, maybe what I could do is use poetry, visuals, garbage cans, whatever! You are limited by your imagination, and your imagination sometimes is limited by what you have learned in school. So while that is part of your learning, your real learning is going to happen when you are in the community and you start thinking, Yeah, there’s more to evaluation than questionnaires and focus groups, or even talking circles. There is a lot more to evaluation than that.

Gladys: Oh my goodness, there are so many ways to sit in, to gather, to be with story. But the talking circle is the go-to method now. If you want an Indigenous evaluation, you go run a talking circle. But there’s so many different ways of being in story, and sometimes it doesn’t involve actually having a conversation. Sometimes it’s being in a space, as someone who is just sitting and observing.

Larry: I’ve often said as evaluators and researchers we have responsibilities, and one of the important responsibilities is to educate the funder. Before COVID, we had a provincial government department send us a request for proposal. I looked through it and it was a standard template provincial government request for proposal, but they put in one sentence that said it must include an Indigenous approach to evaluation. The evaluation was to start in January, and to be completed by the end of March, government year-end. So I wrote them a long letter saying we were not interested, explaining what Indigenous approaches to evaluation are, and if in fact they were serious about using Indigenous approaches, they would have adjusted the timeline to reflect their seriousness. It was basically the same old approach. That is one of the things funders need to take into account: If they are serious about looking at Indigenous approaches to evaluation in particular communities, their seriousness is going to be reflected by their timeline and their funding model. If it is the same old timeline, the same old funding model, then they are just spouting new verbiage and are not serious about changing their approaches.

Gladys: Absolutely. The cookie-cutter addition of the last sentence – “Oh, yeah, and please include an Indigenous approach” – when that should have been the very first framing concept in the very first paragraph.

Larry: Yeah, exactly.

Gladys: And that’s reflected in everything coming afterwards, as you said.

Well, I’m so grateful to have spent this time with you today and to have heard more about some of your experiences in Indigenous evaluation. I’m wondering if there’s any final thoughts or reflections you wanted to share before we close.

Larry: If people who listen to your podcast are considering becoming evaluators, I would highly recommend that they seriously consider evaluation as a vocation. It has been just a wonderful life experience for me, but it also has helped me understand the important impact we can make in community. I have always said, I can’t change the world, but if I can change one little bit of the world, I think that’s really important. For a young or emerging evaluator, this is an opportunity for you to change one little bit of the world. So take that opportunity and run with it because it is probably the most important opportunity you will have in your life.

Gladys: Powerful words. Ekosi, thanks so much, Larry.

Larry: Okay, thank you, Gladys.

Footnotes

About the author

name: Larry Bremner, (Métis)

institution: Proactive Information Services Inc.

Larry has worked in evaluation for 45+ years. In 1984, he established Proactive Information Services Inc. to provide evaluation services for the not-for-profit and public sectors. He is recognized for his advocacy and expertise in Indigenous approaches to evaluation. As part of his evaluation journey, he has worked throughout Canada, including Yukon, Northwest Territories, and Nunavut, as well as in East Central and Southeastern Europe and Mongolia. In 2012, Larry was elected National President of the Canadian Evaluation Society. As Past President, he represented CES on the international stage where he was the driving force behind the creation and first Chair of the global EvalPartners network EvalIndigenous. In 2017, Larry received the CES Service Award and in 2018 the Contribution to Evaluation in Canada Award. In 2019, he was inducted as a Fellow of the Canadian Evaluation Society, the Society’s highest honour. Larry is proud of his Métis heritage and is passionate about issues of equity and access. He believes too many evaluations focus on programs whose ‘beneficiaries’ (individuals, communities) are seen as having needs or deficits, rather than recognizing and building on their strengths and assets. As evaluators, Larry believes we are compelled to expand our future to one that is inclusive in terms of both voices and approaches, if we are to support reconciliation and address the crucial social, environmental, and economic issues that we face in today’s world. We must create a future where evaluation not only benefits Indigenous communities, but also where Indigenous communities are in control of their own evaluation agendas.