Brian Belcher
Professor
School of Interdisciplinary Studies
Royal Roads University

Abstract

New approaches to research engage with and deliberately contribute to transformations in complex social and natural systems. Such research is problem driven, transcends disciplinary boundaries, and it is often grounded in the belief that knowledge needs to be co-produced through collaboration to effectively support decision making and practical action for sustainable development. Royal Roads University is uniquely equipped for this kind of contribution, with low disciplinary boundaries, faculty and students who blend academic and professional experience, and an emphasis on problem-oriented research. This paper examines the research component of the RRU Learning and Teaching Model at the university. It provides an overview of RRU’s unique research model and develops and discusses a prototype theory of change. Promising new approaches to evaluate transdisciplinary research (TDR) are discussed, including broader principles and criteria of research quality, and approaches that use theories of change (ToC) to identify key stages and changes in a hypothetical change process and seek evidence to test the ToC empirically. A review of 506 abstracts from completed RRU graduate research projects found that a relatively small proportion (<8%) fully apply key principles and criteria of effective TDR. A full assessment of 48 projects selected for having high potential impact scored well on many criteria, but with systematic areas needing strengthening to improve effectiveness. It is intended that this paper will help build a conceptual and theoretical basis for improved research design, monitoring, evaluation, and learning.

*

Introduction

Contemporary social and environmental problems are complex and profoundly challenging. They transcend the traditional disciplinary boundaries that underpin the structure and functioning of many research enterprises (Carew & Wickson, 2010). Increasing calls for knowledge aimed at solving consequential problems, an urgent need for research that considers complex contexts, including interactions between natural and social systems, and increasingly engaged populaces that demand more consultative and participatory approaches have changed the research landscape (Wickson et al., 2006). Gibbons et al. (1994) influentially identified the need for new approaches in their path-breaking book, The new production of knowledge. Numerous authors have proposed “transdisciplinary research” to deal with problems that transcend disciplinary boundaries, are intertwined with sociopolitical context, and engage stakeholders to generate socially relevant and acceptable outcomes (Carew & Wickson, 2010). A new discipline of sustainability science has emerged, with problem-driven approaches that seek to create and apply knowledge in support of decision making for sustainable development, grounded in the belief that such knowledge needs to be co-produced through close collaboration between scholars and practitioners (Holling, 1993; Gunderson & Holling, 2002; Clark & Dickson, 2003; Berkes, 2009).

Royal Roads University (RRU) is uniquely equipped to undertake research of this kind and indeed to take a leadership position. The university has a mission to do teaching and research that contributes to transformation in its students and in the world. The RRU research model aims to be solution-oriented and real-world focused. The research component in the strategy emphasizes “action oriented research as a process of inquiry—students develop meaningful research questions and engage in worthwhile investigations to solve real organizational, community-based, or societal problems” (RRU, 2013, p. 17). This approach complements and is supported by an interdisciplinary curriculum and organization. Guided by the university’s strategic research themes^[1]—innovative learning, thriving organizations, and sustainable societies and communities—RRU research aims to produce relevant knowledge that responds to global, national, community-, and organization-based issues and problems. Unlike traditional universities, RRU is fundamentally interdisciplinary by design. There are just two interdisciplinary faculties (Faculty of Social and Applied Sciences and Faculty of Management) organized into Schools. The faculty, including a large group of associate faculty, bring a rich and deep combination of academic and practical experience and interests. Faculty responsibilities for research and scholarship are broadly defined and faculty performance is evaluated based on a range of criteria that include traditional peer reviewed publications, but also: applied research, community and public contributions, contributions to academic quality, and activities that enhance the transfer and translation of new knowledge into practice, foster knowledge exchange between academic and professional worlds, or encourage integrated research/change processes. The majority of Royal Roads graduate students are employed professionals who also bring a tremendous breadth of experience. They choose to study at RRU to cultivate their skills and knowledge for practical application. As a result, they develop high levels of professional and personal competence and influence that allow them to contribute effectively to change in their organizations or communities.

The combination of low disciplinary boundaries, strong blended academic and professional experience, interests and linkages among both faculty and students, and a deliberate and explicit focus on solution-oriented research should situate RRU well to address contemporary, real-world problems. This claim needs to be tested. As a learning organization, we are interested to know whether and how the research model contributes to change, and to find ways to improve it. We also want to be able to respond to the demands of funders, and society more generally, to demonstrate the value of research in terms of results. Research is notoriously difficult to evaluate because the path from intervention to impact is long and indirect, and all the more so in complex inter- and transdisciplinary approaches (Belcher et al., 2016). An important step to facilitate research evaluation is to clarify the conceptual foundations of the research process and to make it transparent and explicit.

The purpose of this paper is to develop a conceptual framework for evaluating the RRU research environment and research model. It begins with an overview of the RRU research community: the students, faculty, and staff who conduct research at RRU and the organizational and academic context for that research. It reviews the types of research activities done at RRU, and the main areas of thematic focus. It then discusses the range and orientation of research processes followed at RRU, with a strong emphasis on knowledge creation and knowledge co-creation with stakeholders, and on change facilitation. The discussion is organized around a theory of change (ToC) for the overall research and learning process, with attention to the leverage points for change. As a first step towards an empirical evaluation of the RRU research model, we report on the results of a review of RRU graduate student theses. The analysis highlights strengths, but also scope, for improving graduate research design and implementation. The paper concludes with a summary of key lessons and suggestions for next steps.

The RRU Research Community

Student Research

A large proportion of RRU research is done by students as an integral part of their academic programs. Research is highly interdisciplinary^[2] and often transdisciplinary^[3], involving stakeholders in the research design and implementation. Students gain knowledge, practical skills, and experience through applied research, action learning projects, case studies, and other research activities. Student research is intended to respond to global, national, community, and organizational problems. This focus on application and real-world problems is an intentional part of the design of the research courses and related activities for all programs at RRU. Projects range from major papers with a value of 6 credits (e.g., MA in Tourism Management), theses worth 12 credits (e.g., Master’s in Environment Management, MA in Conflict Analysis and Management) to the Doctor of Social Science dissertation (36 credits). A list of RRU academic programs with research components is provided in Appendix 1.

Faculty Research

“Research and scholarship” is one of three main areas of responsibility for RRU faculty (along with “teaching” and “service to the university and to the academic mission”). Research and scholarship is broadly defined as the creation, discovery, integration, synthesis, interpretation, dissemination, and/or application of knowledge related to one’s academic discipline or profession (RRU Faculty Agreement, 2012). This research includes externally-funded projects, supported by Canadian tri-council agencies^[4], private foundations, private companies, international development agencies, and other research, development, and environmental organizations^[5], as well as projects funded internally through RRU Internal Research Grants and Professional Development funds. Other projects are done as contract research, hired by private corporations or public agencies.

There is one research centre at RRU, the Centre for Health Leadership and Research (CHLR), which examines current and emerging challenges related to leadership in the health care system, and three Canada Research Chairs: the Tier 1 CRC in “Sustainability Research Effectiveness” and Tier 2 CRCs in “Innovative Learning and Public Ethnography,” and “Innovative Learning and Technology.” Each chair has its own research program, involving students, research assistants, post-doctoral researchers, and numerous collaborators.

RRU research falls broadly into three strategic research themes. These themes were developed through a consultative process to span individual programs, schools, and faculties. “Sustainable Communities, Livelihoods and the Environment” focuses on resilience, diversity, development, vitality, and the ability to innovate and adapt. “Innovative Learning” focuses on learning that creates opportunities to generate knowledge and to empower; the theme considers learning at the level of society and organizations, as well as the scholarship of teaching and learning. “Thriving Organizations” explores human and operational dimensions, and seeks to foster individual and organizational capabilities in organizations, systems, and sectors.

A schematic overview of the RRU Research Environment is shown as Figure 1.

A Conceptual Framework for Research that Makes a Difference

Faculty and Associate Faculty research supports their work as teachers, keeping abreast of and contributing to theoretical, methodological, and practical issues in their fields of interest. Students’ research projects link systematic inquiry to practical issues and problems, and provide professional context for integrating and applying concepts and skills learned in their programs. The Master’s of Tourism Management (MTM) program puts it nicely, stating that the research components of the academic program help students not only to be critical consumers of research-based knowledge, but also to be capable producers of research-based knowledge (RRU MTM, 2014).

Beyond this important role supporting the RRU educational mission, there is also an explicit intention that RRU research should generate new knowledge and help use that knowledge to solve problems, develop new opportunities, support innovation, and enable change; in other words, to have impact (RRU Office of Research). But how can we know if we are being successful?

In any endeavour, it is necessary to assess progress and to know whether a project is on track to achieving its goals. This facilitates adaptive management, to adjust and improve efforts during a project, and informs better design in subsequent projects. Of all publicly-funded activities, research may be the most difficult to effectively monitor and evaluate because the path from research to impact is long and indirect, especially in complex systems.

Action research, applied research, and inter- and transdisciplinary research approaches of the kind employed in many RRU projects apply a wide range of tools and engage with many actors: researchers from other disciplines, practitioners, policy-makers, civil society actors, and stakeholders. They seek to contribute to change in multiple ways:

Knowledge Contributions

• Identify & draw attention to important issues or problems,
• document a case or set of cases,
• develop conceptual framework for understanding a problem,
• improve theory and methodology,
• provide theoretical and/or empirical analysis of the problem & possible solutions,
• challenge conventional wisdom & myths, and
• provide evidence-based recommendations for improved policy & practice.

Capacity and Process Contributions

• Provide fora and/or facilitate negotiated solutions,
• increase ability of groups to undertake cooperative inquiry into issues of mutual concern,
• build social and scientific capacity to deal with the issue and related issues, and
• influence research agendas through priority setting, methods development, data collection, and publication.

Conventional and widely used measures of “scientific impact” count scientific outputs: journal articles and other publications and citations (e.g., H index, i10 index)^[6]. Notwithstanding some strong criticisms of these kinds of measures (Scott, 2007), they are useful indicators of scholarly influence. However, they are clearly insufficient and inappropriate measures of research effectiveness where research aims to contribute to social learning and change. Such measures only provide an indication of one kind of communication—communication among scientists. They do not measure the usefulness or the effectiveness of the ideas or the recommendations produced by the research, cannot assess the influence of those ideas outside of academia, and usually come too late in the research cycle to allow researchers to adapt their approach to be more effective. For transdisciplinary and applied research, the primary intended audience likely is not a scientific one. The users of such research do not typically publish their experience in the peer-reviewed press, so research of this kind may not be highly cited, even if it is high quality and effective research. Citation records are, in short, an inadequate measure of impact, and evaluations based on publication records offer little formative (learning) value.

New and different ways are needed to monitor and evaluate research, especially transdisciplinary research, to demonstrate value but more importantly, to inform ongoing evolution and improvement of the research process. There is growing interest in “theory of change” approaches (Coryn, Noakes, Westine, & Schroter, 2010; Vogel, 2012). A theory of change is an explicit depiction of the relationships between an intervention (e.g., a research project) and intended results (i.e., outcomes and impacts). Such models have been applied in one form or another in international development and increasingly in the evaluation community generally (Conlin & Stirrat, 2008; White, 2009; Coryn et al., 2011). Theory of Change is also being tested in international research organizations (Belcher et al., 2012; Mayne & Stern, 2013). In the most basic form, a ToC models the stages from project (or program) inputs through outputs, outcomes, and impact. More sophisticated and realistic models include causal mechanisms and assumptions, short- and longer-term outcomes and feedback loops, and reflect changes at different stages in the process as individuals, organizations, systems, and communities engage with and respond to the intervention.

A note on definitions is necessary here, as many evaluation terms are used differently by different authors. Some widely used definitions of outcome and impact (e.g., OECD-DAC 2010)^[7] are based on proximity to the intervention or the time elapsed between intervention and result, leading to ambiguity and confusion. For this discussion we use the following definitions:

Output: The products, goods and services of research and the research process, including knowledge and discourse packaged as publications, presentations, dialogs and discussions, strategies and plans, popular media, and artistic representations.

Outcome: A change in knowledge, attitudes, and skills manifest as a change in behaviour, resulting in whole or in part from the research and related activities.

Impact: A change in state or a change in flow resulting in whole or in part from a chain of events to which research has contributed, directly or indirectly, intended or unintended. These effects can be economic, socio-cultural, institutional, environmental, technological, or of other types.

In a simple example, transdisciplinary research on a watershed management issue might produce an improved model of water flow, document and analyze stakeholder perceptions, values, and aspirations, and create a discussion forum to share information and ideas and promote collective action among stakeholders. These are all outputs. Some outputs may be packaged as research papers, policy briefs, or items in popular media. These are research products. The research process, including but not limited to the dissemination and sharing of research outputs, may influence various stakeholders by changing their knowledge, attitudes, or skills such that they are able and motivated to do something differently. For example, an industrial water user might decide to invest in improved treatment of effluents, or a hydro-electric utility might modify the timing of reservoir drawdowns to accommodate seasonal wildlife habitat requirements. Less directly, advocates armed with new science-based information might lobby legislators for policy change. These are outcomes. If organization or government policy is changed as a result, that is also an outcome. If the resulting actions lead to improved condition (state) of the watershed, or improved water availability (flow) to downstream users, that is an impact. This use of the term “impact” is fundamentally different than the common use in the term “research impact,” which refers to reading and citing of research outputs by other researchers, measured in terms of citation counts, as discussed above.

Using these concepts, RRU’s ToC for research can be modelled several ways. Figure 2 shows an overall ToC for the RRU research portfolio at the scale of the organization. It shows the classic research project cycle in which: 1) a socially relevant problem or issue is identified and defined; 2) a research problem, research question(s), and methods are defined; 3) data are collected and analyzed, and results are communicated, often within a reflexive process and with ongoing or periodic adaptive management; 4) the research process and research outputs contribute to changes in the knowledge, attitudes, and skills of partners, clients, stakeholders, and society more broadly; 5) those actors respond, and longer term organizational development, policy reform, technology development, partnerships, and other system transformations are realized; and 6) changes result in terms of improved community capacity and vitality, improved natural resource or environmental condition, increased economic status, and improved livelihoods. Figure 2 shows these as a series of headings, with examples in the columns of the kinds of actions that are required at each stage. The long bar underlying the main process steps indicates ongoing interactions. Throughout the process, there is a need for ethical and quality management, supported by the RRU ethical review process and guidelines, academic supervision, thesis committees, steering committees, peer review, and stakeholder engagement. Although the diagram is shown as a simple schematic that progresses from left to right, in practice there may be numerous feedback loops, with ongoing adaptive management of project design and implementation. There is also ongoing interaction with teaching and learning. This model illustrates how RRU research collectively creates and co-creates knowledge, and contributes to change.

Figure 3 shows the elements of the research process from an individual researcher or team perspective. The upper pathway (rectangular cells) shows the main elements following a conventional scientific approach, where the research problem is defined in relation to current theoretical and empirical understanding derived from the scientific literature, scientific consultation, and peer review. For students, there are also direct links to course work and academic input from supervisors, committees, and sponsors. There may also be strong links with stakeholders, represented by the lower pathway (oval cells). This can be done at different stages and with different degrees of engagement. Research problem identification and design need to take into account social, economic, and environment context and conditions to make the research socially relevant. This may be achieved through reference to secondary sources, but contemporary theory on research-to-action emphasizes the importance of engaging stakeholders directly in participatory problem definition and research design (Cash et al., 2002; Belcher et al., 2016). In this way, problems are defined, research is conducted, knowledge is generated collaboratively, and uptake is direct—in striking contrast to the classical supply-driven research model.

Note that Figure 3 models the process from the perspective of the research team and it only covers the first three stages shown in Figure 2. Research and related processes are expected to contribute to changes in knowledge, attitudes and skills of partners, stakeholders, and society through communication and sharing of research outputs, and (importantly) through one or more of knowledge, capacity, and process contributions discussed above. In some cases, under the right conditions, system transformation and longer-term outcomes, and social, economic, and environmental impacts may result, but this is by no means guaranteed.

Assessing Transdisciplinary and Applied Research

In evaluating research effectiveness, we need to account for outputs, which is relatively easy. Typical research outputs include journal articles, academic presentations, academic and popular books and book chapters, policy briefs and practice briefs, as well as a range of practical products and processes appropriate to the particular purpose. RRU research produces many privately commissioned studies, with final reports presented in writing and/or in private presentations to client organizations as the main outputs. Researchers can share research-based information by participating in and otherwise contributing to public dialogue, op-eds, interviews and articles in popular media, and through participation and dissemination in social media and other on-line fora. As the variety and reach of popular and social media have expanded in recent years, so too have mechanisms for measuring reach (Piwowar, 2013).

We also need to consider outcomes (at different stages if possible), a substantially more difficult undertaking. Outcomes are often incremental, dispersed and difficult to attribute as there are likely many factors influencing them. Research and other activities done as part of or linked to the research process contribute to change through a range of knowledge, capacity, and process contributions. Mostly it is about stimulating, facilitating, informing, and advising social processes of dialogue, advocacy, and decision making. As Roll-Hansen (2009, p. 3) put it: “Applied science can roughly be understood as the area of intersection between science and politics. It depends highly on advanced scientific knowledge and methods but is dedicated to the solution of practical economic, social and political problems rather than the further development of such knowledge and methods.” RRU research contributes by identifying important social, environmental, economic, or other issues/problems, and in doing so, raises the profile of those issues. The discussion can be facilitated by research that clarifies definitional and conceptual issues, by challenging conventional wisdom or looking at problems in a new way. Science can provide information, but in a political context, not all information is given the same weight or respect. Information needs to be perceived as salient, credible, and legitimate simultaneously for multiple audiences (Cash et al., 2002). This can be aided by involving key stakeholders in research activities. If stakeholders contribute to defining the problem and shaping the research direction, if they are involved in the collection and analysis of data, or even if they are kept informed and aware of the process, they are more inclined to appreciate the results and the recommendations. More profoundly, if stakeholders are involved in the process as participants, the knowledge created is their own knowledge.

Well-designed participatory research on contested issues can also facilitate dialogue and understanding. It can help parties on different sides of an issue to understand and appreciate others’ positions and concerns. It can provide a forum, along with improved information, for negotiations and compromise. Being engaged in the research process, or even just informed by it, can also empower stakeholders to negotiate for their interests more effectively.

Measuring these kinds of outcomes is difficult, and it cannot be done at a global level, as can citation counts. Each case is unique and needs to be assessed in context. There are multiple actors and multiple factors affecting any issue or decision—a research project can contribute in many ways, as discussed above, but there will be many other factors influencing outcomes and it can be difficult (or impossible) to disentangle them. Attribution of a change to a particular activity, such as a research project, is practically difficult. It may also be undesirable in an ethical or political sense. Researchers want to contribute and support change, but cannot and should not take full credit for change that involves many actors and complex social processes.

Impacts often occur over much longer time frames and in the context of multiple other factors in complex systems, and so may be difficult to attribute to the research. By articulating and clarifying the theoretical linkage(s), we can both test the theory and assess whether it is being realized.

Belcher et al. (2016) reviewed the literature on principles and criteria of “transdisciplinary research quality” and identified four main theoretical principles: 1) Relevance (scientific and social), 2) Credibility (mainly scientific, but including consideration of how disciplinary approaches are combined and of reflexivity), 3) Legitimacy (mainly a social/political concept, achieved through transparency and engagement), and 4) Effectiveness (in terms of potential and/or actual outcomes). The first three principles and their associated criteria can be used to evaluate research design ex ante or implementation ex post. Having a deliberate intent to contribute to change, an explicit ToC, engagement with the problem context ,and involvement of stakeholders throughout the research process all figure prominently in the theory.

Methodology for assessing research outcomes (effectiveness) is advancing quickly (Earl, Carden, & Smutylo, 2001; ODI, 2004, 2012; Davies & Dart, 2005; White & Phillips, 2012; Mayne et al., 2013). Modeling the ToC of a research project or program serves as a starting point to identify key actors and expected outcomes. The basic approach, following Mayne (2012), is to model the ToC for a research project and test whether: 1) the expected results occurred, 2) the supporting factors (assumptions in the theory of change) have occurred and provide a reasonable explanation for the results, 3) any other identified supporting factors have been included in the causal logic (thereby potentially revising the theory of change), and 4) plausible rival explanations have been accounted for. The CRC in Sustainability Research Effectiveness program at RRU is applying, testing, and developing this approach in a series of international research case studies. Forthcoming publications present lessons learned in theory-based evaluations of the national and international policy influence of research on forests and climate change (Belcher, Young, & Suryadarma, forthcoming) and of research projects on peatland carbon storage and furniture value chains in Indonesia (Belcher, Suryadarma, & Halimunjaya, forthcoming).

RRU Graduate Student Theses Review

As a first step to assess the RRU research model in practice, we conducted a review of RRU graduate student research projects (henceforth referred to collectively as “theses”) to answer the research question: Does RRU graduate research reflect the espoused RRU model of applied, solution-oriented and real-world focused research?

The objectives were to: apply, test and refine TDR Quality Assessment Framework (QAF) developed by Belcher et al. (2016); select and assess theses with implicit & explicit ToCs; characterize impact pathways of RRU graduate research; and recommend support for more effective graduate research.

The study was done in three steps. First, abstracts of all 492 thesis and major projects for the period 2010 to July 2015 that were available on the RRU thesis database^[8] were reviewed independently by two reviewers to select theses with abstracts that have a theory of change (ToC). In other words, we sought project documents with abstracts that indicate what the research aims to contribute, who will use it, and how the researcher/project aimed to facilitate application and/or utilization of the research. The rationale for this selection process was that theses demonstrating a clear intervention logic would be most likely to realize and to document outcomes. Abstracts that captured all three elements of a TOC were categorized as “explicit,” and those that identified two elements were categorized as “implicit.” Others were classified as having no ToC. This classification yielded 34 theses having either explicit (4) or implicit (30) ToCs, which then proceeded to full thesis review.

Anticipating that the selection process based on abstract review would not capture all highly effective projects, we also solicited recommendations from faculty who, through their roles as teachers and supervisors, are well placed to be aware of projects that had good actual or potential outcomes or impact. We approached all RRU core faculty through notices in the RRU newsletter (“CrossRoads”), direct e-mail, and telephone requests. We received recommendations from 5 faculty which yielded an additional 16 project reports for review. Two of these were not reviewed because informed consent was not obtained. The final set of recommendations included 10 theses, 1 major research project, and 3 Organizational Leadership Projects (OLPs). OLPs were not part of the original abstract review because they are not publicly available on the RRU library database. The 14 recommendations went through the same abstract review and classification process, but all 14 were included in the subsequent full thesis review (i.e., not only those classified as having explicit or implicit ToCs).

Each of the 48 selected dissertations, theses, major projects, and OLPs was then reviewed independently by two reviewers to: 1) classify the ToC as explicit, implicit, or none; and 2) apply the TDR QAF to assess the research in terms of its relevance, credibility, legitimacy, and effectiveness.

The TDR QAF criteria, definitions, rubric statements, and the scoring protocol are provided in Belcher et al. (2016). The QAF is designed to assess research according to its purpose. The objectives, goals, and context of each research project are used to calibrate the scoring. Scoring is based on the examination of evidence within the thesis document, from a high of 2 if a project fully satisfies a criterion to 0 where there is no evidence that the criterion was effectively addressed.

In the context of the RRU research model, we expected to find a substantial proportion of theses that have a deliberate intention to make a contribution to change (as opposed to pure knowledge creation), so the relatively small number classified as having a ToC based on the abstract review (less than 10%) was surprising. However, the selection process based on abstract review may have missed reports that had more fully-fledged ToCs in the main document. This is supported by the fact that the faculty recommendations and full document reviews yielded more “explicit” ToCs than the abstract review alone. Moreover, it has not been common practice to present theories of change in research documentation; it is quite likely that some researchers had plans that were not well described in the research reports.

The application of the QAF to the selected set of theses identified a range of strengths and weaknesses in RRU graduate student research. The reviewed theses generally scored well on most criteria under the principles of “relevance” and “legitimacy.” Specifically, a majority of reports excelled in defining socio-ecological contexts (79.2%), developing socially relevant research questions (79.2%), and appropriately implementing research design (83.3%). Researchers had high engagement with the problem context (77.1%). In other words, researchers were able to clearly and sufficiently define their problem context in a way that demonstrates practical application of the research project, and took into account key factors, needs, and complexities for research design through document review and/or personal communication with key stakeholders. This resulted in project implementation that was appropriate to the context. It reflects the strong engagement many student researchers have with the research issues and the contexts of their research projects. They have personal and professional experience with the problems they are seeking to address and they often have direct links with key actors. Overall, the research reports were able to clearly articulate the importance, significance, and usefulness of the research design (relevance), and fairly and ethically represent stakeholders (legitimacy).

Lower scores on some criteria under the principle of credibility suggest scope for improvement. Notably, there was a tendency for low scores on the criteria of “transferability and generalizability of research findings,” “clear research problem definition,” and “objectives stated and met.” While researchers were able to provide an appropriate breadth and depth of literature and theory from across disciplines in developing their research design, they often failed to discuss how their results are transferable to other contexts by supporting or contradicting the theory that informed the research process. Many projects may be too context specific for direct application of lessons learned in other situations. However, more focused attention to relating individual case studies theoretically to the literature would help to more effectively apply scholarly knowledge to case studies and to link those studies back as learning cases to the scholarly literature.

A more complete presentation of this research is available in Belcher & Soni (2016). It recommends that students should be encouraged to develop explicit theories of change, identifying key audiences and boundary partners, and anticipating change processes. Supporting students at the beginning of their research process to map a theory of change and identify links between research goals (purposes and questions), actions (methods and analysis, collaboration), and outcomes can improve focus and raise the standards for research. Engagement and communication should be supported wherever possible in the research process, when supporting knowledge translation of the research outcomes; simply providing recommendations is insufficient to catalyze change. We also suggest providing more support for improved reporting to better capture research processes and outcomes, including the use of structured abstracts and executive summaries.

This review mainly addresses the potential of RRU student research to contribute to outcomes based on theoretical best practice. As discussed above, there is also a need for empirical assessments of actual outcomes and impacts from RRU research. It would be valuable to include questions about students’ perspectives on research in the various student surveys conducted by RRU (e.g., new student survey, existing student survey, alumni survey). The alumni survey in particular could usefully ask about research purpose, contributions, and outcomes achieved. It would also be valuable to conduct outcome evaluations on a selection of RRU student research projects to assess whether and how the research and associated activities have contributed to change.

Conclusions

The RRU research model has a deliberate and explicit focus on solution-oriented research, and its organization, program design, along with the practical and scholarly experience and interests of its faculty and students, all situate the university well to address contemporary, real-world problems. This paper proposes an analytical perspective to assess whether and to what degree the promise of the RRU research model (or any transdisciplinary research model) is being realized in practise and to identify current strengths and weaknesses as a way to inform and improve teaching and learning at the university and beyond. To that end, the paper develops a conceptual framework for assessing applied, inter- and transdisciplinary research, with a prototype generic theory of change model for RRU research overall, and a more specific ToC for the RRU research and learning process from an individual researcher perspective. These models identify key steps in the research process and the opportunities for engagement, influence, and leverage for social learning and change. They are intended to provide a basis for discussion within the RRU community and can be improved and refined to better reflect the range of approaches and intervention strategies employed by RRU researchers. The model will also facilitate evaluation of RRU research by clarifying the conceptual foundations of the research process.

As a first step toward an evaluation of the RRU research model, we conducted a review of the research approach employed in student research projects completed between 2010 and 2015. We found fewer examples than expected of projects that included deliberate and explicit plans for how the project would contribute to social, environmental, organizational, and economic outcomes. This is at least partly due to our selection process which was based on reviewing abstracts that may not have accurately or fully reflected the research approach. The studies that were selected tended to have strong relevance and legitimacy scores, reflecting the strong engagement many RRU student researchers have with the research issues and the contexts of their research projects. They have personal and professional experience with the problems they are seeking to address with their research and they often have direct links with key actors. The review identified scope for improved attention to some research quality issues, such as clarity in identifying research problems and objectives and in linking theory to practice. It also highlighted the need for improved research reporting, including more accurate abstracts and, in some cases, more accessible reporting (e.g., executive summaries). More fundamentally, it is recommended that RRU research instruction and supervision incorporate lessons and approaches from transdisciplinary research, knowledge translation, sustainability science, and other theory and practice that aims to increasing engagement, utilization, and effectiveness of research. The TDR QAF (Belcher et al., 2016) and similar tools (e.g., Mitchell & Willetts, 2009) can be used as checklists for research design. Deliberate and explicit development of theories of change as part of the research design process can help focus research and develop knowledge translation strategies; they also provide a strong basis for subsequent outcomes assessments as part of a systematic analysis of whether and how RRU research is effectively contributing to meaningful outcomes.

Acknowledgements

The ideas in this paper were improved through discussions with colleagues Katie Rasmussen and Matt Kemshaw in the CRC Sustainability Research Effectiveness program, and with Deborah Zornes and Eileen Piggot-Irvine during the RRU Research Impact Committee work. Katie Rasmussen and Diana Cordoba conducted the master’s project abstract review reported in the paper and Katie also provided a helpful review of a draft version of the manuscript. Since submitting the original manuscript, we have published a paper on the development of a TDR Quality Assessment Framework and applied it in a review of RRU graduate research project reports. Further input was received in this process from Shilpa Soni, Gwen Hill, and Vanessa Rogers. Reviewers Niels Agger-Gupta and Doug Hamilton provided insightful and helpful feedback and suggestions.

References

Belcher, B. & Soni, S. (2016, March) Assessing transdisciplinary “quality” in RRU graduate research. Roads-to-Research Seminar, Victoria, BC.

Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R., & Zornes, D. A. (2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, 25(1), 1-17.

Belcher, B., Place, F., Colomer, J., & Gotor, E. (2012). Monitoring, evaluation, and impact assessment strategy for CRP6 (2012 – 2016): Forests, trees and agroforestry: Livelihoods, landscapes and governance. Internal CRP6 Strategy Document.

Berkes, F. (2009). Indigenous ways of knowing and the study of environmental change. Journal of the Royal Society of New Zealand, 39(4), 151–156.

Carew, A. L., & Wickson, F. (2010). The TD wheel: A heuristic to shape, support and evaluate transdisciplinary research. Futures, 42(10), 1146–1155.

Cash, D., Clark, W., Alcock, F., Eckley, N., & Jäger, J. (2002). Salience, credibility, legitimacy and boundaries: Linking research, assessment and decision making. John F. Kennedy School of Government Harvard University Faculty Research Working Paper Series.

Clark, W. C., & Dickson, N. M. (2003). Sustainability science : The emerging research program. Proceedings of the National Academy of Sciences, 100(14), 8059–8061.

Conlin, S., & Stirrat, R. L. (2008). Current challenges in development evaluation. Evaluation, 14(2), 193–208.

Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schroter, D. C. (2010). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226.

Davies, R., & Dart, J. (2005). The ‘Most Significant Change’ (MSC) technique: A guide to its use. Cambridge, UK. Retrieved from: http://www.mande.co.uk/docs/MSCGuide.pdf

Dixon, A.D. (2009). Organizational leadership and the frontline nurse: Strategies to provide quality, respectful care to verbally abusive clients. Victoria, BC: Royal Roads University.

Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. Ottawa, ON: International Development Research Centre.

Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The new dynamics of science and research in contemporary societies. Stockholm, Sweden: Sage Publications.

Gunderson, L. H., & Holling, C. S. (Eds.). (2002). Panarchy: Understanding transformations in human and natural systems. Washington, DC: Island Press.

Holling, C. S. (1993). Investing in Research for Sustainability. Ecological Applications, 3(4), 552-555.

Kates, R. W., Clark, W. C., Corell, R., Hall, J. M., Jaeger, C. C, Lowe, I., … & Svedin, U. (2001). Sustainability Science. Science, 292(5517), 641–642.

Mayne, J., & Stern, E. (2013). Impact evaluation of natural resource management research programs: A broader view. Canberra, Australia: Australian Centre for International Agricultural Research.

Mayne J., Stern E., & Douthwaite, B. (2013). AAS practice brief: Evaluating natural resource management programs. CGIAR Research Program on Aquatic Agricultural Systems. Penang, Malaysia. Practice Brief: AAS-2013-23.

Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–80.

Mitchell, C., & Willetts, J. (2009). Quality criteria for inter-and trans-disciplinary doctoral research outcomes. Sydney, Australia: Institute for Sustainable Futures, University of Technology, Sydney (UTS).

Overseas Development Institute (ODI). (2004). Bridging research and policy in international development: An analytical and practical framework. ODI Briefing Paper. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/198.pdf

Overseas Development Institute (ODI). (2012). RAPID outcome assessment guide. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/7815.pdf

OECD (2010). Quality standards for development evaluation. Retrieved from http://www.oecd.org/development/evaluation/qualitystandards.pdf

Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.

Royal Roads University. (2012). Collective agreement between Royal Roads University and the RRU Faculty Association April 1, 2012–March 31, 2014.

Royal Roads University. (2013). Learning and teaching model. Retrieved from http://media.royalroads.ca/media/marketing/viewbooks/2013/learning-model/index.html

Royal Roads University. (2014). Master of Arts in Tourism Management. Retrieved from http://royalroads.com/prospective-students/master-arts-tourism-management

Scott, A. (2007). Peer review and the relevance of science. Futures, 39(7), 827–845. doi:10.1016/j.futures.2006.12.009

Vogel, I. (2012). ESPA guide to working with Theory of Change for research projects. Edingurgh: LTS International and ITAD.

White, H. & Phillips, D. (2012). Addressing attribution of cause and effect in small impact evaluations: Towards an integrated framework. International Initiative for Impact Evaluation.Working Paper 15.

White, H. (2009). Theory-based impact evaluation: Principles and practice. Journal of Development Effectiveness, 1, 271-284.

White, H. (2009). Theory-based impact evaluation : Principles and Practice. International Initiative for Impact Evaluation. Working Paper 3, New Delhi: International Initiative for Impact Evaluation.

Wickson, F., Carew, A. L., & Russell, A. W. (2006). Transdisciplinary research: Characteristics, quandaries and quality. Futures, 38(9), 1046–1059.