Using Qualitative Research to Promote Organizational Intelligence -  - ebook

Using Qualitative Research to Promote Organizational Intelligence ebook

104,99 zł


Learn how to use qualitative research as a tool for institutional research. Although institutional researchers often employ informal qualitative methods (their experiences and anecdotal observations), this volume argues that true organizational intelligence is facilitated by formalizing the qualitative data collection and analysis process in institutional research. To that end, it presents a systematic approach to qualitative research in institutional research that connects technical and analytical skills with issues awareness and knowledge of context.Topics covered include: * Technical/analytical awareness practices * Issues awareness * Practices that achieve contextual awareness. This is the 174th volume of this Jossey-Bass quarterly report series. Timely and comprehensive, New Directions for Institutional Research provides planners and administrators in all types of academic institutions with guidelines in such areas as resource coordination, information analysis, program evaluation, and institutional management.

Ebooka przeczytasz w aplikacjach Legimi na:

czytnikach certyfikowanych
przez Legimi

Liczba stron: 195

New Directions for Institutional Research

Gloria Crisp EDITOR

Using Qualitative Research to Promote Organizational Intelligence

Ezekiel Kimball Karla I. Loya EDITORS

Number 174


San Francisco

Using Qualitative Research to Promote Organizational Intelligence

Ezekiel Kimball and Karla I. Loya (eds.)

New Directions for Institutional Research, no. 174

Editor: Gloria Crisp

NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH, (Print ISSN: 0271‐0579; Online ISSN: 1536‐075X), is published quarterly by Wiley Subscription Services, Inc., a Wiley Company, 111 River St., Hoboken, NJ 07030‐5774 USA.

Postmaster: Send all address changes to NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH, John Wiley & Sons Inc., C/O The Sheridan Press, PO Box 465, Hanover, PA 17331 USA.

Copyright and Copying (in any format)

Copyright © 2017 Wiley Periodicals, Inc., a Wiley Company. All rights reserved. No part of this publication may be reproduced, stored or transmitted in any form or by any means without the prior permission in writing from the copyright holder. Authorization to copy items for internal and personal use is granted by the copyright holder for libraries and other users registered with their local Reproduction Rights Organisation (RRO), e.g. Copyright Clearance Center (CCC), 222 Rosewood Drive, Danvers, MA 01923, USA (, provided the appropriate fee is paid directly to the RRO. This consent does not extend to other kinds of copying such as copying for general distribution, for advertising or promotional purposes, for republication, for creating new collective works or for resale. Permissions for such reuse can be obtained using the RightsLink “Request Permissions” link on Wiley Online Library. Special requests should be addressed to: [email protected]

Information for subscribers

NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH is published in 4 issues per year. Institutional subscription prices for 2017 are:

Print & Online: US$461 (US), US$515 (Canada & Mexico), US$561 (Rest of World), €366 (Europe), £2890 (UK). Prices are exclusive of tax. Asia‐Pacific GST, Canadian GST/HST and European VAT will be applied at the appropriate rates. For more information on current tax rates, please go to The price includes online access to the current and all online back‐files to January 1st 2013, where available. For other pricing options, including access information and terms and conditions, please visit

Delivery Terms and Legal Title

Where the subscription price includes print issues and delivery is to the recipient's address, delivery terms are Delivered at Place (DAP); the recipient is responsible for paying any import duty or taxes. Title to all issues transfers FOB our shipping point, freight prepaid. We will endeavour to fulfil claims for missing or damaged copies within six months of publication, within our reasonable discretion and subject to availability.

Back issues: Single issues from current and recent volumes are available at the current single issue price from [email protected]


The Publisher and Editors cannot be held responsible for errors or any consequences arising from the use of information contained in this journal; the views and opinions expressed do not necessarily reflect those of the Publisher and Editors, neither does the publication of advertisements constitute any endorsement by the Publisher and Editors of the products advertised.

Publisher:NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH is published by Wiley Periodicals, Inc., 350 Main St., Malden, MA 02148‐5020.

Journal Customer Services: For ordering information, claims and any enquiry concerning your journal subscription please go to or contact your nearest office.

Americas: Email: [email protected]; Tel: +1 781 388 8598 or +1 800 835 6770 (toll free in the USA & Canada).

Europe, Middle East and Africa: Email: [email protected]; Tel: +44 (0) 1865 778315.

Asia Pacific: Email: [email protected]; Tel: +65 6511 8000.

Japan: For Japanese speaking support, Email: [email protected]

Visit our Online Customer Help available in 7 languages at

Production Editor: Abha Mehta (email: [email protected]).

Wiley's Corporate Citizenship initiative seeks to address the environmental, social, economic, and ethical challenges faced in our business and which are important to our diverse stakeholder groups. Since launching the initiative, we have focused on sharing our content with those in need, enhancing community philanthropy, reducing our carbon impact, creating global guidelines and best practices for paper use, establishing a vendor code of ethics, and engaging our colleagues and other stakeholders in our efforts. Follow our progress at

View this journal online at

Wiley is a founding member of the UN‐backed HINARI, AGORA, and OARE initiatives. They are now collectively known as Research4Life, making online scientific content available free or at nominal cost to researchers in developing countries. Please visit Wiley's Content Access ‐ Corporate Citizenship site:

Address for Editorial Correspondence: Editor‐in chief, Gloria Crisp, NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH, Email: [email protected]

Abstracting and Indexing Services

The Journal is indexed by Academic Search (EBSCO Publishing); Academic Search Alumni Edition (EBSCO Publishing); Academic Search Elite (EBSCO Publishing); Academic Search Premier (EBSCO Publishing); ERA: Educational Research Abstracts Online (T&F); ERIC: Educational Resources Information Center (CSC); Higher Education Abstracts (Claremont Graduate University); Professional Development Collection (EBSCO Publishing).

Cover design: Wiley

Cover Images: © Lava 4 images | Shutterstock

For submission instructions, subscription and all other information visit:

This issue is dedicated to the memory of NDIR Editor, Dr. John Ryan.

THE ASSOCIATION FOR INSTITUTIONAL RESEARCH (AIR) is the world's largest professional association for institutional researchers. The organization provides educational resources, best practices, and professional development opportunities for more than 4,000 members. Its primary purpose is to support members in the process of collecting, analyzing, and converting data into information that supports decision‐making in higher education.


Editors' Notes


1: A Qualitative Toolkit for Institutional Research


Crafting Questions

Overview of Methodologies

Tools for Data Collection

Data Analysis

Research Quality



2: Integrity Is More Than Validity: Seeking Credible, Usable, and Ethical Research

Our Case and Context



3: Using Mixed Methods to Assess Initiatives With Broad-Based Goals

The Challenge With Assessing Broad-Based Initiatives

Key Concepts in Higher Education Program Assessment

An Example: A One-Day Faculty Development Conference

Summative and Formative Results From Our Example

Using This Assessment Method to Inform and Improve Practice


4: Using Qualitative Research to Document Variations in Student Experience

Postpositivist Science and Sampling Methods

Methodological Underpinnings of Issues Intelligence

A More Nuanced Understanding of Disability and Athletics

Concluding Thoughts


5: Context Matters: Using Qualitative Inquiry to Inform Departmental Effectiveness and Student Success

Perspectives on Methodological Pluralism

Deeper Insights Into Departmental Effectiveness and Student Success

Qualitative Inquiry for Contextual Intelligence



6: Cooperative Attention: Using Qualitative Case Studies to Study Peer Institutions

The Divided Brain

Cooperative Attention Framework

Using Case Studies of Peer Institutions in the Cooperative Attention Framework

New-Faculty Orientation: Applying the Cooperative Attention Framework



7: Using Qualitative Inquiry to Promote Organizational Intelligence

Emergence in Institutional Research

Interpretation in Institutional Research

Naturalism in Institutional Research

The Case for Qualitative Inquiry


Order Form


End User License Agreement

List of Illustrations

Chapter 3

Figure 3.1 Sample question from follow-up assessment instrument.

Chapter 5

Figure 5.1 Coded and quantitatively summarized open-ended senior survey responses.

Figure 5.2 A visual illustration of the clusters and frequencies of suggested Gen Ed themes.

Figure 5.3 One-page visual depictions developed to facilitate effective dissemination of qualitative research findings.

Chapter 6

Figure 6.1 Cooperative attention framework.



Table of Contents































































































Editors' Notes

In 2016, the Association for Institutional Research (AIR) released its Statement of Aspirational Practice for Institutional Research (Swing & Ross, 2016a, 2016b). The Statement represents the efforts of the institution (AIR) to respond to the changing needs of postsecondary institutions. Its creation involved more than a year-long process and included an open call for ideas, crafting by six “subject matter experts” (2016b, p. 7), pilot testing at 10 institutions who vetted the statement, and feedback from more than 260 researchers (2016b). The statement release emphasized that some of its elements already exist at many institutions of higher education, but also recognized that much of what is included in the statement remains aspirational for most institutions (2016b).

The Statement suggests a new model of institutional research that moves it away from being the “one source of truth” (2016a, p. 2) and instead incorporates other institutional agents (i.e., students, faculty, and staff) not just as consumers or potential data sources, but also as institutional stewards and decision-makers in their own right. This is an important shift, with institutional research no longer confined to a service unit or office, but rather coaching and interacting with multiple decision makers. Importantly, the Statement proposes a student-focused paradigm, that is, one aimed at improving the student experience by “intentionally grounding institutional research initiatives and reports in a student-focused perspective” (2016b, p. 6, bold in original).

In this volume, we propose that one way to move institutional research (and researchers) toward the aspirational practices presented in the Statement is by more systematically incorporating qualitative research. We posit that qualitative research techniques can add depth to current institutional research practices and promote a holistic understanding of student experiences. Our work is anchored by Terenzini's (1993) definition of institutional research as a form of organizational intelligence and discussion of the various awarenesses required to cultivate organizational intelligence. Terenzini (1993) described organizational intelligence as arising from technical/analytical awareness, issues awareness, and contextual awareness. The technical/analytical domain of organizational intelligence consists of having the requisite methodological skills to construct a systematic research design and execute it with rigor and probity. Issues awareness is based on the ability of the institutional researcher to identify pressing organizational problems and to generate information that can help facilitate decision making related to them. Finally, knowledge of context focuses on the way in which these individual problems fit into the broader complex systems in the institution and the broader ecologies of which the institution is a part.

Individually and collectively the chapters in this volume present the argument that qualitative research can serve as a mechanism to produce organizational intelligence. They also hint at how very difficult it may be to achieve true organizational intelligence without at least some use of qualitative research. Although institutional researchers often employ informal qualitative methods (their experiences and anecdotal observations) to anchor their understandings of issues and context, we argue that the creation of true organizational intelligence is facilitated by formalizing the qualitative data collection and analysis process in institutional research. To that end, we present a systematic approach to qualitative research in institutional research that connects technical and analytical skills with issues awareness and knowledge of context. As a result, we organized the volume following Terenzini's (1993) definition of organizational intelligence. Chapters 1 and 2 describe technical/analytical awareness practices, Chapters 3 and 4 are dedicated to issues awareness, and Chapters 5 and 6 present practices that achieve contextual awareness.

In Chapter 1 (“A Qualitative Toolkit for Institutional Research”), Chrystal A. George Mwangi and Genia M. Bettencourt provide tools, resources, and examples for effectively grounding and conducting qualitative inquiry as a part of institutional research and assessment. The authors provide a review of key qualitative skills and knowledge areas, focusing on research paradigms, specific methodologies and methods, and data analysis. This chapter presents concrete ways for institutional researchers to build qualitative research competencies and to support the development of organizational intelligence in institutional research and assessment through the use of qualitative inquiry among its professionals. In Chapter 2 (“Integrity Is More Than Validity: Seeking Credible, Usable, and Ethical Research”), Sharon F. Rallis and Rachael B. Lawrence present a use-centered understanding of validity that is anchored in practical and responsive use of data. Together, these chapters illustrate practical ways to use qualitative research to enrich common technical and analytical awareness practices in institutional research through the use of alternative epistemological, methodological, and analytic frameworks.

In Chapter 3 (“Using Mixed Methods to Assess Initiatives with Broad-Based Goals”), Karen Kurotsuchi Inkelas describes an innovative, mixed-methods assessment design that can be used to provide evidence of an initiative's effectiveness in changing behaviors, and to inform decision-making for future programming needs. She presents a teaching and pedagogy workshop to exemplify how this type of assessment can yield not only summative but also formative data to inform decision-making. Using examples drawn from a qualitative study of belonging among students with disabilities, Rachel E. Friedensen, Byron P. McCrae, and Ezekiel Kimball demonstrate in Chapter 4 (“Using Qualitative Research to Document Variations in Student Experience”) how qualitative evidence can add depth to quantitative information about what is generally true of all students while also complicating that messaging by describing differences in experiences for a specific group of students. These two chapters exemplify issues-awareness types of organizational intelligence through the use of qualitative data to include and inform multiple institutional decision-makers.

Chapters 5 and 6 provide examples of qualitative practices that reach contextual awareness in institutional research projects that use a qualitative research approach (the case study). Elizabeth A. Williams and Martha L.A. Stassen present a case study describing their three-pronged approach integrating information derived from qualitative inquiry into planning and assessment cycles typically dominated by quantitative information in Chapter 5 (“Context Matters: Using Qualitative Inquiry to Inform Departmental Effectiveness and Student Success”). Using their qualitative inquiry cycle as a guide, the authors describe the creation of contextual awareness as an integral part of effective institutional research work. In Chapter 6 (“Cooperative Attention: Using Qualitative Case Studies to Study Peer Institutions”), Bethany Lisi uses the metaphor of the divided brain to describe the difficult work of promoting contextual awareness as a part of organizational intelligence and to demonstrate the importance of case studies in its cultivation. Lisi uses the analogy of cooperative attention of the two sides of a brain that must work cooperatively to describe both the interplay between technical, issues, and contextual intelligence, as well as the need for institutional researchers to activate both levels of attention—narrow focus on institutional issues and broad awareness of contextual patterns—to maintain organizational intelligence.

In the concluding Chapter 7 (“Using Qualitative Inquiry to Promote Organizational Intelligence”), we (the editors) integrate the arguments and examples deployed throughout the volume into a coherent call for action for institutional and educational researchers to rethink their work and practice to respond to the changing needs of higher education institutions. This chapter illustrates the importance of qualitative research in institutional research practices that respond to the aspirational practices captured in the Statement. This is possible for at least two reasons: First, qualitative research can help institutional researchers capture, include, and communicate the voices, perspectives, and needs of different institutional agents in colleges and universities. Second, qualitative research, framed under Terenzini's (1993) view of institutional research as organizational intelligence, can support the efforts of institutional researchers to gain the knowledge and skills required to push beyond technical/analytical intelligence (first tier) and toward issues (second tier) and contextual (third tier) intelligences.

Karla I. LoyaEzekiel KimballEditors


Swing, R. L., & Ross, L. E. (2016a). A new vision for institutional research.

Change: The Magazine of Higher Learning



(2), 6–13.

Swing, R. L., & Ross, L. E. (2016b).

Statement of aspirational practice for institutional research

(pp. 1–11). Tallahassee, FL: Association for Institutional Research. Retrieved from

Terenzini, P. T. (1993). On the nature of institutional research and skills if requires.

Research in Higher Education



(1), 1–10.




Karla I. Loya

is an assistant professor of educational leadership at the University of Hartford.

Ezekiel Kimball

is an assistant professor of higher education at the University of Massachusetts Amherst.


This chapter provides tools, resources, and examples for engaging qualitative inquiry as a part of institutional research and assessment. It supports the development of individual ability and organizational intelligence in qualitative inquiry.

A Qualitative Toolkit for Institutional Research

Chrystal A. George Mwangi, Genia M. Bettencourt

As an institutional researcher, Sam has just finished analyzing the results of their institutions’ most recent campus climate study. The quantitative findings show clearly that Students of Color have negative experiences both within academic courses and co-curricular involvement. Students of Color responded in high numbers to questions about microaggressions on campus, indicating that these pervasive acts of racism permeate their daily experiences. Students of Color were also more likely to report feeling isolation on campus and dissatisfaction with the institution. Sam wants to know more about microaggressions on campus to be able to understand their different manifestations, the impact they have on Students of Color, and potential strategies for intervention. To meet these goals, Sam decides to conduct qualitative research centered on the voices of these students experiencing microaggressions.

Qualitative research is the result of many different decisions, all of which are made within unique contexts. To illustrate these decisions and contexts, we use the example of Sam throughout this chapter. Like Sam, many institutional researchers find they need to integrate traditionally quantitative approaches with qualitative methodologies to obtain the full picture of student experiences in higher education. Qualitative methods naturally align with institutional inquiry that focuses on students’ experiences within a certain context or set of conditions (Harper & Kuh, 2007). As institutions engage in increasingly complex data-driven decision-making, “the best decisions are based on a deeper understanding than quantitative methods alone can provide” (Van Note Chism & Banta, 2007, p. 15). As such, it is crucial for institutional researchers and institutional research offices to develop qualitative expertise to support methodologies and methods that can be applied to a spectrum of research questions (McLaughlin, McLaughlin, & Muffo, 2001). This chapter provides tools, resources, and examples for effectively grounding and conducting qualitative inquiry as a part of institutional research and assessment. We review key qualitative skills and knowledge areas such as research paradigms, methodologies, and methods.


Paradigms, also known as worldviews, are “systems of beliefs and practices that influence how researchers select both the questions they study and methods that they use to study them” (Morgan, 2007, p. 50). All types of research are rooted in researchers’ paradigms. Paradigms emerge out of researchers’ epistemology, ontology, and axiology, shaping how knowledge is sought out and interpreted. These approaches shape the choices a researcher makes in what and how to pursue their topic.

Although there are multiple classifications of paradigms, for simplicity, we utilize four overarching categories (Creswell, 2014; Mertens, 2015):

The positivist paradigm focuses on explaining, testing, and predicting phenomena (Guido, Chávez, & Lincoln, 2010). Information is objective and value-free, and exists within one true reality. This paradigm has evolved into postpositivism by incorporating a more critical lens to examine how a cause determines an effect or outcome (Creswell, 2014). In the former, a researcher might conduct a study to prove a hypothesis is correct and to discover the truth. In the latter, researchers aim to reject a null (false) hypothesis to move closer to the truth.

The constructivist, or interpretive, paradigm views knowledge as socially constructed and individuals’ experiences as framed by their unique context. Individuals have a subjective reality based on understanding their views (Creswell, 2014). Instead of a universal Truth, there are only truths that exist for individuals that are reliant on their context and time (Guido et al., 2010).

The critical, or transformative, paradigm can incorporate numerous theories that examine the experiences of marginalized individuals and unequal distributions of power. This approach tends to emphasize collaborative research processes to avoid perpetuating power imbalances (Creswell, 2014). These approaches look to restructure the status quo, with the goal of social change. Critical designs may utilize nonhierarchical methodologies that aim to involve participants as co-researchers on investigating a problem and implementing change, such as participatory action research. More widely, critical researchers also cite this paradigm as a way of interpreting results.

The pragmatic paradigm emphasizes that researchers choose the methods, processes, and tools that best answer the research question at hand (Creswell, 2014). Pragmatic paradigms are most commonly associated with mixed-methods research.

Sam is interested in engaging in-depth with student voices and experiences, to understand how their experiences on campus are informed by their interactions with others, their daily lives, and their social identities. As such, Sam identifies that their research is rooted in a constructivist paradigm that prioritizes the context of diverse groups of students to learn more about their experiences and perspectives.

Crafting Questions

Qualitative data can provide a great deal of information, some of which may be beyond the scope and nature of what the researcher wants to investigate. Like research paradigms, crafting a research question(s) helps to constrain the scope of a study. Research questions provide guidance for one's inquiry and require a response that emerges from data and analysis. When a study becomes overwhelming, it is important to remember that a primary goal is to answer the research question(s). Good research questions stem from the purpose of the study. Consider whether the research purpose is to describe a phenomenon or explain and theorize about it (Marshall & Rossman, 2006). Is it to explore a problem that has not been previously examined or to empower others and create greater equity (Marshall & Rossman, 2006)? Answering these can help determine how to craft the research question(s). The methodology is another way to help develop the research question(s). For example, an ethnographic study often incorporates a question about culture. Similarly, a theoretical/conceptual framework may also influence the nature of the question(s).