Grantham University Using Both Quantitative and Qualitative Methods Discussion
I need support with this Health & Medical question so I can learn better.
Discussion: Using Both Quantitative and Qualitative Methods
Many clinical practice issues are sufficiently complicated that neither a purely quantitative or qualitative approach can generate a comprehensive perspective of the issue. Particularly for investigating evaluative questions—such as the effectiveness of a program or treatment or the impact of a policy—some combination of quantitative and qualitative methods can be much more illuminating than relying on one method alone.
In this Discussion, consider the use of a mixed-methods design in healthcare research studies.
Copyright © eContent Management Pty Ltd. International Journal of Multiple Research Approaches (2009) 3: 114–139. Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches ANTHONY J ONWUEGBUZIE Department of Educational Leadership & Counseling, Sam Houston State University, Huntsville TX, USA R BURKE JOHNSON College of Education, University of South Alabama, Mobile AL, USA K ATHLEEN MT COLLINS Department of Curriculum & Instruction, University of Arkansas at Fayetteville, Fayetteville AR, USA ABSTRACT We provide a philosophical justification for analyzing qualitative and quantitative data within the same study. First, we present several recent typologies of analyses in social science research that incorporate both monomethod (i.e. purely quantitative research or purely qualitative research) and mixed research studies. Second, we discuss what has been referred to as the fundamental principle of empirical data analysis, wherein both qualitative and quantitative data analysis techniques are shaped by an attempt to analyze data in a way that yields at least one of five types of generalizations. Third, building on the frameworks of Denzin and Lincoln (2005), Heron and Reason (1997) and Johnson and Onwuegbuzie (2004), we compare and contrast three qualitative-based paradigms (i.e. constructivism, critical theory, participatory), one quantitative-based paradigm (i.e. postpositivism) and one mixed research-based paradigm (i.e. pragmatism) with respect to three axiomatic components (i.e. ontological, epistemological and methodological foundations) and seven issues (i.e. nature of knowledge, knowledge accumulation, goodness or quality criteria, values, ethics, inquirer posture and training). Also, we link each paradigm to data analysis strategies. Fourth, we illustrate similarities in goals between some qualitative and quantitative analyses; in so doing, we deconstruct the strong claim that analysis must be either qualitative or quantitative and illustrate that regardless of perspective (e.g. postpositivist or constructivist), both qualitative and quantitative data can be jointly analyzed. Finally, we compare and contrast 11 mixed research paradigms/worldviews, linking them to mixed analysis strategies, thereby situating mixed analyses in the philosophy of social science and promoting mixed research as a distinctive methodology. Keywords: multiple operationalism, quasi statistics, inferential statistics, descriptive statistics, empirical data analysis, mixed analysis strategies, multiple research approaches, mixed methods, qualitative and quantitative, research paradigm, methodology, epistemology, ontology, post-positivism, critical theory, participatory, constructivism, pragmatism, social science research typology 114 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches T he process of mixed research – involving ‘mix[ing] or combin[ing] quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study’ (Johnson & Onwuegbuzie 2004: 17) – has developed a long way since Campbell and Fiske (1959) coined the term multiple operationalism, wherein more than one method is used as part of a validation process to help ensure that the variance explained culminates from the underlying phenomenon or trait and is not a function of the method. However, most of the biggest formalized developments in mixed research have occurred within the last 25 years, being given impetus in the 1980s by several prominent researchers (e.g. Brewer & Hunter 1989; Bryman 1988; Greene, Caracelli, & Graham 1989; Jick 1983; Kidder & Fine 1987; Louis 1982; Madey 1982; Mark & Shotland 1987; Maxwell, Bashook, & Sandlow 1986; Phelan 1987; Rossman & Wilson 1985), who called for the integration of quantitative and qualitative approaches. However, the last five years have witnessed a significant increase in the number of mixed research studies, marked by a handbook on mixed methods (Tashakkori & Teddlie 2003); several mixed research textbooks (e.g. Bergman 2008; Creswell & Plano Clark 2007; Greene 2007; Johnson & Christensen 2008; Plano Clark & Creswell 2007; Ridenour & Newman 2008; Teddlie & Tashakkori 2009); several mixed research articles contained in methodological handbooks (e.g. Teddlie, Tashakkori, & Johnson 2008); mixed research articles contained in encyclopedias (e.g. Onwuegbuzie 2007); two journals devoted to mixed research (i.e. Journal of Mixed Methods Research, International Journal of Multiple Research Approaches); several journals now routinely publishing mixed research (e.g. Field Methods, Educational Evaluation & Policy Analysis, Quality & Quantity, Evaluation, Evaluation Practice, Research in Nursing & Health, Research in the Schools, The Qualitative Report); and websites (e.g. http://www.fiu.edu/~bridges/), conferences (e.g. http://www.mixedmethods.leeds.ac.uk/) and workshops (Creswell & Plano Clark 2008; Mertens 2008; O’Cathain 2008; Onwuegbuzie & Collins 2008; Onwuegbuzie, Slate, Leech, & Collins 2008) devoted to mixed research, and by special issues (Gorard & Smith 2006; Johnson 2006; O’Cathain & Collins 2009) with another underway (Leech & Onwuegbuzie, 2010). These and other sources have helped to increase the visibility of mixed research. Greene (2008: 8) recently asked the following questions: • ‘Is mixed methods social inquiry a distinctive methodology? • Is the field moving in that direction? • What is needed for mixed methods to become a distinctive methodology? According to Greene (2006, 2008), the development of a methodological or research paradigm (i.e. qualitative, quantitative and mixed research) in the social and behavioral sciences requires a thorough critique of four interrelated but conceptually distinct domains: (i) philosophical assumptions and stances (i.e. what are the core philosophical or epistemological assumptions of the methodology?); (ii) inquiry logics (i.e. what traditionally is called methodology and refers to broad inquiry purposes and questions, logic, procedures and designs, quality standards and writing and reporting forms that guide the researcher’s gaze); (iii) guidelines for research practice (i.e. specific strategies and tools that are used to conduct research; the how to component of research methodology); (iv) sociopolitical commitments (i.e. interests, commitments and power relations surrounding the location in society in which an inquiry is situated; proclamation of values-based rationales and meanings for the practice of social and behavioral science research in society). Together, Greene’s four domains provide a cohesive and interactive framework and an array Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 115 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins of practical guidelines for a methodological or research paradigm. Although these domains have been more fully developed with respect to both the quantitative and qualitative research paradigms, this is not the case for the field of mixed research. Indeed, not all of Greene’s (2006) domains have been fully articulated and developed in mixed research (Greene 2006, 2008). Consequently, more theoretical, conceptual and practical work is needed in the area of mixed research. In recent years, much has been written about most of the 13 distinct, interactive, iterative steps of the mixed research process identified by Collins, Onwuegbuzie and Sutton (2006), namely: (a) determining the mixed goal of the study (b) formulating the mixed research objective(s) (c) determining the rationale(s) for mixing quantitative and qualitative approaches (d) determining the purpose(s) for mixing quantitative and qualitative approaches (e) determining the mixed research question(s) (f ) selecting the mixed sampling design (g) selecting the mixed research design (h) collecting quantitative and qualitative data (i) transforming and analyzing the quantitative and qualitative data (j) legitimating the data sets and mixed research findings (k) interpreting the mixed research findings (l) writing the mixed research report (m) reformulating the mixed research question(s). However, despite the fact that the mixed analysis step is considered by beginning researchers to be the most difficult step of the mixed research process, because typically this step necessitates competence in conducting both quantitative and qualitative data analyses, it is one of the least developed areas in the mixed research literature, with relatively few published articles on mixed analysis (e.g. Bazeley 2003 2006; Caracelli & Greene 1993; Chi 1997; Greene et al. 1989; Lee & Greene 2007; Li, Marquart, & Zercher 2000; Onwuegbuzie 2003; 116 Onwuegbuzie & Dickinson 2008; Onwuegbuzie & Leech 2004, 2006; Onwuegbuzie, Slate, Leech & Collins 2007; Onwuegbuzie, Slate, Leech, & Collins 2009; Onwuegbuzie & Teddlie 2003; Sandelowski 2000, 2001; Teddlie et al. 2008). Currently, published mixed research textbooks (e.g. Bergman 2008; Creswell & Plano Clark 2007; Greene 2007; Ridenour & Newman 2008) – although groundbreaking – contain at most one chapter on mixed analysis. Thus, as noted by Greene (2008: 14): ‘There has also been some work in the area of integrated mixed methods data analysis, although this work has not yet cohered into a widely accepted framework or set of ideas’. Moreover, when discussing mixed analysis strategies, none of these authors discuss the philosophical underpinnings. Yet, by linking mixed analysis techniques to philosophical assumptions and stances, an iterative, interactive and dynamic linkage is provided among Greene’s (2006, 2008) four domains. With this in mind, the present article is a first attempt explicitly to provide a philosophical justification for conducting mixed analyses. First, we present several recent typologies of analyses in social science research that incorporate both monomethod (i.e. purely quantitative research or purely qualitative research) and mixed research studies. Second, we discuss what has been referred to as the fundamental principle of data analysis, wherein both qualitative and quantitative data analysis techniques are shaped by an attempt to analyze data in a way that yields at least one of five types of generalizations. Third, building on the frameworks of Denzin and Lincoln (2005), Heron and Reason (1997) and Johnson and Onwuegbuzie (2004), we compare and contrast three qualitative-based paradigms (i.e. constructivism, critical theory, participatory), one quantitative-based paradigm (i.e. postpositivism) and one mixed research-based paradigm (i.e. pragmatism) with respect to three axiomatic components (i.e. ontological, epistemological and methodological foundations) and seven issues INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches (i.e. nature of knowledge, knowledge accumulation, goodness or quality criteria, values, ethics, inquirer posture and training). Also, we link each paradigm to data analysis strategies. Fourth, we illustrate similarities in goals between some qualitative and quantitative analyses; in so doing, we deconstruct the strong claim that analysis must be either qualitative or quantitative and illustrate that regardless of perspective (e.g. postpositivist or constructivist), both qualitative and quantitative data can be jointly analyzed. Finally, we compare and contrast 11 mixed research paradigms/worldviews, linking them to mixed analysis strategies. TYPOLOGY OF ANALYSES IN SOCIAL SCIENCE RESEARCH Onwuegbuzie et al. (2007) outlined the concept of monotype data, which represents the use of one data type (e.g. qualitative data) that is available for analysis – in contrast to multitype data wherein both types of data (i.e. qualitative and quantitative data) are collected and thus are available for analysis. Onwuegbuzie et al. (2007) coined the phrase monoanalysis to denote when one class of analysis (e.g. qualitative analysis) is used to analyze one data analysis type (e.g. qualitative data) – as opposed to multianalysis, wherein both classes of analyses (i.e. qualitative analysis and quantitative analysis) are used to analyze one or more data analysis types. For example, a quantitative researcher might use multiple regression (Fox 1997) to examine which variables predict some quantitative outcome of interest. Alternatively, a qualitative researcher might use the method of constant comparison (Glaser & Strauss 1967) to analyze responses to open-ended interview questions. Three dimensional framework for qualitative and quantitative analyses Onwuegbuzie et al. (2009) also conceptualized a typology for classifying qualitative and quantitative analysis techniques. Specifically, these authors presented a three-dimensional representation for classifying and organizing both qualitative and quantitative analyses, which involves reframing qualitative and quantitative analyses as a case-oriented, variable-oriented, or process/experienceoriented analyses. 1. Case-oriented analyses are analyses that focus primarily or exclusively on the selected case(s), wherein the goal is to analyze and interpret the meanings, experiences, attitudes, opinions, or the like of one or more persons – with a tendency toward particularizing and analytical generalizations. Although case-oriented analyses are best suited for identifying patterns common to one or a relatively small number of cases – and thus lend themselves better to qualitative research in general and qualitative analyses in particular – this class of analyses can be used for any number of cases and, as such, also is pertinent for quantitative research, leading to the use of quantitative analysis techniques such as single-subject analyses, descriptive analyses and profile analyses (Onwuegbuzie et al. 2009). 2. In contrast, variable-oriented analyses involve identifying relationships – often probabilistic in nature – among entities that are treated as variables such that this class of analysis tends to be conceptual and theory-centered from the onset and has a proclivity toward external generalizations. Therefore, variable-oriented analyses, whose ‘“building blocks” are variables and their intercorrelations, rather than cases’ (Miles & Huberman 1994: 174), are more apt for quantitative research in general and quantitative analyses in particular. However, although the use of large and representative samples often facilitates identification of relationships among variables, small samples also can serve this purpose, making variable-oriented analyses also relevant for qualitative data along with the use of qualitative analysis techniques (e.g. examining themes that cut across cases). 3. Finally, process/experience-oriented analyses involve evaluating processes or experiences Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 117 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins pertaining to one or more cases within a specific context over time, with processes tending to be associated with variables and experiences tending to be associated with people (i.e. cases). Because each of these three analysis orientations represent a continuum rather than a dichotomy (e.g. variable-oriented analyses are conceptualized by Onwuegbuzie et al. (2009) as falling on a particularistic–universalistic continuum, classifying the extent to which the metainferences stemming from the variable-oriented analysis can be generalized), this three-dimensional framework supports Johnson and Onwuegbuzie’s (2004: 20) assertion that ‘the possible number of ways that studies can involve mixing is very large because of the many potential classification dimensions’. Cross-over mixed analysis Onwuegbuzie and Combs (2009), building on the works of Greene (2008) and Onwuegbuzie and Teddlie (2003), outlined the concept of cross-over mixed analyses, which involves using one or more analysis types associated with one tradition (e.g. quantitative analysis) to analyze data associated with a different tradition (e.g. qualitative data). Such an analysis can be used to reduce, display, transform, correlate, consolidate, compare, integrate, assert, or import data. Table 1 presents the cross-over mixed analysis types and strategies that Onwuegbuzie and Combs (2009) identified. Cross-over mixed analyses are distinct from other types of mixed analyses (i.e. non-cross-over mixed analyses) such as parallel mixed analysis, wherein both quantitative and qualitative data analyses are conducted separately, neither type of analysis builds on or interacts with the other during the data analysis stage and the findings from each type of analysis are neither compared nor consolidated until both sets of data analyses have been completed. Whereas non-cross-over mixed analyses involve collection of both types of data and the analysis conducted per data set represents the same paradigmatic tradition (i.e. either quantita118 tive or qualitative) – ‘within-paradigm analysis’ (Onwuegbuzie et al. 2007: 12) – cross-over mixed analyses involve a between-paradigm analysis, which involves ‘an analysis technique more associated with one traditional paradigm (e.g. quantitative) to analyze data that originally represented the type of data collected associated with the other traditional paradigm (e.g. qualitative)’ (Onwuegbuzie et al. 2007: 12). Thus, cross-over mixed analyses involve more integration of qualitative and quantitative analyses than do other types of mixed analyses because they involve the mixing or combining of qualitative- and quantitative-based paradigmatic assumptions and stances (e.g. using exploratory factor analysis to examine the structure of themes that emerged from a qualitative analysis; cf. Onwuegbuzie 2003), which involves either maintaining an analytical-philosophical stance that the human mind/perception and mathematical algorithms can be used sequentially to examine patterns in qualitative data or adopting an analytical-philosophical stance that transcends the stances underlying both paradigms (e.g. assuming that data saturation [i.e. qualitative information repeats itself such that no new or relevant information seem to emerge pertaining to a category and the category development is well established and validated; Morse 1995] and reliability [i.e. consistency or repeatability of participants’ quantitative responses] represent parallel constructs). This distinction between cross-over mixed analyses and non-cross-over mixed analyses is important because certain epistemological, ontological, axiological and methodological stances might lend themselves more to conducting cross-over mixed analyses than do other stances. Building on the works of Onwuegbuzie et al. (2007), Onwuegbuzie et al. (2009) and Onwuegbuzie and Combs (2009), before conducting an analysis, a researcher explicitly or implicitly makes the following six decisions: (a) the number of data types that will be analyzed – yielding either monotype data (i.e. use of one data type, namely: qualitative data or quantitative data) or multitype data INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches TABLE 1: C ROSS -O VER (M IXED ) A NALYSES S TRATEGIES Analysis Step Cross-Case Analysis Strategy Integrated Data Reduction Reducing the dimensionality of qualitative data/findings using quantitative analysis (e.g. exploratory factor analysis of qualitative data) and/or quantitative data/findings (e.g. thematic analysis of quantitative data) (Onwuegbuzie 2003; Onwuegbuzie and Teddlie 2003) Integrated Data Display Visually presenting both qualitative and quantitative results within the same display (Lee and Greene 2007; Onwuegbuzie and Dickinson 2008) Data Transformation Converting quantitative data into data that can be analyzed qualitatively (i.e. qualitizing data; Tashakkori and Teddlie 1998) and/or qualitative data into numerical codes that can be analyzed statistically (i.e. quantitizing data; Tashakkori and Teddlie 1998) Data Correlation Correlating qualitative data with quantitized data and/or quantitative data with qualitized data (Onwuegbuzie and Teddlie 2003) Data Consolidation Combining or merging multiple data sets to create new or consolidated codes, variables, or data sets (Louis 1982; Onwuegbuzie and Teddlie 2003) Data Comparison Comparing qualitative and quantitative data/findings (Onwuegbuzie and Teddlie 2003) Data Integration Integrating qualitative and quantitative data/findings either into a coherent whole or two separate sets (i.e. qualitative and quantitative) of coherent wholes (McConney, Rudd and Ayres 2002; Onwuegbuzie and Teddlie 2003) Warranted Assertion Analysis Data Importation Reviewing all qualitative and quantitative data to yield meta-inferences (Smith 1997) Utilizing follow-up findings from qualitative analysis to inform the quantitative analysis (e.g. qualitative contrasting case analysis, qualitative residual analysis, qualitative followup interaction analysis and qualitative internal replication analysis; Li et al. 2000; Onwuegbuzie and Teddlie 2003) or follow-up findings from quantitative analysis to inform the qualitative analysis (e.g. quantitative extreme case analysis, quantitative negative case analysis; Onwuegbuzie and Teddlie 2003) (i.e. use of both data types, namely: qualitative data and quantitative data); (b) the number of data analysis types that will be used – yielding monoanalysis (i.e. use of one data analysis type, namely: qualitative data analysis or quantitative data analysis) or multianalysis (i.e. use of both data analysis types, namely: qualitative data analysis and quantitative data analysis); (c) the analysis emphasis of interest – comprising case-oriented analyses (i.e. analyses that focus primarily or exclusively on the selected case(s)), variable-oriented analyses (i.e. identifying relationships among entities that are conceived as variables) and/or process/experience-oriented analyses (i.e. evaluating processes or experiences pertaining to one or more cases within a specific context over time); (d) whether or not analysis types associated with one tradition will be used to analyze data associated with a different tradition (i.e. cross-over mixed analysis vs. non-crossover mixed analysis); (e) whether the qualitative and quantitative analyses will be carried out concurrently (i.e. results stemming from one data analysis phase do not inform the results stemming from the other phase) or sequentially (i.e. the qualitative analyses are conducted first, which then inform the subsequent quantitative analyses, or vice versa); (f ) whether the qualitative or quantitative analyses will be given priority (i.e. quantitative analyses carry the most weight or the qualitative phase carry the most weight), or whether they will be assigned equal status. Monomethod studies involve the use of monotype data, monoanalysis and non-cross-over Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 119 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins mixed analysis, thereby making the fifth (i.e. time orientation of analyses) and sixth (i.e. priority of analyses) decisions irrelevant. Monomethod studies also include a decision as to the analysis emphasis. In contrast, in mixed research, consideration of these six decisions yields a large variety of possible combinations (i.e. except the aforementioned combination associated with monomethod research) that is impossible to capture completely in any single typology. FUNDAMENTAL PRINCIPLE OF DATA ANALYSIS Onwuegbuzie et al. (2009) outlined what they refer to as the fundamental principle of data analysis,1 wherein both qualitative and quantitative data analysis techniques are shaped by an attempt to analyze data in a way that yields at least one of the following five types of generalizations: external (statistical)2 generalizations, internal (statistical)3 generalizations, analytical generalizations, case-tocase transfer and/or naturalistic generalizations. External (statistical) generalization involves making generalizations, inferences, or predictions on data obtained from a representative statistical (i.e. optimally random) sample to the population from which the sample was drawn (i.e. universalistic generalizability). Contrastingly, internal (statistical) generalization involves making generalizations, inferences, or predictions on data obtained from one or more representative or elite participants (e.g. key informants, politically important cases, sub-sample members) to the sample from which the participant(s) was selected (i.e. particularistic generalizability). It should be noted that internal (statistical) generalization can stem from 1 2 3 120 qualitative, quantitative, or mixed analyses. Conversely, with analytic generalizations ‘the investigator is striving to generalize a particular set of [case study] results to some broader theory’ (Yin 2009: 43). In other words, analytical generalizations are ‘applied to wider theory on the basis of how selected cases “fit” with general constructs’ (Curtis, Gesler, Smith, & Washburn 2000: 1002) (i.e. particularistic generalizability). Case-to-case transfer involves making generalizations or inferences from one case to another (similar) case (Firestone 1993; Kennedy 1979; Miles & Huberman 1994) (i.e. particularistic generalizability). Finally, with naturalistic generalization, the readers of the article (i.e. consumers of the findings) make generalizations entirely, or at least in part, from their personal or vicarious experiences (Stake 2005), such that meanings stem from personal experience and are adapted and reified by repeated encounter (Stake 1980; Stake & Trumbull 1982). Rather than the researcher making this sort of generalization, it is the reader who generalizes (often based on proximal similarity of the case data to the reader’s focal context of interest). LINKING PARADIGMS TO DATA ANALYSIS STRATEGIES Over the years, in an attempt to distinguish qualitative-based paradigms from quantitative-based paradigms and to demonstrate that qualitative research represents a distinct tradition, several eminent qualitative researchers have presented frameworks that contrast qualitative-based paradigms (e.g. constructivism, critical theory, participatory) and quantitative-based paradigms (i.e. [logical] positivism and postpositivism). Indu- Onwuegbuzie et al. (2009) used the word fundamental because this principle is pertinent (i.e. fundamental) to qualitative, quantitative and mixed analyses. Unlike Onwuegbuzie et al. (2009), we place the word ‘statistical’ in parentheses to denote the fact that external generalization can arise predominantly or exclusively from either quantitative or qualitative analyses. The term ‘statistical’ is used to denote the possible assumption that the sample was representative in some way (e.g. probabilistically, vicariously) of the larger group from which the sample was drawn. Unlike Onwuegbuzie et al. (2009), as is the case for external generalizations, we place the word ‘statistical’ in parentheses to denote the fact that internal generalization can arise predominantly or exclusively from either quantitative or qualitative analyses. Again, the term ‘statistical’ is used to denote the possible assumption that the sample was representative in some way (e.g. probabilistically, vicariously) of the larger group from which the sample was drawn. INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches bitably, the most popularized framework is that developed by Guba (1990) and extended, more recently, by Denzin and Lincoln (2005). Denzin and Lincoln outlined their views of the axiomatic nature of paradigms and the issues they believed were most fundamental to differentiating the paradigms by contrasting three qualitative-based paradigms (i.e. constructivism, critical theory, participatory) and two quantitative-based paradigms (i.e. [logical] positivism and postpositivism) with respect to three axiomatic components (i.e. ontological, epistemological and methodological foundations) and seven issues (i.e. nature of knowledge, knowledge accumulation, goodness or quality criteria, values, ethics, inquirer posture and training). In Table 2, we build on Denzin and Lincoln’s (2005) table of axioms and issues. Specifically, Table 2 contains Denzin and Lincoln’s (2005: 193-194) axioms of the paradigms of postpositivism and constructivism, as well as axioms of the participatory paradigms outlined by Heron and Reason (1997) (which was also included in Denzin and Lincoln’s  table). Table 2 also contains our proposed pragmatist paradigm, which incorporates some of the major concepts outlined in Johnson and Onwuegbuzie’s (2004) seminal article on mixed research. However, Table 2 does not include the axioms of the paradigm of positivism because this paradigm, which incorporates several movements (e.g. analytical philosophy, logical atomism, logical empiricism and semantics), was discredited shortly after World War II.4 As noted by Johnson and Onwuegbuzie (2004): Positivism is a poor choice for labeling quantitative researchers today because positivism has long been replaced by newer philosophies of science (Yu 2003). The term is more of a straw man (easily knocked down) for attack than standing for any actual practicing researchers. 4 A term that better represents today’s practicing quantitative researchers is postpositivism (Phillips & Burbules 2000: 24). In this table, we have included two additional issues to Denzin and Lincoln’s seven issues – namely, qualitative analysis and quantitative analysis – in an attempt to take the first step toward linking data analysis techniques to philosophical assumptions and stances. In Figure 1, we reproduce Onwuegbuzie et al.’s (2009) typology for classifying qualitative and quantitative analysis techniques. This figure presents an array of analytic techniques undertaken in quantitative and qualitative research as a function of the following three analysis emphases: case-oriented analyses, variable-oriented analyses and process/ experience-oriented analyses. Although this list is by no means exhaustive, it does capture most of the major qualitative and quantitative analytical techniques (Onwuegbuzie et al. 2009). Postpositivist paradigm Often under-emphasized in the literature, postpositivism is a rejection or modification of several core tenets of positivism. Although postpositivist researchers believe that there is an independent reality that can be studied, they assert that all observation is inherently theory-laden and fallible and that all theory can be modified. They also believe that, as a result of their cultural experiences and worldviews, people are always partially biased in their objective perceptions of reality. Given these inherent biases and perceptions and observations that are fallible, ensuing constructions are imperfect (Phillips & Burbules 2000). Thus, postpositivist researchers assert that we can only approximate the truth of reality but can never explain it perfectly or completely. Notwithstanding, postpositivist researchers believe that objectivity can be approximated by triangulating Disturbingly, many authors appear to be unaware of the six major tenets of positivism, namely: (a) an emphasis on verification, (b) pro-observation, (c) anti-cause, (d) downplaying explanation, (e) anti-theoretical entities and (f ) anti-metaphysics (Hacking 1983). Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 121 122 External replication and external statistical generalization Knowledge accumulation Historical revisionism; generalization by similarity; internal statistical generalization; analytical generalization; case-to-case transfer; naturalistic generalization Structural/historical insights Individual and collective reconstructions that may unite around consensus Nonfalsified hypotheses that are probably facts or laws Nature of knowledge Elaborate reconstructions; vicarious experience; internal statistical generalization; analytical generalization; case-to-case transfer; naturalistic generalization Critical discourse Detailed, rich and thick (empathic) description, written directly and somewhat informally Rhetorical neutrality, involving formal writing style using impersonal passive voice and technical terminology, in which establishing and describing social laws is the major focus; may include qualitative methods Rhetorical In communities of inquiry contained in communities of practice Entrenched epistemolog-ical emphasis on practical knowing and critical subjectivity Use of language based on shared experiential context Political participation in collaborative action research; emphasis on practical Dialogic/dialectical Hermeneutical/ dialectical; impossible to differentiate fully causes and effects; inductive reasoning; timeand context-free generalizations are neither desirable nor possible Time- and context-free generalizations are desirable and possible and real causes of social scientific outcomes can be determined reliably and validly via quantitative (and sometimes qualitative) methods Methodology Experiential, propositional and practical knowing; co-created findings Transactional/subjectivist; value-mediated findings Subjective knower and known are not separable; Transactional/ subjectivist; co-created findings/meaning Researchers should eliminate their biases, remain emotionally detached and uninvolved with the objects of study and test or empirically justify their stated hypotheses Epistemology Participatory b Subjective-objective reality co-created by mind and given world order Critical Theory a D ISTINGUISHING CHARACTERISTICS Virtual reality influenced by social, political, cultural, ethnic, racial, economic and gender values that evolve over time AND Multiple contradictory, but equally valid accounts of the same phenomenon representing multiple realities Constructivisma,c CONTEMPORARY R ESEARCH PARADIGMS Social science inquiry should be objective Postpositivism a,c OF Ontology Paradigmatic Element TABLE 2: U NDERLYING B ELIEF S YSTEMS Follows dynamic homeostatic process of belief, doubt, inquiry, modified belief, new doubt, new inquiry, in an infinite loop, where the person or researcher (and research community) constantly tries to improve upon past understandings in a way that fits and works in the world in which he or she operates; internal statistical generalization; analytical generalization; case-to-case transfer; naturalistic generalization Intersubjectivity, emic and etic viewpoints; respect for nomological and ideographic knowledge; Use of both impersonal passive voice and technical terminology, as well as rich and thick (empathic) description Thoughtful/ dialectical eclecticism and pluralism of methods and perspectives; determine what works and solves individual and social problems Knowledge is both constructed and based on the reality of the world we experience and live in; justification comes via warranted assertability Multiple realities (i.e. subjective, objective, intersubjective); rejects traditional dualisms (e.g. subjectivism vs. objectivism; facts vs. values); high regard for the reality and influence of the inner world of human experience in action; current truth, meaning and knowledge are tentative and changing Pragmatismc Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 All forms of qualitative analyses Descriptive statistics; most, if not all, forms of inferential statistics that lead to internal (statistical) generalizations and external (statistical) generalizations Resocialization; qualitative and quantitative; history; values of altruism, empowerment and liberation All forms of qualitative analyses Descriptive statistics; some inferential statistics that lead to internal (statistical) generalization but not to external (statistical) generalization Technical; quantitative and qualitative; substantive theories Some forms of qualitative analysis are possible, especially qualitative analyses that generate numbers as part of the findings such as word count and classical content analysis All forms of descriptive and inferential statistics, with an ultimate goal of making external (statistical) generalizations Training Qualitative analysis Quantitative analysis c b Extracted from Denzin and Lincoln (2005, pp. 195-196) Extracted from Heron and Reason (1997) Extracted from Johnson and Onwuegbuzie (2004, pp. 14, 18-20) ‘Transformative intellectual’ as advocate and activist Passionate participant as facilitator of multivoice reconstruction Objective scientist and informer of decision makers, policy makers and change agents Inquirer posture a Intrinsic; moral proclivity toward revelation Intrinsic; process proclivity toward revelation Extrinsic Ethics Resocialization; qualitative and quantitative; history; values of altruism, empowerment and liberation Research is value-bound; formative; seeks to reveal injustice Research is value-bound Research is value-free Critical Theory a All forms of qualitative analyses All forms of descriptive and inferential statistics Descriptive statistics; inferential statistics that lead to both internal (statistical) generalizations and external (statistical) generalizations Qualitative, quantitative, mixed research; substantive theories; values of altruism, empowerment and liberation Offers the pragmatic method for solving traditional philosophica dualisms as well as for making methodological choices Extrinsic and intrinsic; justification comes in the form of warranted assertability Takes an explicitly value-oriented approach to research that is derived from cultural values; specifically endorses shared values such as democracy, freedom, equality and progress. Reliability, internal validity, external validity, objectivity; Trustworthiness, dependability, confirmability, transferability; authenticity Pragmatismc All forms of qualitative analyses Researchers, who learn via active engagement in study, need emotional competence, democratic disposition and skills Primary voice manifest via aware self-reflective action; secondary voices in revealing theory, narrative, etc. Intrinsic; moral proclivity toward revelation Research is value-bound Congruence of experiential, presentational, propositional and practical knowing leads to action to transform the world Participatory b D ISTINGUISHING CHARACTERISTICS (Continued) Historical situatedness; reduction of ignorance and misperceptions; involve participants in knowledge construction and validation AND Values (i.e. Axiology) Trustworthiness, dependability, confirmability, transferability; authenticity Constructivisma,c CONTEMPORARY R ESEARCH PARADIGMS Reliability, internal validity, external validity, objectivity Postpositivism a,c OF Goodness or quality criteria Paradigmatic Element TABLE 2: U NDERLYING B ELIEF S YSTEMS Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 123 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins Phase Case-oriented Variable-oriented Process/Experience-oriented Quantitative Descriptive Analyses (e.g. measures of central tendency, variability, position) Cluster Analysis Q Methodology Time Series Analysis Profile Analysis Panel Data Analysis Single-Subject Analysis Classical Test Theory Item Response Theory Multidimensional Scaling Hazard Proportional Hazards Model Descriptive Analyses Correlation Analysis Independent t-tests Dependent t-tests Analysis of Variance Analysis of Covariance Multiple Analysis of Variance Multiple Analysis of Covariance Multiple Regression (Multivariate) Logistic Regression Descriptive/Predictive Discriminant Analysis Log-Linear Analysis Canonical Correlation Analysis Path Analysis Structural Equation Modeling Hierarchical Linear Modeling Correspondence Analysis Multidimensional Scaling Exploratory/Confirmatory Factor Analysis Time Series Analysis Classical Test Theory Item Response Theory Descriptive Analyses (e.g. measures of central tendency, variability, position) Dependent t-tests Time Series Analysis Profile Analysis Panel Data Analysis Single-Subject Analysis Classical Test Theory Item Response Theory Repeated Measures Analysis of Variance Repeated Measures Analysis of Covariance Survival Analysis Path Analysis Structural Equation Modeling Hierarchical Linear Modeling Qualitative Word Count Keywords-in-Context Classical Content Analysis Secondary Data Analysis Taxonomic Analysis Componential Analysis Text Mining Qualitative Comparative Analysis Semantic Network Analysis Cognitive Map Analysis Causal Network Analysis Conceptually Ordered Matrix Analysis Case-ordered Matrix/Network Analysis Time-ordered Matrix/Network Analysis Variable-by-Variable Matrix Analysis Predictor-Outcome Matrix Analysis Explanatory Effect Matrix Analysis Method of Constant Comparison Word Count Keywords-in-Context Classical Content Analysis Domain Analysis Taxonomic Analysis Componential Analysis Conversation Analysis Discourse Analysis Secondary Data Analysis Text Mining Narrative Analysis Manifest Content Analysis Latent Content Analysis Qualitative Comparative Analysis Semantic Network Analysis Cognitive Map Analysis Causal Network Analysis Time-ordered Matrix/Network Analysis Method of Constant Comparison Word Count Keywords-in-Context Classical Content Analysis Domain Analysis Taxonomic Analysis Componential Analysis Conversation Analysis Discourse Analysis Secondary Data Analysis Membership Categorization Analysis Narrative Analysis Semiotics Manifest Content Analysis Latent Content Analysis Text Mining Qualitative Comparative Analysis Micro-interlocutor Analysis Partially Ordered Matrix Analysis Time-ordered Matrix/Network Analysis Note: All quantitative analyses above include non-parametric counterparts. F IGURE 1: T HREE -DIMENSIONAL MATRIX INDICATING ANALYTICAL TECHNIQUES AS A FUNCTION OF APPROACH (i.e. quantitative vs. qualitative) and analysis emphasis (i.e. case-oriented vs. variable-oriented vs. process/experience-oriented). Reproduced with permission: International Journal of Multiple Research Approaches (2009) 3(1): 27 [Figure 4]. across these multiple fallible perspectives (i.e. triangulation of method, data and theory). Karl Popper (1994) identified three worlds that postpositivist researchers can address: (a) an objective 124 physical external world (World 1); (b) an interpretative, subjective inner world (World 2); and (c) the theory world where humans mentally and physically represent the first two worlds (World INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches 3). Popper’s views of knowledge and reality are just one example of complexity and nuance commonly expressed by quantitative researchers that is not represented in simple statements found in paradigm comparison tables popular in the research literature. Belief in the fallibility of observations renders statistics in general, and inferential statistics in particular, as suited to postpositivist research due to its emphasis on assigning probabilities (e.g. pvalues, levels of confidence or error) to observed findings. Thus, as well as utilizing descriptive statistics, postpositivist researchers use the whole array of inferential statistics for making external (statistical) generalizations (cf. Figure 1). Contrary to popular depictions of quantitative research as being deductive (and qualitative as inductive), inferential statistical generalizations are inductive. Postpositivist researchers also utilize some qualitative analysis techniques, especially those that yield frequency data such as Word Count and Classical Content Analysis (cf. Leech & Onwuegbuzie 2007, 2008). Postpositivist researchers also employ qualitative data analysis techniques that help them develop quantitative instruments. Constructivist paradigm As can be seen in Table 2, constructivist researchers (e.g. radical constructivists, cognitive constructivists, cultural constructivists, social constructivists/constructionists, communal constructivists, critical constructivists, genetic epistemology) often claim to believe that multiple, contradictory, but equally valid accounts of the same phenomenon (i.e. multiple realities) can exist. Thus, it is somewhat contradictory that those who ascribe to strong relativism or strong constructivism do not view the use of quantitative methods in general and quantitative analysis in particular as representing one such valid account of a phenomenon – albeit not their preferred account. However, the fact that constructivists tend to believe that time- and context-free generalizations are neither desirable nor possible, likely renders inappropriate the use of inferential statis- tics such as the sample mean for the purpose of making external (statistical) generalizations across populations (i.e. claiming that a statistical parameter applies broadly to nearly everyone in a population). Generalizing to a population (e.g. claiming that the sample statistic such as the sample mean is a reasonable representation of the corresponding population parameter) should be less controversial. This distinction of generalizing across versus generalizing to a population (Cook & Campbell 1979) seldom is addressed in the paradigm debates. Parameter estimation would probably be of more interest to a social constructivist than a radical or cognitive constructivist. Descriptive statistics has long been an important part of ethnography in anthropology as ethnographers include quantitative descriptors to complement narrative description. Descriptive statistics, more generally, can be used to enhance the qualitative researcher’s quest for detailed description. Sechrest and Sidani note (1995: 79) ‘qualitative researchers regularly use terms such as ‘many,’ ‘most,’ ‘frequently,’ ‘several,’ ‘never,’ and so on. These terms are fundamentally quantitative.’ Qualitative researchers also can obtain more meaning by obtaining counts of words in addition to their narrative descriptions (Sandelowski 2001). For example, in examining the lived experience in the classroom of Johnny, a child with attention deficit hyperactivity disorder, rather than telling readers that Johnny left his seat on many occasions during a class, it would be informative for the qualitative researcher to note that Johnny left his seat six times during the course of 30 minutes. This way, the readers can (if contextual detail is provided) decide whether six incidences of out-ofseat behavior are significant/meaningful and also will be in a better position to decide whether to make naturalistic generalizations. Consistent with our recommendation for qualitative researchers to use descriptive statistics more frequently where applicable, more than one half a century ago, Barton and Lazarsfeld (1955) advocated the use of what they coined quasi-statistics in qualitative research. According to these Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 125 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins authors, quasi-statistics pertain to the use of descriptive statistics that can be extracted from qualitative data. Interestingly, the prominent symbolic interactionist Howard Becker (1970: 81-82) contended ‘one of the greatest faults in most observational case studies has been their failure to make explicit the quasi-statistical basis of their conclusions.’. Further, Joseph Maxwell (1996: 95) noted that: Quasi-statistics not only allow you to test and support claims that are inherently quantitative, but also enable you to assess the amount of evidence in your data that bears on a particular conclusion or threat, such as how many discrepant instances exist and from how many different sources they were obtained. [emphasis in original] Interestingly, Becker, Geer, Hughes and Strauss (1961/1977) provided more than 50 tables and graphs in their qualitative works. These tables and graphs complemented the narrative descriptions of their qualitative data. Internal (statistical) generalizations also are possible in constructivist research. In particular, it is not unusual for constructivist researchers to utilize key informants to provide them with an insider’s understanding and to provide information that the researcher is unable to experience. This often includes both qualitative and quantitative information. For example, the informant might state what many members believe or describe as characteristics of many (or only a few) people in the group. These numbers would contribute to internal generalizations. In particular, inferential statistics can be used to make internal (statistical) generalizations. Educational ethnographers Margaret LeCompte and Jude Preissle (1993) discuss enumeration in ethnographic data analysis in some depth in their qualitative research textbook. The fact that many computer-assisted qualitative data analysis software (CAQDAS) programs allow data to be imported to statistical software programs (e.g. Excel, SIMSTAT) supports our 126 assertion that descriptive and inferential statistical analyses are an option for constructivist researchers should they deem it appropriate (e.g. based on their philosophical stance and research question(s)) to make internal (statistical) generalizations. These statistical analysis tools also can be used to bolster analytical generalizations. One of the most popular methods of qualitative research, grounded theory, is premised on the development of theory for the purposes of generalization. Descriptive and inferential statistics can be used to facilitate rich and detailed description, and to assess and enhance trustworthiness, dependability, confirmability, transferability and authenticity. Consistent with this assertion, Denzin and Lincoln (2005) state that the training of constructivists includes both qualitative and quantitative techniques (cf. Table 2). Whatever findings emerge from descriptive and/or inferential statistical analyses utilized, it should be noted that for constructivists, these findings represent just one of the multiple valid accounts of the phenomenon. Critical theory paradigm Critical theory researchers seek to understand the relationship between societal structures (e.g. economic, political) and ideological patterns of thought that impede a person or group from identifying, confronting and addressing unjust social systems. Simply put, critical theory researchers primarily are interested in social change as it emerges in relation to social struggle. Moreover, critical theory researchers operate under the assumption that the knowledge gleaned from their research represents an initial step toward addressing social injustices and promoting social change. As such, critical theory research represents a form of transformative research by providing emancipatory knowledge that identifies the contradictions that are masked or distorted by our everyday thoughts and perceptions (Lather 1986). Whereas many other types of qualitative researchers (e.g. phenomenological researchers, constructivist researchers) are interested in the meanings that people attach to INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches their own actions, critical theory researchers aim to place such actions in a broader context that is framed by social, economic, political and ideological forces that have previously not been identified or acknowledged. That is, critical theory researchers are more concerned with the forces that limit actions rather than the actions themselves. Thus, critical theory researchers tend to embrace a more etic (i.e. outsider’s) stance than an emic (i.e. insider’s) stance. Although it might appear that critical theory research is similar to postpositivist researchers – where critical theory researchers seek cause-andeffect relationships wherein the actions of individuals or groups are influenced directly by social, economic, political and/or ideological variables – many critical theory researchers would consider this strategy as being inappropriately reductionistic. Rather, many postmodern critical theory researchers believe that it is not possible to predict reliably how these variables determine actions. Further, instead of focusing on foundational criteria to justify its research findings, critical theory researchers contend that, by attending to the role of power in social systems, their analyses occur at the meta-theoretical level. Drawing upon research from other paradigms, critical theory researchers identify power structures and their processes that are typically ignored in both postpositivist and constructivist research. Like constructivist researchers, critical theory researchers believe that meaning and language are socially constructed and are neither time- nor context-free, although critical theory researchers assert that social injustice has real (i.e. objective, material) consequences. Also, like constructivist researchers, critical theory researchers primarily assess their findings with respect to the community of researchers to which they belong (i.e. authoritative consensus). Critical theory researchers contend that the subjective/objective dualism camouflages the ways in which both standpoints are constrained by power dynamics and social forces. However, critical theory researchers endorse subjectivism inasmuch as the subjective knower and known cannot be separated and that knowledge is subjective (culturally and historically embedded) and constructed on the basis of power issues (Lather 2006); yet, they reject the stance that all analyses are relative, instead, believing that rational analysis is essential to social justice. Thus, many critical theorists endorse critical realism (Morrow & Brown 1994). Critical theory researchers believe that constructivists place too much weight on people’s perceptions and too little emphasis on the more complex social forces that shape and constrain experiences, events and actions. As such, critical theory researchers deem constructivism to be just as much at risk as is postpositivism of uncritically bolstering the status quo of social injustice. According to Habermas, a leading contemporary proponent of critical theory, critical theory involves the following three types of knowledge: (a) the empirical-analytic sciences (i.e. including both the natural sciences and social sciences, which seek to manipulate knowledge for the purposes of prediction and control over nature and social structure); (b) historical-hermeneutic sciences (i.e. including the cultural and human sciences, which seek to understand communication among and within social groups); and (c) critical theory (i.e. which seeks freedom from oppression and social justice) (Blaikie 1993). Thus, Habermas advocates the use of both the method of the empirical-analytic sciences, historical-hermeneutic sciences and critical theory. However, critical theory researchers believe that critical theory utilizes but transcends both the empirical-analytic and historical-hermeneutic sciences inasmuch as it provides knowledge that is subjected to rigorous processes of free and rationale discourse, which prevents it from being socially distorted (Blaikie 1993). This suggests that the ontological, epistemological and axiological stances of critical theorist researchers do not prevent them from using, as appropriate, all forms of qualitative analysis techniques and quantitative analysis techniques that include both descriptive and inferential statistics (cf. Figure 1). Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 127 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins As noted by Morrow and Brown (1994: 200), ‘critical theory has no basis for a priori rejection of any particular methods or techniques as such’. However, whatever analysis is used, critical theory researchers make no claims that their analyses are objective in the manner claimed by some postpositivists. Participatory paradigm. As noted by Heron and Reason (1997), proponents of the participatory paradigm subscribe to an experiential subjective–objective reality that stems from a co-creative interaction between the given cosmos and the way the mind connects with it. Thus, what can be known about the given cosmos is that it represents a subjectively articulated world, whose objectivity reflects how it is intersubjectively formed by the knower. A participative worldview acknowledges the following four interdependent ways of knowing: experiential (i.e. direct encounter), presentational (i.e. emerges from and grounded on experiential knowing), propositional (i.e. conceptually knowing that something or other is the case) and practical (i.e. knowing how to do something) (Heron & Reason 1997). Of these, propositional knowing is most compatible with statistical analyses. In particular, propositional knowing represents knowledge that comes to the fore by describing some person, group, entity, location, situation, process, or the like. Simply put, it represents knowledge of facts. This way of knowing is articulated via declarations and theories that stem from mastery of concepts that language provides. This suggests that propositional knowledge can be expressed not only via qualitative analysis techniques but also via an array of quantitative techniques that include both descriptive and inferential statistics. For example, inferential statistics could be used to attach levels of certainty (i.e. probability) to knowing that something is the case. Interestingly, as noted by Heron and Reason (1997), intervention research represents a form of participative inquiry that includes statistical analyses (cf. Fryer & Feather 1994). 128 When studying a group or community, participatory researchers might use inferential techniques to test theory that represents propositional knowledge. Confirmation of a theory then would lead the participatory researcher to make external (statistical) generalizations from the participants to the group or community from which the participants were selected. Alternatively, when studying key informants or sub-sample members, a participatory researcher might use inferential statistics to make internal (statistical) generalizations to assess propositional knowledge. Alternatively still, when studying one or a few people, then only descriptive statistics might be appropriate. The quest for practical knowing also invites statistical analyses. In particular, practical knowing assumes a conceptual understanding of standards and principles of practice, presentational stylishness and experiential grounding in the situation within which the action takes place. In this respect, participatory research coincides with pragmatist research, which endorses a strong and practical empiricism, in which both qualitative and qualitative data are collected and analyzed to generate and to test theories against observations of the natural world. As such, participatory research does not appear to invalidate the use of descriptive or inferential statistical analyses and thus all analyses in Figure 1 are available to participatory researchers. Pragmatist paradigm The pragmatist paradigm offers an epistemological justification (i.e. via pragmatic epistemic principles and standards) and logic (i.e. combining approaches that help researchers optimally frame, examine and provide tentative answers to one’s research question[s]) for mixing approaches and methods (Johnson, Onwuegbuzie, & Turner 2007; Onwuegbuzie & Leech 2005). Further, a pragmatist would reject the incompatibility thesis that qualitative and quantitative research are fully incompatible and cannot, in any useful way, be used in combination in social or behavioral research. A serious problem with the incompati- INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches bility thesis is that it is an a priori argument; it seems to be based on rationalistic, foundational, deductive logic (which oftentimes is said to have been rejected in qualitative research). The incompatibility thesis does not appear to be based on observation of social and behavioral research (i.e. through an examination of how researchers actually conduct their research). In pragmatist research, research paradigms can remain separate, but they also can be mixed or combined into another research paradigm (i.e. mixed research). Johnson et al. (2007: 129) asserts that pragmatist researchers may ascribe to the view of mixed methods research is a research paradigm that: (a) partners with the philosophy of pragmatism in one of its forms (left, right, middle); (b) follows the logic of mixed methods research (including the logic of the fundamental principle and any other useful logics imported from qualitative or quantitative research that are helpful for producing defensible and usable research findings); (c) relies on qualitative and quantitative viewpoints, data collection, analysis and inference techniques combined according to the logic of mixed methods research to address one’s research question(s); and (d) is cognizant, appreciative and inclusive of local and broader sociopolitical realities, resources and needs. The mixed methods research paradigm offers an important approach for generating important research questions and providing warranted answers to those questions. This type of research should be used when the nexus of contingencies in a situation, in relation to one’s research question(s), suggests that mixed methods research is likely to provide superior research findings and outcomes. As noted in Table 2, pragmatist researchers can use the whole range of qualitative analyses and quantitative (i.e. descriptive and inferential analytical techniques) in an attempt to fulfill one or more of five mixed research purposes identified by Greene et al. (1989): tri-angulation (i.e. comparing findings from quantitative data with qualitative results in hopes of convergence); complementarity (i.e. seeking elaboration, illustration, enhancement and clarification of the results from one method with findings from the other method); development (i.e. using the results from one method to help inform the other method); initiation (i.e. discovering paradoxes and contradictions that culminate in a re-framing of the research question); and expansion (i.e. expanding the breadth and range of a study by using multiple methods for different study phases). Decisions made regarding these five mixed research purposes and the resulting mixed research designs help pragmatist researchers to determine which of three types of mixed analysis should be undertaken: a parallel mixed analysis (i.e. findings obtained from both analysis phases are interpreted separately), concurrent mixed analysis (i.e. results stemming from one data analysis phase do not inform the results stemming from the other phase), or sequential mixed analysis (i.e. the qualitative analysis phase is conducted first, which then informs the subsequent quantitative analysis phase, or vice versa). In particular, if the purpose of mixing is complementarity, then all three families of mixed analyses (i.e. parallel, concurrent and sequential) can be used. If triangulation or initiation represents the purpose, then both parallel and concurrent mixed analyses are viable. If development is the purpose, then concurrent and sequential mixed analyses are appropriate. Finally, if the purpose of mixing analyses is expansion then a sequential mixed analysis is pertinent (Onwuegbuzie et al. 2007). The purposes of mixed research and the resulting kinds of analyses also will help pragmatist researchers determine the number of data types that will be analyzed (i.e. monotype data vs. multitype data), the number of data analysis types that will be used (i.e. monoanalysis vs. multianalysis), the analysis emphasis of interest (i.e. case-oriented analyses, variable-oriented analyses Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 129 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins and/or process/experience-oriented analyses), whether or not analysis types associated with one tradition will be used to analyze data associated with a different tradition (i.e. cross-over mixed analysis vs. non-cross-over mixed analysis) and whether the qualitative or quantitative analyses will be given priority, or whether they will be assigned equal status. Pragmatist researchers also can use inferential statistics to make internal (statistical) generalizations. For instance, in the field of linguistics, political science and the like, it is not unusual for researchers analyzing qualitative data to conduct sequence analyses wherein hypotheses regarding patterns and trends in the qualitative data are tested (i.e. using p-values, confidence intervals and/or effect sizes). As an example, these researchers might test whether themes occurring among the participants statistically and/or practically significantly tend to emerge in a particular order. QDA Miner 3.0 is a computer-assisted qualitative data analysis software (CAQDAS) program that allows researchers to conduct qualitative analyses, such as thematic analysis and also to sequence analyses and other statistical and visualization tools such as clustering, multidimensional scaling, heatmaps and correspondence analysis. The fact that CAQDAS programs increasingly are allowing inferential statistical analyses to be conducted supports our assertion that inferential statistical analyses are an option that analysts of qualitative data have should they deem it appropriate to make statistical (i.e. internal and to a lesser degree external) generalizations. These inferential statistical analysis tools also can be used to bolster analytical generalizations. The descriptive and inferential statistics could be used not only to facilitate rich and detailed description but also could be used to assess and enhance trustworthiness, dependability, confirmability, transferability and authenticity. For pragmatist research, having a postpositivist orientation does not prevent a researcher from conducting qualitative analyses – especially analy130 ses such as word count and content analysis that involve, in part, the counting of words, codes, categories, or other aspects of the qualitative data. Such analyses can form part of quantitative-dominant mixed analyses (cf. Johnson et al. 2007), in which the analyst adopts a postpositivist stance, while, at the same time, believing that the inclusion of qualitative data and approaches are likely to enhance the findings. Similarly, having a constructivist orientation – or any other qualitativebased orientation – does not prevent a researcher from conducting quantitative analyses – especially descriptive statistical analyses that do not involve the analyst making inferences beyond the research participants at hand, which is typically not the goal of qualitative researchers. Such analyses can form part of qualitative-dominant mixed analyses (cf. Johnson et al. 2007), in which the researcher takes a constructivist-poststructuralist-critical stance with respect to the mixed analysis process, while, at the same time, deeming the addition of quantitative data and approaches as helpful in providing richer data and interpretations. Commonalities regarding data analysis strategies across paradigms From Table 2, the table of axioms and issues, it can be seen that the ontological, epistemological and methodological assumptions and stances representing all five paradigms allow the conduct of both quantitative and qualitative analyses – at least to a small degree – with postpositivist and constructivist paradigms having the least potential to use analytical techniques that belong to a different paradigm, the critical theory and the participatory paradigms having excellent potential to use analytical techniques that belong to a different paradigm and the pragmatist paradigm, almost by definition, having the greatest potential. That the ontological, epistemological and methodological assumptions and stances representing all five paradigms legitimate both quantitative and qualitative analyses to be undertaken is especially apparent when one INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches examines the similarities in goals between many quantitative and qualitative analyses rather than emphasizing the differences. 5 First and foremost, both quantitative and qualitative researchers analyze empirical observations (i.e. data coming from personal experience, observation, or experiment) to address research questions (Johnson & Onwuegbuzie 2004). Sechrest and Sidani (1995: 78) note that both sets of researchers ‘describe their data, construct explanatory arguments from their data and speculate about why the outcomes they observed happened as they did.’. At the level of data analysis, for example, numeric data are similar to textual/visual data inasmuch as they represent (descriptive) codes that characterize the person’s meanings, beliefs, attitudes and so on; thus, both data types might be available for collection and analysis regardless of the research paradigm involved, depending on the research question(s). Indeed, both numeric data and textual/visual data can be used to facilitate any of the following types of coding: inductive coding, deductive coding, abductive coding, interpretive coding, open coding, axial coding and selective coding (cf. Miles & Huberman 1994). As another example, thematic analysis (qualitative technique) and exploratory factor analysis (quantitative technique) have very similar goals, namely, to reduce the dimensionality of the raw data. This similarity allows mixed researchers to use analyses associated with one paradigm on data associated with another paradigm. For instance, it is not unusual for a mixed 5 6 researcher to factor analyze themes that emerged from textual data (cf. Onwuegbuzie 2003)6 or to undertake a profile analysis of a set of quantitative measures stemming from one or more cases (Tashakkori & Teddlie 1998). Statistical factor analysis is an exploratory data analysis technique that searches for sets of variables that are similar to one another but are different from the other sets of variables found in the data. The researcher must label the factors that emerge from the data; that is, the research must give a name to the factors and interpret what the named factors mean. The naming part of factor analysis is an example of the use of qualitative coding within a technique traditionally viewed as quantitative. Factor analysis, we argue, has both quantitative and qualitative elements and, therefore, is an inherently mixed analysis procedure. An even more compelling explanation for our assertion – that the ontological, epistemological and methodological assumptions and stances representing all five paradigms allow both quantitative and qualitative analyses to be undertaken – stems from the nature of qualitative and quantitative analyses themselves. Many of the core analytical techniques that are associated with both qualitative and quantitative paradigms are not as pure as is contended by proponents of monomethod research. For instance, with respect to qualitative research, the concepts of data saturation, informational redundancy and/or theoretical saturation (i.e. when no new or relevant information seem to emerge pertaining to a category and the It is legitimate and useful to examine and emphasize the differences between qualitative and quantitative research. We should remain aware, however, that this emphasis glosses over the many, sometimes great, differences within qualitative and quantitative research. Furthermore, our point here is that it also is legitimate to examine similarities that sometimes are present in applications of qualitative and quantitative research. The exploratory factor analysis is conducted after converting the themes to a ‘1’ if the theme is present for the research participant and a ‘0’ if the theme is not present for the research participant. This conversion yields what Onwuegbuzie (2003) refers to as an inter-respondent matrix (i.e. participant x theme matrix), consisting only of 0s and 1s that is subsequently converted to a matrix of bivariate associations among the responses pertaining to each of the emergent themes. These bivariate associations then are converted to tetrachoric correlation coefficients because the themes had been quantitized to dichotomous data (i.e. ‘0’ vs. ‘1’) and tetrachoric correlation coefficients are appropriate to use when one is determining the relationship between two (artificial) dichotomous variables. The matrix of tetrachoric correlation coefficients then becomes the basis of the exploratory factor analysis (see, for example Onwuegbuzie, Witcher, Collins, Filer, Wiedmaier and Moore 2007). Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 131 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins category development is well established and validated; Flick 1998; Lincoln & Guba 1985; Morse 1995; Strauss & Corbin 1990), although inherently interpretivist, have quantitative undertones. These concepts incorporate quantitative assumptions – including the idea of internal replication (i.e. findings that are replicated by other participants in the study). In order to reach conclusions about saturation or informational redundancy, some kind of formal (i.e. conscious) or informal (e.g. subconscious) assessment of degree or amount typically takes place to determine how exhaustive the data are. Also, any conclusion on the part of the qualitative researcher that saturation or informational redundancy has taken place is accompanied by some degree of confidence or even certainty (i.e. high probability) – whether or not this confidence is estimated (which is a strategy that a postpositivist researcher likely would pursue). As the early modern continental philosopher Immanuel Kant (1781/1998) pointed out, the categories of qualitative and quantitative are necessarily part of human thought and the conclusions we construct about entities that are important to us. With respect to quantitative research, techniques such as exploratory factor analysis and cluster analysis, although inherently postpositivist, have constructivist leanings inasmuch as for any given dataset, there are myriad mathematical solutions (i.e. factor/cluster structures) that can be constructed and the analyst has to make sense of the selected mathematical solution (i.e. meaningmaking). Indeed, there are numerous strategies (e.g. principal component analysis vs. factor analysis; correlation matrix vs. variance–covariance matrix; maximum likelihood vs. unweighted least squares vs. generalized least squares vs. principle axis factoring vs. alpha factoring vs. image factoring; eigenvalues; trace; scree plot; parallel analysis; number of iterations; orthogonal vs. oblique rotation; factor pattern matrix vs. factor structure matrix; communality estimates; internal replication techniques such as bootstrapping, jackknifing and cross-validation; cf. Henson, Capraro, & 132 Capraro 2004; Henson & Roberts 2006; Hetzel 1996; Onwuegbuzie & Daniel 2003) and criteria that are available for deciding on the most appropriate factor structure. With so many decisions to make, two or more analysts easily can arrive at different final factor solutions, which indicates that exploratory factor analysts serve, to a significant degree, as research instruments themselves – a concept that some postpositivist researchers might be reluctant to acknowledge due to a stance that social science inquiry should be objective and that postpositivist researchers should eliminate all biases (cf. Table 2). Interestingly, the role of postpositivist researcher-as-instrument is not restricted to analyses that are exploratory in nature (e.g. exploratory factor analysis, cluster analysis, multidimensional scaling) but also include analyses that involve null hypothesis significance testing (NHST). In fact, the more sophisticated the NHST-based analysis, the more subjective decisions the postpositivist analyst has to make, making him/her serve as a research instrument to an even larger degree. For example, structural equation modeling (SEM; e.g. Schumacker & Lomax 1996) and hierarchical linear modeling (HLM; e.g. Bryk & Raudenbush 1992) typically involve the analyst making numerous subjective decisions, including selecting how many and which models to test and selecting from the numerous goodness-of-fit indices and criteria available. Typically, the theoretical models are literally created by the researcher (after an inductive analysis of past research, current theory, hunches, etc.). Thus, it is difficult for advocates of any of the research paradigms to claim justifiably a one-to-one correspondence between ontology/epistemology and type of analysis (i.e. qualitative vs. quantitative). Moreover, we contend that pragmatist researchers who hold a mixture of philosophical positions (i.e. belonging to both quantitative and qualitative traditions) find it natural to combine statistical analyses with an array of qualitative analyses. In order to accomplish such an embed- INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches ded analysis, the mixed researcher has to make Gestalt switches from a quantitative lens to a qualitative lens and vice versa, going back and forth, multiple times (Kuhn 1962). According to Onwuegbuzie and Johnson (2006), this series of switching yields a new or consolidated analysis – such as cross-over mixed analyses – which, if effective, can yield more fully mixed meta-inferences that incorporate a strong use of both quantitative and qualitative assumptions and stances and that represent the strongest paradigmatic mixing. More generally in the mixed research literature, this purposeful switching is called the dialectical approach (Greene 2007; Johnson 2008). This approach can rely on a single individual trained in both qualitative and quantitative research or, if no qualified individual is available, it can rely on a research team composed of at least one qualitative and one quantitative researcher. A recent example of a qualitative–quantitative research team is found in Corden and Hirst (2008). Mixed research philosophical paradigms Pragmatism is only one of many stances that underlie mixed research (Greene 2007, 2008; Johnson et al. 2007). Current stances include the pragmatism-of-the-middle philosophy (Johnson & Onwuegbuzie 2004), pragmatism-of-the-right philosophy (Putnam 2002; Rescher 2000), pragmatism-of-the-left philosophy (Maxcy 2003; Rorty 1991), the anti-conflationist philosophy (Bryman 1992; Hammersley 1992; Layder 1993; Roberts 2002), critical realist orientation (Houston 2001; Maxwell 2004; McEvoy & Richards 2003, 2006), the dialectical stance (Greene 2008; Greene & Caracelli 1997; Maxwell & Loomis 2003), complementary strengths stance (Brewer & Hunter 1989; Morse 2003), transformativeemancipatory stance (Mertens 2003), a-paradigmatic stance (Patton 2002; Reichardt & Cook 1979), substantive theory stance (Chen 2006) and, most recently, communities of practice stance (Denscombe 2008). Table 3 provides one way of representing the major mixed research stances. In this table, these leading mixed research philosophical paradigms are presented (left column), alongside a summary of their major stances (middle column) and the core analyses that are associated with these stances (right column). This table represents a first attempt to link mixed research paradigms and stances to mixed analysis strategies. However, clearly, more work is needed here to enhance our understanding of the mixed analysis choices made by mixed researchers and thus address Greene’s (2008: 13) important question: ‘how do the assumptions and stances…influence inquiry decisions?’. CONCLUSIONS The present article represents a first attempt, albeit a tentative one, explicitly to provide a philosophical justification for conducting mixed analyses. As noted by Greene (2008: 12): ‘Our thinking about the nature and role of philosophical assumptions in our practice needs to make more practical sense, as well as offer possibilities to practitioners not yet envisioned, which is one key role of mixed methods theory’. Although we recognize that much more work is needed in this area, we hope that our article will motivate constructive and positive dialogues among mixed researchers and researchers representing qualitative- and quantitative-based paradigms (Onwuegbuzie 2002), so that, regardless of our differences in philosophical assumptions and stances, we all come to view all researchers as belonging to communities of interest (cf. Fischer 2001). Further, we hope that our article will motivate others either to refute or to build on the ideas and concepts we have presented. To the extent that our ideas and concepts make practical sense, we hope that our article will help better situate mixed analyses in the philosophy of science, thereby promoting mixed research as a distinctive methodology – consistent with the call made by Greene (2006). Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 133 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins TABLE 3: M IXED R ESEARCH PARADIGMS Paradigm/ Worldview Pragmatism-ofthe-middle philosophy Pragmatism-ofthe-right Pragmatism-ofthe-left Anti-conflationist Critical realist Dialectical stance Complementary strengths Transformativeemancipatory A-paradigmatic Substantive theory Communities of practice AND W ORLDVIEWS M IXED A NALYSIS A SSUMPTIONS Stance Offers a practical and outcome-oriented method of inquiry that is based on action and leads, iteratively, to further action and the elimination of doubt; paradigms routinely are mixed (Johnson and Onwuegbuzie 2004) Holding a moderately strong form of realism and a weak form of pluralism (Johnson et al. 2007) Antirealism and strong pluralism (Johnson et al. 2007) Methodology should not be conflated with technical aspects of method because the same method can be used by researchers with different ontological/epistemological stances; adoption of a more principled approach when combining methods – only appropriate to combine methods if a common ontological/epistemological stance can be maintained (McEvoy and Richards 2003) Mix of critical theory and a multilevel, discursive social scientific realism (Maxwell 2004) Dialogical engagement with paradigm differences that generatively produce new knowledge and insights (Greene 2007). Use of ‘dialectical pragmatism’ (i.e. examine qualitative and quantitative stances fully and dialectically and produce a combination solution that works best for the research question) (Teddlie and Johnson 2009) Paradigms are not necessarily incompatible but are substantively different; thus, methods used for different paradigms should be kept separate to preserve paradigmatic and methodological integrity (Greene 2007) Emancipatory, participatory and anti-discriminatory research that focuses directly on the lives, experiences and perceptions of marginalized persons or groups (Mertens 2003) Paradigms are logically independent and thus can be mixed; but although they are useful for reflection, they do not shape practical research decisions; rather, practical characteristics and issues related to the underlying context and problem drive these decisions (Greene 2007) Paradigms may be embedded or intertwined with substantive theories; yet, substantive issues and conceptual theories drive the mixed research, not paradigms (Greene 2007) Consistent with pragmatist philosophy but accommodates variations and inconsistencies that prevail within mixed research by promoting a diversity of researchers, allowing paradigms to operate at different levels, incorporating group influences on methodological decisions, shifting debates about paradigms to level of practice and research culture and allowing methods to be chosen based on their practical value for addressing a research problem (Denscombe 2008) References Barton, A. and Lazarsfeld, F. (1955) Some functions of qualitative data analysis in sociological research. Sociologica, 1, 321-361. Bazeley, P. (2003) Computerized data analysis for mixed methods research. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 385-422) Thousand Oaks CA: Sage. 134 AND THEIR Core Mixed Analysis Strategies Sequential, concurrent, cross-over Sequential, concurrent, cross-over Sequential, concurrent, cross-over Sequential Sequential, concurrent Sequential, concurrent, cross-over Parallel Sequential, concurrent, cross-over Sequential, concurrent, cross-over Sequential, concurrent Sequential, concurrent, cross-over Bazeley, P. (2006) The contribution of computer software to integrating qualitative and quantitative data and analyses. Research in the Schools, 13(1), 64-74. Becker, H. S. (1970) Sociological work: Method and substance. New Brunswick NJ: Transaction Books. Becker, H. S., Geer, B., Hughes, E. C. and Strauss, A. L. (1977) Boys in white: Student culture in medical school. New Brunswick NJ: Transaction INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches Books. (Original work published by University of Chicago Press 1961) Bergman, M. (Ed.) (2008) Advances in mixed methods research. Theories and applications. Thousand Oaks CA: Sage. Blaikie, N. (1993) Approaches to social enquiry. Cambridge: Polity. Brewer, J. and Hunter, A. (1989) Multimethod research. Thousand Oaks CA: Sage. Bryman, A. (1988) Quantity and quality in social research. London: Urwin Hyman. Bryman, A. (1992) Quantitative and qualitative research: Further reflections on their integration. In J. Brannen (Ed.), Mixing methods: Qualitative and quantitative research (pp. 89-111) Aldershot: Avebury Press. Bryk, A. S. and Raudenbush, S. W. (1992) Hierarchical linear models: Applications and data analysis methods. Newbury Park CA: Sage. Campbell, D. T. and Fiske, D. W. (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Caracelli, V. W. and Greene, J. C. (1993) Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15 195-207. Chen, H. T. (2006) A theory-driven evaluation perspective on mixed methods research. Research in the Schools, 13(1), 75-83. Chi, M. T. H. (1997) Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6, 271-315. Collins, K. M. T., Onwuegbuzie, A. J. and Sutton, I. L. (2006) A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67-100. Cook, T. D. and Campbell, D. T. (1979) Quasiexperimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Corden, A. and Hirst, M. (2008) Implementing a mixed methods approach to explore the financial implications of death of a life partner. Journal of Mixed Methods, 2, 208-220. Creswell, J. W. and Plano Clark, V. L. (2007) Designing and conducting mixed methods research. Thousand Oaks CA: Sage. Creswell, J. W. and Plano Clark, V. L. (2008) Conducting Mixed Methods Research and the Practice of Research. Workshop presented at the annual Mixed Methods Conference. Cambridge UK. Curtis, S., Gesler, W., Smith, G. and Washburn, S. (2000) Approaches to sampling and case selection in qualitative research: Examples in the geography of health. Social Science and Medicine, 50, 1001-1014. Denscombe, M. (2008) Communities of practice: A research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2, 270-283. Denzin, N. K. and Lincoln, Y. S. (2005) Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin and Y. S. Lincoln (Eds.), Sage handbook of qualitative research (2nd edn, pp. 191-216) Thousand Oaks CA: Sage. Firestone, W. A. (1993) Alternative arguments for generalizing from data, as applied to qualitative research. Educational Researcher, 22(4), 16-23. Fischer, G. (2001) Communities of interest: Learning through the interaction of multiple knowledge systems. In S. Bjornestad, R. Moe, A. Morch and A. Opdahl (Eds.), Proceedings of the 24th IRIS Conference, Ulvik, Norway. Flick, U. (1998) An introduction to qualitative research: Theory, method and applications. London: Sage. Fox, J. (1997) Applied regression analysis, linear models and related methods. Thousand Oaks CA: Sage. Fryer, D. and Feather, N. T. (1994) Intervention techniques. In C. Cassell and G. Symon (Eds.), Qualitative methods in organizational research (pp. 230-247) London: Sage. Glaser, B. G. and Strauss, A. L. (1967) The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine. Gorard, S. and Smith, E. (Eds.) (2006) Special Issue on ‘Combining numbers with narratives.’ Evaluation and Research in Education 19(2) Retrieved March 28 2009, from http://www.multilingual-matters .net/erie/019/2/ Greene, J. C. (2006) Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93-98. Greene, J. C. (2007) Mixed methods in social inquiry. San Francisco: Jossey-Bass. Greene, J. C. (2008) Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2, 7-22. Greene, J. C. and Caracelli, V. J. (1997) Defining and describing the paradigm issue in mixedmethod evaluation. In J. C. Greene and V. J. Caracelli (Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms (New Directions for Evaluation, No. 74, pp. 5-17) San Francisco: Jossey-Bass. Greene, J. C., Caracelli, V. J. and Graham, W. F. (1989) Toward a conceptual framework for Volume 3, Issue 2, August 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES 135 Anthony J Onwuegbuzie, R Burke Johnson and Kathleen MT Collins mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274. Guba, E. G. (1990) The alternative paradigm dialog. In E. G. Guba (Ed.), The paradigm dialog (pp. 17-27) Newbury Park CA: Sage. Hacking, I. (1983) Representing and intervening: Introductory topics in the philosophy of natural science. New York: Cambridge University Press. Hammersley, M. (1992) Deconstructing the qualitative–quantitative divide. In J. Brannen (Ed.) Mixing methods: Qualitative and quantitative research (pp. 39-55) Aldershot: Avebury Press. Henson, R. K., Capraro, R. M. and Capraro, M. M. (2004) Reporting practice and use of exploratory factor analysis in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61-72. Henson, R. K. and Roberts, J. K. (2006) Use of exploratory factor analysis in published research. Educational and Psychological Measurement, 66, 393-416. Heron, J. and Reason, P. (1997) A participatory inquiry paradigm. Qualitative Inquiry, 3, 274-294. Hetzel, R. D. (1996) A primer on factor analysis with comments on patterns of practice and reporting. In B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 175-206) Greenwich CT: JAI Press. Houston, S. (2001) Beyond social constructionism: Critical realism and social work. British Journal of Social Work, 31, 845-861. Jick, T. D. (1983) Mixing qualitative and quantitative methods: Triangulation in action. In J. Vam Maanen (Ed.), Qualitative methodology (pp. 135-148) Thousand Oaks CA: Sage. Johnson, R. B. (2006) Special issue on ‘New directions in mixed methods research.’ Research in the Schools, 13(1) Retrieved July 8 2008, from http://www.msera.org/rits_131.htm Johnson, R. B. (2008) Editorial: Living with tensions. Journal of Mixed Methods Research, 2, 202-207. Johnson, R. B. and Christensen, L. B. (2008) Educational research: Quantitative, qualitative and mixed approaches (3rd edn) Thousand Oaks CA: Sage. Johnson, R. B. and Onwuegbuzie, A. J. (2004) Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. Johnson, R. B., Onwuegbuzie, A. J. and Turner, L. A. (2007) Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112-133. Kant, I. (1781/1998) The Cambridge edition of the 136 works of Immanuel Kant: Critique of pure reason. Cambridge UK: Cambridge University Press. Kennedy, M. (1979) Generalizing from single case studies. Evaluation Quarterly, 3, 661-678. Kidder, L. H. and Fine, M. (1987) Qualitative and quantitative methods: When stories converge. In M. M. Mark and R. L. Shotland (Eds.), Multiple methods in program evaluation (New Directions of Program Evaluation, No. 35, pp. 57-75) San Francisco: Jossey-Bass. Kuhn, T. S. (1962) The structure of scientific revolutions. Chicago: University of Chicago Press. Lather, P. (1986) Research as praxis. Harvard Educational Review, 56, 257-277. Lather, P. (2006) Paradigm proliferation as a good thing to think with: Teaching research in education as a wild profusion. International Journal of Qualitative Studies 19(1), 35-57. Layder, D. (1993) New strategies in social research: An introduction and guide. Cambridge UK: Polity Press. LeCompte, M. D. and Preissle, J. (1993) Ethnography and qualitative design in educational research (2nd edn) San Diego CA: Academic Press. Lee, Y-j. and Greene, J. C. (2007) The predictive validity of an ESL placement test: A mixed methods approach. Journal of Mixed Methods Research, 1, 366-389. Leech, N. L. and Onwuegbuzie, A. J. (2007) An array of qualitative data analysis tools: A call for qualitative data analysis triangulation. School Psychology Quarterly, 22, 557-584. Leech, N. L. and Onwuegbuzie, A. J. (2008) Qualitative data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond. School Psychology Quarterly, 23, 587-604. Leech, N. L. and Onwuegbuzie, A. J. (Eds) (2010) Special issue on ‘Teaching mixed methods.’ International Journal of Multiple Research Approaches, 4(1), in press. Li, S., Marquart, J. M. and Zercher, C. (2000) Conceptual issues and analytical strategies in mixed-method studies of preschool inclusion. Journal of Early Intervention, 23, 116-132. Lincoln, Y. S. and Guba, E. G. (1985) Naturalistic inquiry. Beverly Hills CA: Sage. Louis, K. S. (1982) Sociologist as sleuth: Integrating methods in the RDU study. American Behavioral Scientist, 26(1), 101-120. Madey, D. L. (1982) Some benefits of integrating qualitative and quantitative methods in program evaluation, with some illustrations. Educational Evaluation and Policy Analysis, 4, 223-236. INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 2, August 2009 Call for mixed analysis: A philosophical framework for combining qualitative and quantitative approaches Mark, M. M. and Shotland, R. L. (1987) Alternative models for the use of multiple methods. In M. M. Mark and R. L. Shotland (Eds.), Multiple methods in program evaluation (New Directions of Program Evaluation, No. 35 pp. 95-100) San Francisco: Jossey-Bass. Maxcy, S. J. (2003) Pragmatic threads in mixed methods research in the social sciences: The search for multiple modes of inquiry and the end of the philosophy of formalism. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 5189) Thousand Oaks CA: Sage. Maxwell, J. A. (1996) Qualitative research design. Newbury Park CA: Sage. Maxwell, J. A. (2004, April) Realism as a stance for mixed methods research. Paper presented at the annual meeting of the American Educational Research Association, San Diego CA. Maxwell, J. A., Bashook, G. and Sandlow, C. J. (1986) Combining ethnographic and experimental methods in educational evaluation. In D. M. Fetterman and M. A. Pittman (Eds.), Educational evaluation: Ethnography in theory, practice and politics (pp. 121-143) Thousand Oaks CA: Sage. Maxwell, J. A. and Loomis, D. M. (2003) Mixed methods design: An alternative approach. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 241-272) Thousand Oaks CA: Sage. McConney, A., Rudd, A. and Ayres, R. (2002) Getting to the bottom line: A method for synthesizing findings within mixed-method program evaluations. American Journal of Evaluation, 23, 121-140. McEvoy, P. and Richards, D. (2003) Critical realism: A way forward for evaluation research in nursing? Journal of Advanced Nursing, 43, 411-420. McEvoy, P. and Richards, D. (2006) A critical realist rationale for using a combination of quantitative and qualitative methods. Journal of Research in Nursing, 11, 66-78. Mertens, D. (2003) Mixed methods and the politics of human research: The transformativeemancipatory perspective. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 135-164) Thousand Oaks CA: Sage. Mertens, D. (2008, July) Transformative mixed methods research. Workshop presented at the annual mixed methods conference. Cambridge UK. Miles, M. B. and Huberman, A. M. (1994) Qualitative data analysis: An expanded sourcebook (2nd edn) Thousand Oaks CA: Sage. Morrow, R. A. and Brown, D. D. (1994) Critical theory and methodology (Contemporary Social Theory) Thousand Oaks CA: Sage. Morse, J. M. (1995) The significance of saturation. Qualitative Health Research, 5, 147- 149. Morse, J. M. (2003) Principles of mixed methods and multimethod research design. In A. Tashakkori and C. Teddlie (Eds), Handbook of mixed methods in social and behavioral research (pp. 189-208) Thousand Oaks CA: Sage. O’Cathain, A. (2008, July) ‘An…