Qualitative study critique: open source requirements development

By Mentor Cana on May 12, 2004 8:43 AM | Permalink | Comments (1) | TrackBacks (0)

The qualitative study (Scacchi, 2002) I have selected to critique is published in an electrical engineering oriented scholarly peer-review journal. The author is aware of his quantitative oriented audience and thus from the very beginning sets the expectations that the study is “… not about hypothesis testing or testing the viability of a perspective software engineering methodology or notational form” (p. 24). Similarly to Lincoln and Guba (1985) in defining naturalistic inquiry in terms of what it is not, Scacchi deems it necessary to define a qualitative research in terms that it is not quantitative research. The tensions emerging from the struggle to present non-quantitative type study to a quantitative expecting audience are pervasive throughout the article. Because of these tensions, in the attempt not to alienate his audience, the author has either decided to take many shortcuts—showing in the lack of proper definition and utilization of qualitative methods; or, the author himself is in the process of becoming familiar with various qualitative methods. In the rest of this paper I will concentrate on these struggles, attempts, and what could have been done better, not forgetting that maybe what the author has done is a purposefully chosen middle ground because the audience was not prepared for the full switch from quantitative to qualitative methodology and methods.

The core of this article is to understand the nature and the processes around requirements for the development of open source software (Scacchi, p. 24). Since the open source development framework is a new approach to software development, the author rightfully suggests qualitative methods for doing so: “… investigation of the socio-technical processes, work practices and community forms found in the open source software development. The purpose of this investigation, over several years, is to develop narrative, semi-structured (i.e. hypertextual) and formal computational models of these processes, practices and community forms” (p. 24). The preceding quote also suggest a mix method approach where the findings of the qualitative part of the study (i.e. ‘investigation’) would inform the quantitative part in building computational models. However, this article is restricted to the investigative part of the effort.

One of the bridges between quantitative and qualitative is built by setting the study as a pseudo experiment with control and ‘treatment group’, by choosing four distinct open source projects/communities for the purpose of abstraction and generalization (p. 25). In addition to staying at home within the quantitative realm because of his audience, the author has also tried to see what commonalities emerge amongst the four projects in order to discover a potential underlying open source requirement process and model that might be applicable across various projects and communities of practice.

In the description of the methodologies used in the study, the qualitative methodologies and methods are wrapped around a quantitative language: “The study reported here entails the use of empirical field study methods [9] that conform to the principles for conducting interpretive research design [10] as defined here” (Scacchi, p. 25), followed by the seven principles that are mostly qualitative in nature: i) ‘hermeneutic circle’, analyzing the emergent open source software requirement process and its sub-processes end-to-end, ii) contextualization of the processes and the activities, iii) “revealing the interaction of the researcher and the subject/artifacts”, iv) the principle of abstraction and generalization, v) ‘dialogic reasoning’ as an attempt to constant comparison and fitness of the emerging data to available theories about requirements and software development, vi) ‘multiple interpretation’ as means of viewpoint and source triangulation, and vii) alternative and revealing explanations mostly an attempt to identify and reconcile potential biases due to researcher’s interpretation of the data and the conclusions (pp. 25-26).

While in principle and from the perspective of qualitative research methodologies and methods wrapping a qualitative research within the language of quantitative does not do any justice to the clarity of the research, considering the context and the target audience, the author appears to have done a good job.

However, in the process of perhaps worrying too much about the publishing and proactively addressing readers’ skepticism, a very important and critical part has been overlooked: based on the article it does not seem possible to redo this study. The data collection and analysis are only mentioned expecting the readers to know how it is done. This seems to be one of the main weaknesses of the article especially when it attempts to introduce new methodologies and methods to a community that needs persuasion of the viability of qualitative data collection and analysis to understanding processes related to software development—which traditionally have been dealt by quantitative methods.

From the onset, the article begins with a clear statement about its focus and purpose. However, due to lack of qualitative language the author has described the methodology by describing seven interpretive principle [listed above], which mostly resemble various steps of a qualitative study, a sort of a participant observer oriented ethnography, termed virtual ethnography by the author (p. 25). The nature of ‘virtual ethnography’ is not defined or described in the article; it is only referenced. Again, a paragraph or two describing and presenting ethnography, especially virtual ethnography would have strengthened the study, especially since virtual ethnography seems to be leading this study. While it seems that virtual ethnography is a viable methodology, where the researcher has embedded himself as a junior developer in various online communities pertinent to the four projects presented in this study, no explanation is provided to the target audience why it is so.

The first two pages of the article presented some expectation in terms of describing the data collection in detail and how it would be analyzed. Instead, seven pages of the article are covered with about 50-60% of exhibits—manly screen shots of web pages, community portals, e-mail archives that contain functional and non-functional requirements, in narrative form, for open source software development. Instead of a textual analysis and detailed coding processes of various types of web pages that describe various types of requirement, after the author describes the narratives and the different discourses in which the requirements are expressed, he concludes that the requirements emerge in webs of discourses in reference to: a) e-mail and bulletin boards, b) systems vision statements, c) ideas about what systems ought to do, d) encouragement to develop potentially good functionality, and e) scholarly publication relevant to a particular open source project of interest to the scholarly community (p. 32). This finding is one of the conclusions that inform the rest of the conclusion. Yet, no data collection and analysis is demonstrated beyond the description that various types of online sources were analyzed and above was observed. The article should have as well reported on the data that was collected. Some data must have been collected, at least simple statistics as to the proportion of requirements found on each type of resource, dissected across the four projects. This would have provided further understanding about the commonalities and differences about the four projects.

As a result of author’s participant observation in the context of virtual ethnography, what have emerged are informalisms for open software systems requirements, which are much different than the classic software requirements engineering processes. In contrast with the classic software requirements engineering defined by the following five steps and process: requirements elicitation, requirements specification or modeling, requirements validation, and communicating the requirements to the needed stakeholders (p. 28); the informalisms that emerge from the virtual ethnography are as follows: a) community communication, b) scenarios of usage as linked web pages, c) HOWTO guides, d) links to external publication, e) open source software support websites and portal, f) bug tracking, g) lack of traditional software documentation, and h) emphasis on interoperability and open architectures (pp. 36-37).

These informalisms clearly demonstrate a qualitative type of study. They represent concepts and constructs that emerged as a result of the undertaking by the author. In terms of credibility, I am personally convinced by the conclusions, especially with the informalisms. This study has confirmed what I have observed and known for some time as a participant and user of various open source software tools and packages.

However, the study might not be as convincing for a broad range of readers without experience in open source software. Especially problematic is the lack of data collection process description as well as how they are analyzed. To this extend, textual analysis starting with open coding and followed by axial coding would have built the required level of credibility. In addition, it would have provided data upon which more precise claims could have been made about the differences and commonalities across the four projects and their respective communities of practice. In defense of the author however, his effort to present qualitative study to a quantitative oriented audience by strategically using language not to alienate his audience is worth noting.

There is an attempt for transferability by including four distinct open source projects built by various communities (gamming, operating systems and intern software, astronomy research, and academic software development). Author’s idea is that transferability will be established if central properties/process emerge which are common to all four distinct projects. Even though the informalisms that emerged are applicable to all four projects with varying degrees, the lack of data collection procedures and analysis put the findings in doubt in terms of the strength of the transferability claim.

Another major weakness is that the sampling process. How was it decided which web pages and web resources to look at, has not been described. It is not clear whether the aim was for purposeful sampling or exhaustivity.

Some of the drawbacks of this study might be due to the fact that the author claims these are preliminary findings. However, there is no reason why the preliminary findings should not have been presented in completeness and properly substantiated to build trustworthiness.

Despite some of the drawbacks, as far as presentation is concerned, because the author must have done some type of textual analysis and coding to see the emergent informalisms, the conclusion of the article suggests an emergent theoretical framework with the claim that open source software requirements development are distinct processes different than the underplaying process that govern requirements development for the classic/traditional software development. The claim is substantiated by the emergent software formalisms mentioned above.

In conclusion, is should be noted that the author might have made strategic decision in the use of language for better understanding by the target audience. Secondly, the author seems to present this study as a preliminary report about an ongoing study. However, there is no reason why this study should not have been a complete piece of research on its own. This could have been achieved by the inclusion of detailed planning procedures, the data collection and analyses methods, which would have demonstrated the required trustworthiness from the perspective of a qualitative research.

References:

Lincoln, Y., & Guba, E. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications, Inc.

Scacchi, W. (2002). Understanding the Requirements for Developing Open Source Software Systems. IEE Proceedings--Software, 149 (1), 24-39.