14.5 Reliability in unobtrusive research

This final section of this chapter investigates a few particularities related to reliability in unobtrusive research projects that warrant our attention. These particularities have to do with how and by whom the coding of data occurs. Issues of stability, reproducibility, and accuracy all speak to the unique problems—and opportunities—with establishing reliability in unobtrusive research projects (Krippendorff, 2009). ^[1]

Stability refers to the extent to which the results of coding vary across different time periods. If stability is a problem, it will reveal itself when the same person codes the same content at different times and comes up with different results. Coding is said to be stable when the same content has been coded multiple times by the same person with the same result each time. If you discover problems of instability in your coding procedures, it is possible that your coding rules are ambiguous and need to be clarified. Ambiguities in the text itself might also contribute to problems of stability. While you cannot alter your original textual data sources, simply being aware of possible ambiguities in the data as you code may help reduce the likelihood of problems with stability. It is also possible that problems with stability may result from a simple coding error, such as inadvertently writing a 1 instead of a 10 on your code sheet.

Reproducibility, also referred to as inter-rater reliability (Lombard, Snyder-Duch, & Campanella Bracken, 2010), ^[2] is the extent to which your coding procedures will yield the same results when the same text is coded by different people. We covered this problem in Chapter 9 when we talked about reliability of quantitative measures. Cognitive differences among the individuals coding data may result in problems with reproducibility, as could ambiguous coding instructions and random coding errors.

One way of overcoming problems of reproducibility is to have coders code together. While working as a graduate research assistant, a colleague participated in a content analysis project in which four individuals shared the responsibility for coding data. To reduce the potential for reproducibility problems with their coding, they conducted the coding at the same time in the same room, so they could consult one another when they ran into problems or had questions about what they were coding. The coders grew to have a shared understanding of how to code various bits of data by resolving ambiguities together.

Finally, accuracy refers to the extent to which your coding procedures correspond to some preexisting standard. Let’s say you are interested in the accessibility of informational pamphlets and brochures to clients of a public health clinic or students at your university. You could get a sample of the documents given to your target population and code using your own scheme—perhaps looking at reading level, attractiveness, and organization. To ensure the accuracy of your coding, you could consult the Centers for Disease Control’s Clear Communication Index, a standard measure of the clarity of a written product. ^[3]

This example presumes that a standard coding strategy has already been established for whatever text you’re analyzing. Official standards may not exist for the type of text you are analyzing but perusing the prior literature for the collective wisdom on coding on your topic area is time well spent. Scholarship that is focused on similar data or coding procedures will certainly help you clarify and improve your own coding procedures.