Adaptation Guidelines for Researchers
We believe that the online data warehouses of Web 2.0 learning environments provide an extraordinary opportunity for researchers to study the processes of learning in real-time and at scale. Large scale content analysis using instruments like the WQI represents one method of mining this new vein of data. We hope that other researchers will consider conducting studies similar to ours and help map the evolving landscape of social media use in classrooms.
Applying the Current Version of the WQI in Alternate Settings
Certain kinds of research could be conducted using the WQI without any modification. For instance, researchers could look at the use of PBworks wikis in K-12 context in other countries, especially if those education systems have strong similarities to the U.S. system. It also may be possible to use the WQI to assess wikis used in higher education contexts, although it may be that certain kinds of measurable wiki behaviors appear in higher education contexts that we did not find in our analysis of K-12 settings. If this is true, then the new items would need to be developed to assess those behaviors.
Modifying the WQI to Study other Wiki Providers
Most additional studies of the use of social media and wikis in K-12 settings, however, will probably require modifying or adapting the WQI. One promising research study would be to compare behaviors on Wikispaces with patterns of behavior on PBworks. Our hypothesis is that the two platforms would be used in very similar ways, but that hypothesis deserves to be tested. Since the Wikispaces platform has slightly different affordances than the PBworks platform, certain items might need to be adjusted. In particular, Wikispaces has no special place for comments on a content page, but it does have a threaded discussion board associated with every content page. Thus, the criteria for commenting and discussion might need to be altered.
To make these modifications, it would first make sense to search the literature and examine whether other scholars have analyzed these elements of Wikispaces. If so, the research methods of other scholars might provide guidance as to how best to analyze opportunities to develop 21st century skills on the Wikispaces platform. After this step, we recommend analyzing a random sample of perhaps 200 wikis from the population of interest and conducting an open-ended round of coding where researchers briefly describe the ways in which teachers and students using the discussion pages and content pages for commenting and discussion. From these descriptions, researchers can craft new items for the WQI or determine that the original version of the WQI adequately describes measurable behaviors on Wikispaces wikis. This same process would work for wiki platforms other than Wikispaces, such as a proprietary wiki solution within a university or Content Management System(CMS) platform or an installation of MediaWiki in a district or university.
Modifying the WQI to study Wiki Usage in Particular Curricular Domains
The further afield researchers go from the original purposes of the WQI, the more the instrument will need to be adapted. One logical direction for this line of research is to look at wikis used in specific contexts, such as in particular academic subject (Earth Science), a particular grade (7th grade), a particular set of schools (Iowa public schools), or some combination of these factors (7th Grade Earth Science classrooms in Iowa.) The WQI, without modification, could be used in such a context to coarsely measure opportunities for students to develop 21st century skills. However, in more specific settings, we believe that it is considerably more feasible to develop more fine grained measures of the quality of these 21st century skill learning opportunities. For instance, we found it extremely challenging to develop a taxonomy of behaviors associated with the development of expert thinking that might apply across grade levels and across subject areas. Within Earth Science, however, researchers might be able to identify a series of discursive moves on Earth Science wikis that represent students applying scientific thinking skills: posing questions, generating hypotheses, presenting evidence, testing hypotheses with evidence, designing experiments, and so forth. Researchers could also identify when students work with course content that aligns with state standards or extends beyond those standards.
It might be that these kinds of discursive moves could be added as additional dichotomous items to the WQI (Do students generate hypotheses concerning Earth Science phenomena?) However, it might also be possible to go even further and measure the actual quality of these discursive moves. Researchers could develop a rubric of criteria distinguishing high-quality and low-quality hypothesis generation as established by national standards, state standards, or other research efforts. Our various efforts at scalar measures of quality failed because the diversity of our sample was too great—how would one compare copyediting quality of 11th graders working on poetry with 3rd graders working on local history? Within a more specific context however, it might be possible to assess not just whether 7th graders in Iowa use their Earth Science wikis to generate hypotheses, but to evaluate the quality of their efforts. Similarly, researchers could evaluate not just whether 7th graders in Iowa use academic content about soil conservation, but whether they demonstrate mastery of Iowa 7th Grade Earth Science Standard 3, Interval Benchmark 2d: “Knows conservation methods that lessen the effects of soil erosion.”
If we were to attempt such as study, we would generally follow the process that we describe in the section on Part III: Developing the WQI, with a focus on these particular wikis in Iowa. We would begin with three overlapping methods for considering 7th grade Earth Science wiki quality. First, we would conduct observational studies and interview research with wiki using Earth Science teachers and students to evaluate how they use wikis, the role that wikis play in their classrooms, and the ways in which teachers and students define and assess high quality work in wikis. Second, we would conduct a literature review to identify existing measures of quality in online learning environments for science or science learning broadly. Third, we would assess our own theoretical frameworks for high quality work in these wikis, so as to be cognizant of our own beliefs and biases.
From these three methods, we would attempt to identify the domains that we believed would be worthy of analysis. It might prove that participation, information consumption, expert thinking, complex communication, and new media literacy remain germane categories for these Earth Science wikis. It seems very likely that we might need to refine our categories, such as making scientific thinking an explicit dimension of expert thinking or perhaps its own subdomain. It might be, however, that teachers and students discuss very different criteria for evaluating wiki quality and therefore entirely different domains of analysis are warranted.
With a better understanding of how wikis are used in their contexts in the instructional core, it might then make sense to examine a large set of Earth Science wikis to get a better sense of how these wikis are used. One of our first steps in developing the WQI was conducting a round of open coding on our sample of 1,799 wikis where we had researchers briefly describe how the wiki was being used. We conducted a second round of open coding on all 411 U.S. K-12 wikis to generate descriptions of behaviors that coders thought could promote the development of 21st century skills. From these qualitative descriptions, we began to develop decision rules for items in the WQI. For instance, the descriptions of various ways that students collaborated and communicated on wikis informed our development of the seven complex communication items on the WQI. A similar protocol might be used in a specific domain like Earth Science, where researchers examine a set of relevant wikis to begin to identify common discursive moves on Earth Science wikis that can be measured systematically. With a set of preliminary items developed from these efforts, researchers could then pilot test the items and determine their validity and reliability, in a process of developing a refined 7th Grade Iowa Earth Science Wiki Quality Instrument.
Adapting the WQI for the Study of Other Social Media Platforms in K-12 Settings
All of these suggestions thus far have involved adapting the WQI to be used in evaluating wikis in different settings. We also hope that researchers consider conducting similar kinds of large-scale content analysis on other kinds of online learning platforms. Public discussion boards, blogs, niche social networks like Ning, media sharing and commenting platforms like Voicethread, Google Docs, and other emerging social media platforms are all promising sites for exploring student learning in online venues. Certain items from the WQI might be adapted relatively easily to these new contexts; for instance blogs and wikis offer similar affordances for communicating with new media. Google Docs and wikis offer a similar breadth of possibility for collaboration. Platforms that are different from wikis, such as the media sharing and commenting site Voicethread, might require a very different analytical approach from the one that we took in analyzing wikis.
In addition to developing new instruments for evaluating quality in other online learning environments, researchers will also need to develop protocols for evaluating quality in those environments. To develop our protocols for measuring wiki quality, we needed to address the questions of “how often should we measure quality?” and “when should we measure quality?” Part III: Developing WQI protocols explains our processes for determining these protocols in wikis. Briefly, the question of how often to measure quality needs to be based on the complexity of typical quality development trajectories. If these trajectories are linear, then three data points are sufficient to capture the trend. The more complicated the trajectory, the more data points are needed to accurately represent the trajectory. We used wiki page edits to provide a basic model of wiki development trajectories. To answer the question of when should we measure quality, we used survival analysis to determine the range and median lifetime of wiki lifecycles. Findings from these analyses informed our decisions about when to measure wiki quality. While it might be reasonable to adopt our protocols for measuring other wiki hosting platforms, for researchers adapting the WQI for other platforms, it would be necessary to develop new protocols for measuring quality in these new domains.
Conclusion
Whether or not the specific items from the WQI can be adapted to other platforms, we hope that our general methodological approach can be adapted for use in the study of other public online learning environments. The key features of our work are that we:
- Access a population of online learning environments with extensive real-time data about individual behaviors
- Randomly sampled from those environments to study a representative sample of the population
- Devise protocols for the frequency of quality measures by assessing typical developmental trajectories for the online learning environments.
- Devise measures of quality. These measures emerge both from what teachers and students believe represents high-quality work and from what the literature says about elements of high-quality learning.
- Treat quality as time-varying, and measure quality at multiple time points throughout the lifecycle of the online learning environment. Treat quality as time-varying rather than a static feature of the online learning environment.
- Evaluate how covariates associated with the learning environment (teacher attitudes, school resources, student characteristics, etc.) affect the initial position and rate of change of the quality trajectories.
Our approach has yielded important insights about who uses wikis, how they are used, the kinds of learning opportunities that students have with wikis, and how these learning opportunities are distributed across schools serving different populations. We believe that similar approaches applied to other online learning environments could prove to be similarly productive, and we are available to consult with other researchers conducting these kinds of studies.
Wiki Quality Instrument and Protocol by Justin Reich is licensed under a Creative Commons Attribution 3.0 Unported License.
Based on a work at www.edtechresearcher.com.