Part V: Adaptation for Educators

Adaptation Guidelines for Educators

            Since we began presenting our research on evaluating wiki quality, educators have asked us to share our Wiki Quality Instrument (WQI) and our thoughts on assessing wiki quality for classroom purposes. We designed the WQI for the large scale evaluation of wikis used for diverse purposes in diverse educational settings, so in its current form the WQI is not a suitable tool for assessing individual students in particular classrooms. We do believe, however, that the WQI has some application for assessing technology use in schools and school districts.  We also believe that the categories of the WQI could serve as a useful conceptual framework for teachers designing rubrics for evaluating student performances in online learning environments. In this section, we describe several ideas that we have for adapting the WQI for educational settings. First, we provide a bit of background on the WQI as a way of introducing some of its strengths and limitations. Second, we suggest possible ways of using the WQI to assess technology use at the district and school levels. Then, we present findings from our research about how teachers’ assess wiki quality as an introduction to our suggestions for using the WQI as a conceptual framework for developing wiki rubrics.
Before beginning, it is important to note that our early research agenda was designed to assess wiki usage in schools, not to be able to translate those findings directly into actionable advice. What follows therefore is advice which is not the result of a set of research studies designed to tell educators how to evaluate wikis; rather it is advice that emerges from people who have spent years thinking about wikis in schools and are extrapolating from general experience rather than specific research findings. (By way of analogy, I recently went to my orthopedic surgeon for a check-up several years after knee surgery to repair a traumatic injury to my knee cartilage. He suggested that I was now well enough to participate in sports, but I should probably not pick up marathon running. He emphasized that his advice was not based on specific scientific research; there are not randomized controlled trials evaluating the impact of marathon running on victims of traumatic knee cartilage injuries. Rather the foundation for his advice came from spending a lot of time looking at and thinking about knees. The advice herein is in a similar vein, based on our personal insights rather than findings from investigations of specific research questions.)

Background on the WQI

 
The WQI was designed to evaluate an extraordinarily diverse set of learning environments. We examine wikis used in pre-K classrooms and senior electives, in AP classes and ESL classes, in every subject area imaginable, all across the country and in U.S. schools overseas. Across these different sectors, wikis are used for all kinds of purposes: as teacher Web sites, electronic worksheets, individual presentations, collaborative workspaces, and portfolios. Moreover, we can only see these learning environments from one perspective: the wiki itself. We know that every wiki we analyze is nested within a larger ecology of learning, an ecology which we cannot examine in its entirety.
To consistently measure dimensions of activity across these diverse spaces, our instrument takes very coarse measures. We attempt to identify the presence or absence of 24 behaviors that appear across different types of wiki learning environments that provide opportunities for students to develop 21st century skills. During the course of our pilot testing, we tried several other approaches. We attempted to consistently measure the frequency of a behavior, but we were stymied by the diverse size of wikis. If something happens 20 times on one page, is that more frequent than 2 times each on 8 different pages? We also attempted to qualitatively assess certain actions; for instance by trying to measure whether typeface formatting was used “decoratively” or “substantively;” We found that our research assistants could not consistently agree on what constituted substantive use of formatting.  As a result, the WQI does not necessarily evaluate whether or not high quality student work is happening. Since we cannot reasonably compare the quality of copyediting done by third graders with the copyediting done by twelfth graders, we do not attempt to evaluate the quality of copyediting at all. Rather we assess whether or not there is any copyediting to be found. The WQI instrument is best understood as a tool to identify whether or not certain preconditions for 21st century learning are present on a wiki.
For districts and schools that are in the early phases of evaluating technology initiatives, identifying these kinds of preconditions might prove to be a very useful exercise. The WQI offers a cost-effective, relatively simple method for evaluating whether students are using wikis in ways that could potentially foster the development of 21st century skills. For classroom teachers who have specific learning goals in mind for students, the broad indicators of the WQI are less useful. They may, however, be useful for teachers who are using wikis without clear goals, who need to develop specific benchmarks of quality. In the sections that follow, I suggest how school leaders and classroom teachers might apply or adopt the WQI for different circumstances.
Before beginning, it is important to note that WQI in its present form would only be useful in evaluating student activity with wikis. While other media are similar, such as blogs or the Google Docs package, the affordances of these tools does not align exactly with the affordances of wikis. For instance, it is not really possible to “copyedit” an individually-authored response to a blog post, and there are functions that can be performed with Google Docs and blogs that cannot be accomplished with wikis. At this time the WQI is well suited only to evaluate the use of wikis. In Part 6: Adaptation Guidelines for Researchers, we discuss suggestions for using the WQI as a template for devising other quality instruments for online learning environments. Here, however, we restrict ourselves to only examining the use of wikis in schools and districts.
 

Using the WQI to assess wiki usage in schools and districts

 
A school or district administrator charged with evaluating the efficacy of school or district-wide technology initiatives faces many of the same challenges that faced our research team as we sought to evaluate wiki usage at a national scale. Across a district, wikis can be used in many grade levels, in many subject areas and for many purposes. The WQI is one tool that could be used or adapted to assess the degree to which wikis in a school or district provide opportunities for 21st century skill development.
Ideally, a school or district-wide assessment of wiki usage should evaluate the degree to which wikis support a school or district’s learning goals related to technology. Unfortunately, in our experience, many American schools and districts do not have organized goals for learning technologies. Instead they support radical teacher autonomy and let 1,000 flowers bloom.
To highlight this common omission in district technology planning, it is helpful to look at school systems that do have clear visions for how technology should support student learning. For instance, in 2008 the Ministry of Education in Singapore launched the Third Masterplan for ICT in Education (http://ictconnection.edumall.sg/), which directed educators across Singapore to focus on using technology to support only two learning goals: to develop student capacities for self-directed learning and collaborative learning.  There are, of course, other learning goals that technology can support. However, by choosing to focus on just two, Singapore has the opportunity to develop a coherent approach to teacher training, student evaluation and technology investment across the nation’s schools.
American school districts, in our experience, almost never have this clarity of focus around technology investment. The norm is probably to have no clear goals for how those investments should improve student learning outcomes. Often times, technology is viewed as an end in itself—where teachers use more technology so that students learn more about technology—rather than as a suite of tools to help students develop mastery of more important learning goals. In some districts, there are vaguely articulated goals for technology use, sometime related to ideas around 21st century thinking, but these goals lack shared definitions of key terms and shared outcome measures.
For our purposes here, let us imagine how three different types of districts might use the WQI. The first district has very clearly demarked goals for student outcomes from technology investments, the second district has a vague notion that technology should support 21st century learning but no institutional consensus or conception of what that means, and the third district has defined goals for their technology investments.                        For districts with clearly demarked goals for student outcomes from technology investments, it might make sense to use only those parts of the WQI that correlate with district goals. For instance, to continue with the example of Singapore, educators in that country might use the Complex Communication subdomain to capture collaborative learning opportunities with wikis. Some of the questions on the Expert Thinking subscale might capture certain elements of what Singaporeans mean by “self-directed learning,” but those items which do not correlate could be discarded. If Singaporeans are not interested in the development of New Media Literacy, that entire scale could be ignored. The advantage of discarding unnecessary parts of the WQI is twofold. First, a shortened instrument will require less time to apply. Second, analyzing elements unrelated to the core learning goals of a district can lead to distractions and tangential analyses. With the limited time available for district staff to analyze student work, it makes sense to focus carefully on school-wide and district-wide goals.
For districts with fuzzy goals or without clear learning goals around technology use, using the entire WQI might provide a kind of “baseline assessment” of the ways in which wikis are used throughout a school or district. This kind of data can be useful in jumpstarting conversations around learning goals for technology interventions and investments. For instance, if a school found that its use of wikis paralleled national trends, with most wikis serving as individual content delivery devices with minimal collaboration, that could open conversations about the kinds of support teachers need to conduct more innovative work. If wikis in a school scored low on the WQI across the board, that could also stimulate conversation about why teachers were using wikis and what help they need to use them successfully. If a school’s wikis scored high in a particular dimension, then it could be sensible to focus on improving outcomes within that domain. If a school’s wikis score highly in several domains but not in another, then it could be sensible to focus on the weak area. The best pathway for improvement would depend upon a wide variety of factors: how much capacity the school has to take on an instructional initiative, how important wikis are to teaching practice, the percentage of faculty members using wikis, the support that can be engendered for this kind of work among key stakeholder groups, and so forth. An analysis of wiki quality can provide some baseline data for school and district administrators to evaluate whether or not wikis are providing opportunities for students to develop 21st century skills. That baseline data can then be used as a shared text to ground a conversation about next steps for the district.
For districts that are exclusively focused on raising scores on standardized tests of reading comprehension and mathematics, the WQI is unlikely to be of much help. There are not compelling reasons to believe that opportunities to develop expert thinking, complex communication, and new media literacy will be correlated with gains of scores on state tests.
In terms of the practicalities of using the WQI in a school or district self-study, the procedures that we outline above could be simplified and used within a district. The first step would be collecting a district-wide sample of wikis. If the district has a district-wide wiki provider, like a contract with PBworks or Wikispaces, then obtaining a sample of wikis is simply a matter of randomly sampling from the list of all wikis created on the wiki service. If teachers are individually creating wikis with their own accounts, then the best strategy would be to survey all teachers in the district and ask them to share their wiki URLs, and then sample from this list. (An alternative approach would be to randomly sample a set of teachers, and personally request wiki URLs from them. This might be more effective if teachers are unlikely to complete district surveys).  How big should the sample be? In all likelihood the optimal number is higher than the number that a district can afford to evaluate, so districts should sample as many as they can afford the time to examine. If very few wikis are used in the district, a team might be able to examine all of them. But if a team cannot examine every wiki, a randomly drawn sample will work fine. Bigger samples are better, but small samples are better than doing no assessment at all.
With a sample of wikis identified, the next step would be to identify which parts of the WQI to use. Use only the quality subdomains that align with district goals. If the district has no goals, consider using the whole instrument as a mechanism for developing some baseline data about technology use. Most districts should consider collecting demographic data in terms of which subject areas and grades levels are supported by the wikis, so that the district can analyze wiki usage in different parts of the organization.
The protocols described in Part II of this document lay out a sophisticated series of strategies for using multiple raters to evaluate wiki learning environments and for resolving disagreements. While our protocols call for wikis to be independently rated by two coders, districts may not be able to devote the resources to fully using our protocols. Having only one rater evaluate each wiki is not perfect, but it is better to have one rating per wiki than not doing any assessment at all. Educators should not let the perfect be the enemy of the good.
Schools and districts interested in conducting these kinds of study are welcome to get in touch with the first author at justin@edtechteacher.org for further advice and counsel on this process. In most cases he can offer some guidance over the phone, and he may be available for consulting or partnering on a larger research study.

Adapting the Wiki Quality Instrument for Classroom Use

 
In this section we discuss how the WQI could be adapted by classroom teachers for evaluating student activity and learning with wikis. Before directly considering adapting the WQI for K-12 settings, it is worth providing some context about how classroom teachers typically evaluate wiki quality. In our research, we spent a year conducting qualitative research in U.S. schools trying to answer the question “How do K-12 teachers define and assess wiki quality?” We found that teachers often could articulate clear learning goals for their wikis: they hoped that wikis would help students develop collaboration skills, facilitate course logistics, build students’ technological fluency, and allow students to deepen and demonstrate understanding of course materials.
That said, when we asked teachers and students how high quality work on wikis was evaluated, we got a different set of answers. Teachers told us that they primarily evaluate students on whether they contributed to the wiki at required intervals, whether they included factually correct content, and whether they met project guidelines. In short, rather than evaluating students on whether they demonstrated mastery of the stated learning goals, teachers evaluated students on whether or not they followed the directions of the assignment.
The items of the WQI can help teachers think about new ways to assess 21st century skills in the domains of expert thinking, complex communication, and new media literacy. The items will probably not be useful to teachers by themselves. The Wiki Quality Instrument was designed to evaluate wikis across a diverse population of wiki used in many subjects, grades, and schools. As a result, our measures are very coarse, compared to the fine grained measures that teachers should use to evaluate their students. For instance, we use the WQI to assess whether or not copyediting occurs on a wiki. We do not assess whether students do that copyediting thoroughly or effectively, we simply note if the activity exists. Teachers, however, should be interested in whether a student copyeditor assesses an entire piece thoroughly, finds a high percentage of errors, corrects them accurately, and so forth.
Thus, the items of the WQI are probably better suited to helping teachers brainstorm rubric categories than for generating benchmarks in specific categories. For instance, imagine a teacher who is hoping to have seventh grade students in an Earth Science class use wikis to develop collaboration skills. One way to begin developing a rubric for her wiki project would be to look at the Complex Communication items in the WQI. She could then decide which categories would be relevant to her rubric. For instance, she might decide that students have no need to use the wiki for scheduling, so that should be left out of the rubric. She might decide that commenting is essential to the project, and then develop criteria for four benchmarks: Exceeds Standards, Meets Standards, Approaches Standards, Just Beginning.  The Exceeding Standards benchmark  might be defined as “Providing specific, constructive feedback on multiple pages for all collaborators,” and a Just Beginning might be defined as “Adds comments to at least one collaborators page, but comments are vague general praise or inappropriate criticism.” Certain aspects of the standards might be tied even more specifically to the wiki project goals or to the Earth Science content.
This teacher also might decide to create one rubric category drawing on several items from the WQI. For instance, she might have one category called “Collaboration”, where the Just Beginning benchmark describes concatenation, the Approaches Standard benchmark describes copyediting, the Meets Standards benchmark describes co-construction, and the exceeds standards benchmark describes specific performance criteria for high quality co-construction in the context of the wiki project goals.
The WQI has potential as a source of inspiration for classroom teachers as they develop rubrics for work on wikis, but by itself it will not be sufficient. Wiki-using educators need to think carefully about their learning goals, and then develop strategies for communicating those goals to students. The WQI offers a framework for breaking down 21st century skills into some of their constituent parts. Those parts may be useful in inspiring teachers to consider how to develop rubrics for wiki work that evaluate the degree to which students demonstrate mastery of the learning goals associated with a wiki project.
 

Creative Commons License
Wiki Quality Instrument and Protocol by Justin Reich is licensed under a Creative Commons Attribution 3.0 Unported License.
Based on a work at www.edtechresearcher.com.