The Wiki Quality Instrument

  • Section 1: Summary Questions
  • Section 2: Definitions
  • Section 3: Decision Rules

Section 1: Summary Questions

We present these summary questions to introduce new educators and researchers to the WQI. These summary questions were not used by our trained researchers. In piloting the WQI, we found that we achieved the highest interrater agreement among trained coders when we used short, declarative sentences as decision rules. However, in presenting our findings at conferences, lab meetings, and in print, we have found that a list of summary questions is an effective way to communicate the items of the Wiki Quality Instruments to new audiences.

 

Summary questions of coding categories used in the Wiki Quality Instrument

CategorySub-category Summary question
Information Consumption
Course Materials Do students come to the wiki to access academic materials?
Information Gateway Do students come to the wiki to access links to other Web sites?
Participation
Contribution Does at least one student contribute, in any form, to the wiki?
Individual Pages Does at least one student own their own page on the wiki?
Shared Pages Does at least one pair (or group) of students own their own wiki page?
Ownership Do student(s) serve as primary facilitator and content creator of the wiki?
Expert Thinking
Academic Knowledge Does at least one student complete a task requiring academic knowledge (as opposed to simply writing about hobbies or one’s family)?
Information Organization Does at least one student complete a task requiring information organization, rather than routine information retrieval?
Metacognition Does at least on student reflect on his/her work product or process?
Crediting Does at least one student credit their sources of his/her work?
Teacher Feedback Do teachers provide feedback on student work?
Complex Communication
Concatenation Do multiple students add discrete sections of text to the same page?
Copyediting Does at least one student copyedit text created by another student?
Co-Construction Does at least one student substantively edit text created by another student?
Commenting Does at least one student comment upon another student’s work on the wiki?
Discussion Do students respond to each others’ comments for at least four conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
New Media Literacy
Formatting Does at least one student use formatting elements beyond plain text?
Links Does at least one student post a link to another page or document?
Hyperlinks Does at least one student create links rendered as simple text or images?
Images Does at least one student embed an image into a page?
Uploads Does at least one student upload a document?
Multimedia Does at least one student embed a multimedia element into a page?

Section 2: Definitions

The following key terms are referenced in the WQI decision rules:

Educator: A person acting in the capacity as a professional educator: including teachers, librarians, school administrators, IT staff, homeschool teachers, coaches, etc.

Student: Students are young people enrolled in a learning experience: classroom students, homeschool students, athletes on a team, members of a club, etc.

Others: Persons who do not fit in one of the above two categories including parents, families, and community members not employed by some kind of educational institution.

Creator: The primary content-creator or facilitator of the wiki environment. Usually the person with the most edits and the earliest edits. The person who “owns” and exerts the primarily editorial control over the content, structure, and presentation of the wiki. Each wiki should be coded as either “Student”, “Educator”, or “Other”, but not in multiple categories.

Participants: A person or persons who are not the creator who contribute to the wiki. These are people who made some kind of direct change to the wiki (page edit, comment post, document upload, etc.), but are not the creator.  Each wiki can be coded as Student, Educator, or Other, and can be coded in multiple categories.

Audience: A person or persons who are the intended viewers of a given wiki. Each wiki can be coded as Student, Educator, or Other, and can be coded in multiple categories.  By default, we assume that student-created wikis have an educator audience unless we find evidence to the contrary (such as a student who creates a wiki with study guides for all of her classes intended to be used just by other students and not evaluated by an educator). We also assume that educator-created wikis with “student-facing” materials—such as naming the wiki for a course or class, posting syllabi, instructions, assignments, etc.—have a student audience. For all other audience designations, we look for specific in text or contextual references to a specific audience. For instance, to code a student-created wiki as having a student-audience, we would need evidence that the student creator intends for other students to view the wiki (“Hey guys, I hope this study guide for chapter 4 is helpful; let me know if you have suggestions!). To code an educator-created wiki as having an educator audience, we look for similar kinds of evidence (such as a page of links to lesson plans for teaching about the area of a circle).

Section 3: Decision Rules

Note: I’m having some trouble rendering the table correctly below. In the meanwhile, here it is as a word document.

WQI For Web

 

Demographic Questions

Category Indicator Score as “0” if there is Score as “1” if there is
Meets Criteria
Viewable When the URL is entered, it returns an error because the wiki is private, deleted, or unchanged The wiki can be viewed in a browser.
US Based? No evidence that wiki supports learning in U.S. based schools Evidence that the wiki supports learning in U.S. based schools, including Department of Defense schools overseas
K-12 No evidence that the wiki supports teaching and learning in the K-12 grades. Evidence that the wiki supports teaching and learning in the K-12 grades.
Grade Level
K-5 No evidence that the wiki supports teaching and learning in the K-5 grades. Evidence that the wiki supports teaching and learning in the K-5 grades.
6-8 No evidence that the wiki supports teaching and learning in the 6-8 grades. Evidence that the wiki supports teaching and learning in the 6-8 grades.
9-12 No evidence that the wiki supports teaching and learning in the 9-12 grades. Evidence that the wiki supports teaching and learning in the 9-12 grades.
University No evidence that the wiki supports teaching and learning in a post-secondary setting. Evidence that the wiki supports teaching and learning in a post-secondary setting.

 


Category
Indicator Code as “0” if… If not “0”, code as text with the following information
Narrative Narrative The wiki has only been barely started with no information about the users or planned activities. (Note: sometimes even a URL is enough to begin to recognize probable users and activities) Write a few phrases describing the usage and purpose of the wiki
Create Date Create Date Never. If a wiki is viewable, the create date is viewable. Enter the date of the first version of the Front Page, which is automatically generated by PBworks.
Host School Name Cannot identify the hosting school or no school is involved Write the name of the primary hosting school
District Name Cannot identify the hosting district or no district is involved Write the name of the primary hosting district
Site Name Cannot identify another hosting institution (supra-district organization, public library, homeschool organization, etc.) or no other institution is involved Write the name of the primary hosting institution
School (2) Name Cannot identify the secondary hosting school or no secondary school is involved Write the name of the secondary hosting school
District (2) Name Cannot identify the secondary hosting district or no secondary district is involved Write the name of the secondary hosting district
Site (2) Name Cannot identify another secondary hosting institution or no other secondary institution is involved Write the name of the secondary hosting institution
Category Indicator Score as “0” if there is Score as “1” if there is
Subject Area
Contained Elementary No evidence that the wiki supports teaching and learning in a contained elementary classroom. Contained elementary refers to wikis used in elementary school classrooms where a single teacher teaches all subjects to a classroom. Evidence that the wiki supports teaching and learning in a contained elementary classroom. Contained elementary refers to wikis used in elementary school classrooms where a single teacher teaches all subjects to a classroom. In contained elementary wikis, it’s likely that you will give a 1 for “Contained Elementary” as well as a 1 in several subject areas.
English/ Language Arts No evidence that the wiki supports teaching and learning in English or Language Arts subjects. ELA classes are devoted to the study of English, reading, writing, literature, poetry, prose, textual analysis of drama, spelling, and so forth. Evidence that the wiki supports teaching and learning in English or Language Arts subjects. ELA classes are devoted to the study of English, reading, writing, literature, poetry, prose, textual analysis of drama, spelling, and so forth.
Math No evidence that the wiki supports teaching and learning in Mathematics. Evidence that the wiki supports teaching and learning in Mathematics.
Science and Engineering No evidence that the wiki supports teaching and learning in the natural and physical sciences- biology, chemistry, physics, geology, etc—or engineering courses.  Evidence that the wiki supports teaching and learning in the natural and physical sciences- biology, chemistry, physics, geology, etc—or engineering courses..
English as a Second Language No evidence that the wiki supports teaching and learning of English to non-native English speakers. (Sometimes also called English as a Foreign Language or English Language Learners). Evidence that the wiki supports teaching and learning of English to non-native English speakers. (Sometimes also called English as a Foreign Language or English Language Learners).
Social Studies No evidence that the wiki supports teaching and learning in the Social Studies, including history, religion, psychology, economics, geography, and other social sciences. Evidence that the wiki supports teaching and learning in the Social Studies, including history, religion, psychology, economics, geography, and other social sciences.
Computer Science and Technology No evidence that the wiki supports teaching and learning in Computer, Computer Science, Programming (Java, C++, etc), Technology Applications, or Typing Classes, or teaching and learning about educational Web/technology applications independent of an academic subject area.  Evidence that the wiki supports teaching and learning in Computer, Computer Science, Programming (Java, C++, etc), Technology Applications, or Typing Classes. Also includes wikis that support teaching and learning about educational Web/technology applications independent of an academic subject area.
Modern Foreign Language No evidence that the wiki supports teaching and learning in Modern Foreign Languages such as Spanish, French, German, Chinese, etc.  Evidence that the wiki supports teaching and learning in Modern Foreign Languages such as Spanish, French, German, Chinese, etc.
Classic Languages No evidence that the wiki supports teaching and learning in Classical Languages such as Latin and Greek  Evidence that the wiki supports teaching and learning in Classical Languages such as Latin and Greek
Art No evidence that the wiki supports teaching and learning in any form of visual, studio, or performing arts. Evidence that the wiki supports teaching and learning in any form of visual, studio, or performing arts.
Business No evidence that the wiki supports teaching and learning in business classes. Evidence that the wiki supports teaching and learning in business classes.
Health/Physical Education No evidence that the wiki supports teaching and learning in health classes, sexual education, physical education, or extra-curricular sports teams. Evidence that the wiki supports teaching and learning in health classes, sexual education, physical education, or extra-curricular sports teams.
Education No evidence that the wiki supports teaching and learning in education classes for teachers or pre-service teachers. Evidence that the wiki supports teaching and learning in education classes for teachers or pre-service teachers.
Library No evidence that the wiki supports a library site or teaching and learning in library or media studies courses. Evidence that the wiki supports a library site or teaching and learning in library or media studies courses.
Other No evidence that the wiki supports teaching and learning in a subject area not defined above. Evidence that the wiki supports teaching and learning in a subject area not defined above.
Creator
Student No evidence that a student is the primary content-creator or facilitator of the wiki Evidence that a student is the primary content-creator of facilitator of the wiki. Common evidence in this category includes (but is not limited too) the creator identifying themselves as a student; academic work which is in response to a particular prompt; naming or identifying the wiki as in service of a student or a working group; grammar, spelling and syntax characteristic of young people. 
Educator No evidence that an educator is the primary content-creator or facilitator of the wiki. Evidence that an educator is the primary content-creator or facilitator of the wiki. Common evidence includes: self-identifying as a teacher, posting instructional materials, posting instructions for academic work, naming or identifying a wiki as in service of a course or class rather than for a student or working group.
Other No evidence that an educator is the primary content-creator or facilitator of the wiki. Evidence that a non-student, non-educator is the primary content-creator or facilitator of the wiki. Rare.
Participant
Student No evidence that any student (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document. Evidence that at least one student (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.
Educator No evidence that any educator (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document. Evidence that at least one educator (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.
Other No evidence that any non-student, non-educator (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document. Evidence that at least one non-student, non-educator (not counting the creator) contributes to the wiki by making a page edit, leaving a comment, or uploading a document.

 

Audience
Student No evidence that students are meant to be the audience of the wiki. Evidence that students are meant to be the audience of the wiki. Most educator-created wikis have students as their audience. By default, if an educator is posting academic materials or instructions for academic work, we assume the audience of the wiki is students. If a student is the wiki creator, only code 1 here is there is evidence that the wiki is intended for viewing by other students. Evidence would include a specific reference to a student audience, such as a reference to how other students can use the presented material.
Educator No evidence that educators are meant to be the audience of the wiki. Evidence that educators are meant to be the audience of the wiki. Unless there is clear evidence to the contrary, we assume that all sites created by students have educators as an audience. If an educator is creating a resource site for other educators, code this category as a 1.
Other No evidence that others are meant to be the audience of the wiki. Evidence that others are meant to be the audience of the wiki. Only code 1 if there is explicit evidence that parents, experts or others are intended to be an audience for the site—such as an invitation to parents to view, messages from teachers and students to parents, or communication from the wiki creator and/or participants to others. 

 

 

Quality Questions

 

Category Indicator Score as “0” if there is Score as “1” if there is
Information Gateway
Course Materials No evidence that instructional or content materials are posted on the wiki. Evidence that instructional or content materials are posted on the wiki by teachers for viewing by students or posted by students for viewing by other students (i.e. do not code 1 if a student posts instructional or content materials for teacher viewing, other viewing, or for the student’s own benefit)
Information Gateway No evidence that links to resources that are beyond the wiki subdomain are posted on the wiki by teachers for viewing by students or posted by students for viewing by other students. Evidence that links are posted on the wiki to resources that are beyond the wiki subdomain by teachers for viewing by students or posted by students for viewing by other students (i.e. do not code 1 if a student posts links to external sites for teacher viewing, other viewing, or for the student’s own benefit)

 


 

 

Category Indicator Score as “0” if there is Score as “1” if there is
Participation
Contribution No evidence that students contribute by making edits or changes (including but not limited to adding text, uploading files, etc.) to the wiki Evidence that students contribute by making edits or changes (including but not limited to adding text, posting comments, uploading files, etc.) to the wiki
Ownership No evidence that a student (or students) owns the wiki. Wiki “owners” are the wiki’s creator, primary facilitator or primary content producer. Evidence that a student (or students) owns the wiki. Wiki “owners” are the creator, primary facilitator or primary content producer for the wiki.
Individual Pages No evidence that at least one student owns an individual page. Wiki page “owners” are the creator, primary facilitator or content producer. Evidence that at least one student owns an individual page. Wiki page “owners” are the creator, primary facilitator or content producer for the page.
Shared Pages No evidence that at least two students co-own a single shared page. Wiki page “owners” are the creator, primary facilitator or content producer for the page. Evidence that at least two students co-own a single shared page. Wiki page “owners” are the creator, primary facilitator or content producer for the page.


 

Category Indicator Score as “0” if there is Score as “1” if there is
Expert Thinking EVIDENCE IN EXPERT THINKING CATEGORIES CAN COME FROM UPLOADED DOCUMENTS
Content Required No evidence that students complete tasks requiring academic content knowledge, or information and skills that are taught as part of the school curriculum. Evidence that at least one student attempts to complete a task on the wiki requiring academic content knowledge, or information and skills that are taught as part of the school curriculum. Simple reading and writing do not count here, although specialized forms of reading (such as grammatical parsing) or writing (such as writing poetry or drama) do count.
Knowledge organization No evidence that students complete tasks requiring analyzing, organizing, or synthesizing information. Evidence that at least one student attempts to complete tasks on the wiki requiring analyzing, organizing, or synthesizing information.
Self-Reflection No evidence that students use the wiki to reflect on their thinking process or work products. Evidence that at least one student uses the wiki to reflect on his/her thinking process or work product. The reflective exercise must occur on the wiki, but the object of reflection does not have to be on the wiki.
Crediting No evidence that students credit the supports of their work products. Evidence that at least one student attempts to credit the supports of his/her work product on the wiki.
Teacher Feedback No evidence that there is teacher feedback on the wiki to comment on the thinking process or work products of students. Evidence that that there is teacher feedback on the wiki that comments on the thinking process or work products of students.

 


 

Category Indicator Score as “0” if there is Score as “1” if there is
Complex Communication
Commenting No evidence that students comment upon the work of another wiki user (Student or non-student). Evidence that at least one student comments upon the work of another wiki user (Student or non-student).
Discussion No evidence that students sustain a conversation of at least four turns with other users (student or non-student) Evidence that at least one student engages in a discussion with at least four conversational turns with another user
Concatenation No evidence that more than one student contributes discrete text or content to a page. Evidence that at least two students contribute discrete text or content to a page. Evidence of concatenation can come from bylines attributed to discrete text or from edit histories.
Copyediting No evidence that one student corrects the grammar, spelling, punctuation, or syntax of another user’s discretely owned text or content. Evidence that at least one student corrects the grammar, spelling, punctuation, or syntax of another user’s discretely owned text or content. Evidence generally must come from edit histories with clear identities.
Co-Construction No evidence that a student substantively edits the work of another user working on the same page such that the text is no longer discretely owned. At least one student substantively edits the work of another user working on the same page such that the text is no longer “discretely” owned. Evidence generally must come from edit histories with clear identities.
Scheduling No evidence that a student participates in a scheduling activity on the wiki, such as posting their name to a list of times or responsibilities. Evidence that a student participates in a scheduling activity on the wiki, such as posting their name to a list of times or responsibilities.
Student planning No evidence that students discuss and develop strategies on the wiki for completing a work product (on or off wiki) with other students. Evidence that at least one student discusses and develops strategies on the wiki for completing a work product (on or off wiki) with at least one other student.

 

Category Indicator Score as “0” if there is Score as “1” if there is
New Media Literacy
Formatting No evidence that students use formatting such as colored text, boldface, italics, and so forth on a wiki page. (Formatting on attached documents does not count). Evidence that at least one student uses formatting such as colored text, boldface, italics, on a wiki page. (Formatting on attached documents does not count).
Links No evidence that students add Web link to a wiki page. Evidence that at least one student adds Web links to a wiki page. The links can be to other wiki pages, external Web sites, or to uploaded documents. (Links within attached documents do not count). For example: http://www.google.com.
Hyperlinking No evidence that students add links to non-URL text or images on a wiki page. Evidence that at least one student adds a link to non-URL text or images on a wiki page.  For example: Google
Images No evidence that students embed images to a wiki page. Evidence that at least one student embeds an image into a wiki page. You must be able to see the image on a page. Merely uploading a file should be coded under “Upload”. Simple links should be coded under “Links.”
Multimedia No evidence that students embed sounds, videos, or other multimedia files to a wiki page. Evidence that at least one student embeds sounds, videos, or other multimedia files to a wiki page. (Multimedia files within attached documents do not count. The image or multimedia application must appear on the page as an embed frame. Merely uploading a file should be coded under “Upload”. Simple links should be coded under “Links.” ).
Upload No evidence that students upload files to the wiki. Evidence that at least one student uploads a file to the wiki.

 

Final Overall Rating Questions

For the final categories, answer the following questions on a 1-7 scale (7 being highest)

Overall Participation

Overall, to what extent do students participate in this wiki?

No participation

1

Minimal participation

2

A little participation

3

Some participation

4

A fair amount of  participation

5

Considerable participation

6

Extensive participation

7

Overall Expert Thinking

Overall, to what extent does this wiki promote the development of students’ expert thinking skills—such as ill-structured problem solving, critical thinking, creative problem solving, metacognition, initiative taking?

No expert thinking

1

Minimal expert thinking

2

A little expert thinking

3

Some expert thinking

4

A fair amount of expert thinking

5

Considerable expert thinking

6

Extensive expert thinking

7

Overall Complex Communication

Overall, to what extent does this wiki promote the development of students’ complex communication skills—such as communication and collaboration?

No complex communication

1

Minimal complex communication

2

A little complex communication

3

Some complex communication

4

A fair amount of complex communication

5

Considerable complex communication

6

Extensive complex communication

7

Overall New Media Literacy

Overall, to what extent does this wiki promote the development of students’ new media literacy skills- the skills necessary to critically consume and produce multimedia content?

No new media literacy

1

Minimal new media literacy

2

A little new media literacy

3

Some new media literacy

4

A fair amount of new media literacy

5

Considerable new media literacy

6

Extensive new media literacy

7

Creative Commons License
Wiki Quality Instrument and Protocol by Justin Reich is licensed under a Creative Commons Attribution 3.0 Unported License.
Based on a work at www.edtechresearcher.com.

3 Comments

  1. Andi Strackeljahn
    March 5, 2012

    Justin, thank you for sharing the specifics of this tool! Could you email me or re-post the table under decision rules? the last column is cut off.
    Also I am wondering why just a 0 or 1 rating why not a greater continuum?

    Reply
  2. Justin Reich
    March 5, 2012

    Hi Andi,

    Sorry about that… it must have gotten messed up when I changed templates. I’ll fix it, but for now, I’ve upload the table as a word document (and I’ll email you).

    In regards to the decision to go dichotomous, it is explained in Part III (http://www.edtechresearcher.com/wiki-quality-instrument/part-iii-developing-the-wqi/), but the simple answer is that wikis are too diverse for reliable scalar comparison. How would you compare “good” copyediting by second graders working on retelling stories with “good” copyediting by 11th graders working on science fair projects? How do you compare frequency in wikis with one page vs. hundreds of pages. We tried a few different approaches, and they never worked. Essentially, in our model, you get a continuum by adding up the items in a subsection.

    If we had studied a narrower range of wikis (7th grade earth science wikis in Iowa), then we probably would have been able to use some kind of likert scale.

    Reply
  3. Andi Strackeljahn
    March 5, 2012

    Ah yes what an undertaking to sum up all the uses of wikis, this is a great approach!

    Reply

Leave a Reply