I am very pleased to announce that my blog now has a new home with Education Week at http://blogs.edweek.org/edweek/edtechresearcher/. Same sort of content, but at a different home. Please come join me there!

There are no stories in the media about the time a boy thought about making a terrible decision, then chatted with some people online, got some good advice, and didn’t make that terrible decision.

As we used to say when I led wilderness trips, “you can’t tell stories about sunshine” (borrowed from Garrison Keilor I think).

Part of the problem with the media coverage of the Internet is that terrible incidents are magnified and lots of little good things that occur are lost.

But there was a recent NYT article about sunshine: “Spring Break Gets Tamer as World Watches Online.” Basically, Facebook turns the world into a Puritan New England village, where the houses were built tightly together along Main Street so everyone could see everyone’s business and keep them on all the straight and narrow (not that it mattered, since it was all preordained, but anyway…). As one 22 year old says, “At the beach yesterday, I would put my beer can down, out of the picture every time,” Ms. Sawyer said. “I do worry about Facebook. I just know I need a job eventually.”

There will always be stories about people who had their lives destroyed by dreadful behavior in public that was captured by Facebook. It’s nice to see a story that suggest that maybe, in some ways, a networked public improves our behavior as well.

I am currently a doctoral student in Leadership in Schooling.  I am a technology specialist in an urban public school system in Massachusetts, and I am considering doing my dissertation on iPad usage as well.  I read your postings about Auburn and am wondering what types of information would add to the literature about how student achievement in early childhood can be affected by iPad/technology usage.

Well, the good news is that it’s an amazing time to be an education technology researcher! We know virtually nothing about the impact that tablets could or should have in the elementary classroom. Almost anything you do could add to the literature on early ed and iPads–heck, you could invent the literature on early ed and iPads.

There are all kinds of great research questions to ask. Here are three general categories:

1) Can I design an “X” that works?

If you have a idea for a particular pedagogical strategy with iPads, then you might consider doing some kind of design research. Design research is a suite of methods where researchers develop an intervention (which is a fancy word for a lesson, or unit, or app, or platform) and then iteratively tinker with the design of the intervention to gradually improve the efficacy of the effort. So if you have something that you think people should be doing, then start developing it, and find some teacher-partners to help you co-design and test your strategy.

For instance, my colleague Andrew Manches at the Institute of Education at the University of London has built a very cool iPad app, Digicubes, that replicate some of the most commonly used “blocks” that are math manipulatives. Having built the app, he both tests how it works in comparison to the physical world blocks and uses those tests to see how he can make them even better.

If you have an idea for a particular kind of learning strategy with the iPad, that would make for an exciting study.

2) What are people doing with iPads?

I have been stunned by the demand for iPads, a consumer device with a weak educational infrastructure and minimal–and totally unproven–educational software. But schools are buying them in droves and finding all kinds of cool and pointless things to do with them. Get out into the field and figure out who is doing what.

You might start by trying to survey absolutely everyone you can find who uses iPads in elementary school and getting them to provide some short, open ended descriptions of the kinds of things they are doing. You might be able to get Apple to help you by sharing the names of districts that have made big purchases, though there may be privacy issues. So you might have to go to conferences, network online, and make some phone calls to figure out who is using iPads and what they are doing with them.

Ideally in such a survey, you could also collect contact information for people who would let you into their classrooms and show you what they have going on.

Even some basic descriptive research that provides a taxonomy of teacher approaches to using iPads in early ed would be a very valuable contribution at this point.

3) What experiences are iPad-using teachers having in their classrooms?

We also need much more rich, textured, anthropological data about exactly what is happening in these iPad using classrooms. Actually, a great study could involve having Mike Muir and his colleagues help you get connected to Auburn, and let you do a year or two worth of field work in those Kindergarten classrooms. What are teachers doing? How are students responding? What do parents think? In this one town that has made a big investment in iPads, how are all the different stakeholders making meaning of their experience? If learning seems to be happening, how exactly is it happening?

Ideally, Mike and his team would let you have access, and then they’d leave you alone, and you could visit on different days and so forth. It would be great to have a couple of independent teams working Auburn, so they could share findings and compare after months or years.

So it’s a moment of nearly infinite opportunity for those getting into technology research, and I think the early education space is an incredibly promising one, that I hope more university-researchers and teacher-researchers pursue!

I currently have a course wiki (using wikispaces) between our preservice teachers and ninth graders in a remote secondary school. We used the wiki for a specific project, so it won’t fall under your longevity categories but I am interested in citing your work and in rating the quality of the educational wiki.

I’m not sure if you have any advice for me, but I would love to be able to rate and report the quality of the wiki that was created by this collaboration.

I’m going to start with this reminder (which Mike has echoed in his own way): I think it’s completely awesome that Mike and colleagues are running randomized control trials of iPads in Kindergarten classrooms. We know absolutely nothing about how these tools might work with young kids, the teacher experiments going on are promising, and we need this kind of research being done. Kudos to Mike and his colleagues for doing it.

In the end, I think Mike and I agree about how school districts should act upon Mike’s early findings (if they hold up, more on that in a minute). I think Mike and I come to similar conclusions through somewhat different routes.  Mike argues that researchers and practitioners should analyze research findings differently. I disagree. They should analyze findings the same way, and they should act on them differently.

The story in a nutshell: Auburn, Maine has a multi year early literacy initiative (Bravo! Nothing can be more important in schools). As part of that initiative they introduce ipads and they stagger the introduction of iPads in such a way that half of the 16 K classrooms get them in September and half don’t get them until December. Mike and colleagues measured students with ten tests of early literacy, and then compared the intervention group (the kids in iPad classrooms) and the control group (the kids who didn’t get the iPads early). Last week, they released a press release and a two page research report reporting the following: In all 10 tests, the average scores of the students in the intervention classrooms exceeded the scores of the control classrooms. In 9 of those 10 tests, the differences between the two classrooms were not significantly different. In 1 test, the difference was significantly different.

So how should people read these findings? Here are my suggestions:

First, researchers, journalists and practitioners should analyze the statistics in exactly the same way. Mike argues that “It is a major fallacy to think everyone should be a researcher, or think and analyze like one .” I won’t comment about what “everyone” should do, but I will say that anyone making judgments based on statistical findings should approach those findings using the same analytic strategies. Journalists and practitioners whose job involves interpreting statistical findings should learn the basics of how to make sense of tests of statistical significance or should build partnerships with people who can help them. [Edit: Please read the comments, where Mike rephrases his point.] 

That’s a major purpose of this blog: to help people learn how to read research findings in education technology. We simply need more literacy among educators about how to evaluate claims of “research-based” ideas. I’m not writing this blog from “a researcher’s perspective.” I’m writing from a perspective of someone whose job it is to help educators make real-world decisions about research (you can go here for a sense of the consulting that I do).

Whoever you are or whatever your background, when you read the research report, your conclusions should be pretty similar: In 1 of 10 tests, the iPads modestly but significantly improved student learning as measured by a particular test. (Go back to the original post for more discussion of significant).  In 9 of 10 tests, we have little confidence that the iPads improved students literacy scores, and even if you believe they did, the improvements are very, very small.

In regard to these nine Mike says “It is accurate to say we are unsure of the role chance played on those results.” But’s that’s not the way I would put it. The more important issue is “It is accurate to say we are unsure of the role that the intervention– the iPads–played on those results.”  Again, it doesn’t matter if you are a researcher or a practitioner–anyone analyzing those numbers should have little confidence that the iPad had an impact on those 9 measures. If you decide to set aside the results of the statistical testing and believe that the iPads had a positive impact on all 10 measures, then you should at least acknowledge that the effect sizes of the iPad intervention on those 9 measures were very,very  small.

Now, what people do with those findings absolutely should be different. Educators constantly have to make decisions with imperfect information. Researchers get to throw up their hands and say “who knows what it means! Let’s go back to our lairs and devise more experiments.” Educators have to say “we may not be totally sure what the research means, but we have to make decisions regardless.” (Mike and I appear to be in complete agreement here.)

So you are a superintendent interested in iPads for your district… what should you actually do with these findings?

First, no school district outside of Auburn should make any decisions based on these findings under the full report is released and the data is offered to other researchers. I think it would be great to have Mike and colleagues release this report as a white paper rather before releasing a peer reviewed article– the need is great and the information is much needed given the scale of iPad investment happening in schools. But until we have a robust report of the findings, people simply shouldn’t give serious credibility to these findings. That is not at all a knock on Mike and colleagues, but as the recent fiasco at CERN demonstrates, initial findings can be as fleeting as an atomic particle. (Researchers reported the possibility of a faster-than-light particle, and then found that the discovery was due to loose cables rather than our fundamentally misunderstanding of the laws of the universe).

There are all kinds of potential “loose cables” in a research report like this; here are two I would be looking for:

1. Failed randomization: not that they did anything wrong, but by chance, the control and intervention classrooms might have been different. If the baseline scores of the control group are lower than the intervention group, then the results might be better chalked up to catching up than to the iPads.
2. Outliers (or high-leverage cases to be more precise): the intervention-group differences in 9 of the 10 tests are so modest that one really successful classroom or a few kids with big score gains could be entirely responsible for the positive non-significant findings

My estimation is that’s an appropriately cautious suggestion for education leaders. There are traces of good stuff here; we need a few pioneering districts, like Auburn, to keep following the trail.

David has a great insight at the end about the disjunction between norms in formal learning environments and conditions that promote successful online collaborations:

DW: No, but I wonder whether our experience with wikis is in fact showing that wikis by themselves do not enable collaboration. I’m thinking again anecdotally about the successful wikis and business and outside of business, the ones that have worked have not been, as far as I know, not purely open, everybody equal, everybody equally involved. They’ve often been – as friends of mine say, there’s a gardener involved, there’s somebody who is making sure that everything works. There’s somebody who feels primary responsibility and will take the actions required to make it a high quality wiki, not just a social experiment in openness. We’re actually trying to develop something that’s high quality. And in the classroom, there’s probably little experience of this. And it’s not a situation, given the way that we at least imagine classrooms running and imagine them running online, it’s not a situation in which one person assuming control – control is too strong, but the gardening position, the pruning of it – there’s no comfortable way for a student to do that because you will get the “Get your own damn page” kind of response. It’s not maybe a failure of our understanding of wikis.

 

Essentially, schools are designed to reward individual effort, and so there are strong incentives for students to protect and isolate their individual contributions and strong teacher incentives to “make everyone participate” equally. In fact, however, successful online collaborations may depend upon good leadership and a necessarily inegalitarian distribution of work.

I’d encourage readers to listen to the whole thing to get a sense of the context of his observation, and then leave a comment here about what you think!

In partnership with Elon University’s, they’ve released a report on Imagining the Internet, under this headline:

Teens-to-20s to benefit and suffer due to ‘always-on’ lives. From their amazing ability to juggle many tasks to their thirst for instant gratification, survey reveals experts’ hopes and fears

The survey has an interesting methodology: they interviewed over 1,000 experts in the field of technology, education, and social media, asking them to reflect upon their hopes and fears for the future of young people. The report is a kind of word association test, with respondents asked to briefly reflect on topics like multi-tasking, human evolution, the digital divide, education and so forth. A number of my Berkman colleagues, like danah boyd and David Weinberger are featured.

The collection of responses contains hope for a new generation of cyborg-super humans and despair at the miserable generation taking our their white earbuds and preparing to enter the workplace (and many of them are probably leaving their ear buds in). The report makes for easy skimming, with hundreds of comments from scholars and experts on all kinds of topics. Overall, the comments probably say less about the future than they do about the philosophical orientation of the experts.

The takehome message- the Internet continue to be good and bad. It reminds me of a point made by my undergraduate advisor, the historian Edmund Russell. He observed that Shakespeare’s collect plays were organized into three categories: the tragedies, where events led towards disaster; the comedies, where conflicts led to joyful resolutions; and then finally the histories, where things got better in some ways for some people and got worse for others. I’ve always been inclined to the latter perspective, and the Pew report is good fodder for it.

The article is written for a general audience of education researchers, and while I think it’s highly readable (so far as scholarly articles go), many folks may prefer a summaryof the article. This post, therefore, is particularly targeted at classroom teachers and other school educators to share a bit about our findings and suggestions for using social media in classrooms. First, I explain what we did, and then I give some advice for wiki using educators. If you want to skip to the advice, Click Here.

(This post is also available as a white paper: The State of Wiki Usage: A Summary for Educators)

Our Research Questions

The use of social media in K-12 settings has grown at a tremendous rate. According to a very well conducted 2009 federal study (the Fast Response System Survey), 40% of teachers report using blogs or wikis in their classroom and approximately 20% of teachers require their students to contribute to blogs or wikis. Given the recent history of these tools, that is remarkable growth.

Teacher self-reports on surveys, however, don’t tell us anything about what they are doing with those tools, and that was our mission. We asked two kinds of questions: “What kinds of learning opportunities do students have with wikis?” and “Are those learning opportunities equitably distributed among schools serving difference socioeconomic populations?” To put these more simply, are wikis any good and do only certain kids get the good ones?

What We Did

To answer these questions, we first developed a tool to measure wiki quality, called the Wiki Quality Instrument. We used three strategies to develop this instrument (described thoroughly here).

1. We spent about a year asking teachers and students what they thought high quality work on wikis looked like.
2. We randomly sampled thousands of wikis from a population of nearly 200,000 wikis, and we looked at what kinds of things happen on U.S., K-12 wikis.
3. We thoroughly reviewed the research literature on measuring quality and learning in online learning environments.

Teachers told us that they used wikis for four reasons: 1) to help with class logistics, 2) to develop collaboration skills, 3) to deepen and display student understanding, and 4) to learn to use technology to communicate. These ideas cohere very well with the framework of 21st Century Skills, so our Wiki Quality Instrument (WQI) measures the degree to which wikis provide opportunities for students to develop 21st century skills such as expert thinking, complex communication, and new media literacy. The WQI  consists of 24 questions about things students might do on a wiki– questions like “Do students copyedit each other’s work?” or “Do students embed multimedia?’ The questions are in five categories: Information Consumption, Participation, Expert Thinking, Complex Communication, and New Media Literacy, and the whole instrument can be found here.

Having created a great new instrument to measure wiki quality, we needed wikis to measure. PBworks.com helped us (for free) get access to all 179,851 publicly-viewable, education-related wikis hosted on PBWorks from the creation of the company in 2005 through August of 2008 (this is 70% of all ed wikis, another 30% were private). These wikis are used in every grade level from Pre-K through Grad school, in every subject area, in dozens of countries around the world, and for multiple purposes: from posting syllabi to creating portfolios to maintaining mini-topical encyclopedias.

We took a 1% random sample of these wikis, so the wikis we analyzed were representative of the population as a whole (they are certainly representative of viewable PBworks wikis, and we argue in the paper that they are probably representative of classroom wikis in general). We then separated out all wikis used in higher ed, in other countries, in private schools, and in unidentifiable places, and we were left with a subsample of 255 wikis used in U.S., K-12 public schools. We could examine the entire edit history of these wikis to measure their quality, and since we knew what school they came from, we could also gather school level socioeconomic status data from the Common Core of Data (It’s fun! Check out your own school!).

We then measured wiki quality in all of these wikis at multiple time points, at days 7, 14, 30, 60, 100, and 400. This allowed us to represent wiki quality as a trajectory, rather than a single value. We used statistical models to estimate the expected wiki quality trajectory for the whole population.

What We Learned

Where are wikis used?

We found wikis are used throughout the academic subject areas and in all grade levels.

This diverse activity occurred throughout the K–12 sector. Of our 255 public school wikis, 25% supported instruction in Grades K–5, 28% in Grades 6–8, and 52% in Grades 9–12 (the sum of these percentages exceeds 100% because some wikis supported multiple grades). Wikis were used not just in computer classes; they supported instruction throughout the curriculum. We found that 34% of wikis supported English/language arts instruction, 13% supported social studies, 18% supported science, 13% supported math, 14% supported computer science, and 26% supported another subject or no subject.

This coheres with findings from the 2009 FRSS survey.

How does wiki quality develop?

Wiki quality typically grows very quickly early on, and then quality growth levels. In other words, whatever behaviors you see emerge in the first two weeks, tend to be the only behaviors that ever happen on the wiki.

Look at the table below of wiki quality trajectories. Remember that each line represents the number of opportunities that we identified on each wiki for 21st century skill development (out of our list of 24 behaviors).  Each box in the matrix contains the measures for a single wiki. Note that nearly every line is straight, and the others curve up rapidly early on and then flatten out (you should also find one exception). One way to describe this pattern to say that great wikis are born rather than made. By day 14 or so, we can reliably predict a wiki’s quality.

What is the distribution of wiki quality in the U.S.?

The table below shows the distribution of wiki quality in U.S. schools. We summarize wiki quality by using the Day 14 composite wiki quality score. We break wikis into four categories. About 40% of wikis serve no useful purpose for students. Most of these are “trial wikis,” which a teacher creates but does nothing with. About a third of wikis are “Teacher-Centered Content Delivery Devices.” These are teacher websites, where teachers disseminate content–syllabi, facts, links, and so forth–to students. About a quarter of wikis are individual student assignments. Rather than write a paper or keep a portfolio in a folder, individual students complete these tasks on a wikis. About 1% of wikis are “Collaborative, Multimedia Performances of Understanding.” These are the wikis that are what ed-tech enthusiasts imagine when they talk about the potential of wikis. They are pretty inspiring . There aren’t very many of them.

In general, our findings cohere with 30 years of educational technology research. There are a handful of teachers who make remarkable use of new technologies, but for the most part, when teachers adopt new technologies, they use them to extend existing practices rather than to develop innovative practices.

How does wiki usage differ by school level socioeconomic status?

Too much. Wikis created in more affluent schools are significantly more likely to provide opportunities for students to develop 21st century skills compared to wikis created in schools serving low-income families. Wikis also persist longer in affluent schools. Wikis in Title I eligible schools (40% of students eligible for free lunches) typically last 6 days, compared to 30 days in wikis in non-Title I schools. Not only that, but these figures only quantify between-school differences. While interviewing teachers, we found striking within-school inequities as well. Teachers often told us that they primarily use wikis, or do more sophisticated stuff with wikis, with their AP and Honors students, who we know are disproportionately white or Asian and affluent.

What Should Educators Learn from this Study?

This is a descriptive study of wiki usage in U.S., K-12 schools. It tells us a lot about what is happening. It is not designed to provide specific, actionable advice in the way that a randomized experiment is designed to figure out “what works.” But, we live in a world where everyday educators have to make real decisions, so this is my best effort at translating our findings into suggestions. I should emphasize that these are my interpretations of what teachers should consider based on these data; these are not “proven practices” from our findings. You might say that these are all suggestions for action research projects that educators can take on in their own schools and classrooms.

1) Clarify your learning goals and enshrine these goals in rubrics and assessment criteria. 

Teachers told us that they used wikis to develop collaboration skills, technology skills, and critical thinking skills. But most wikis are just content delivery devices. Why the disconnect between goals and reality?

There are lots of reasons, but one we can control has to do with rubrics. We looked at lots of teacher’s rubrics, and they tended to check whether students had 5 pages, 4 paragraphs, 3 links, 2 images, and a partridge in a pear tree. Basically, teachers want deeper learning, but they evaluate whether students are following directions. Following directions is a useful skill, but nobody told us it is why they created a wiki. Teachers should be sure that their assessment criteria align with their most important goals. (I don’t recommend any particular set of rubrics online. I haven’t found ones I think are really great. Make your own.)

2) Carefully structure the development of early norms.

High quality wikis tend to start at high levels of quality. My hunch is that the early norms that teachers set determine the behaviors that follow on wikis. If you want really deep collaboration to occur on your wikis, establish that very early on as a norm. Do exercises that force students to discuss ideas, copyedit, and substantively edit each others work. I believe the first few days and weeks are crucial.

3) Work with a buddy. 

Many teachers do wiki projects in highly isolated environments. Find a teacher– in your school, through Twitter or classroom20.com or somewhere–to be your thought partner in using wikis in the classroom.

4) Focus on between-school inequality rather than within-school inequality. 

We met some remarkable teachers doing awesome work with wikis with students in urban schools. When you asked these teachers about inequality in schools, they focused on between-school inequalities. “Since kids in the suburbs get to do this stuff, I need to give my kids these opportunities, even if access is difficult.” Teachers in low-income schools who shy away from technology tend to focus on within-school inequalities. “Not all my kids have access at home, so it’s not fair to make them use wikis.” I have talked about this at length in another post, but basically, I think we need to coach teachers to focus on between-school inequalities.

5) If you are a school leader, provide professional support 

Lots of schools spend 6 or 7 figure sums on hardware and infrastructure, and then 4 or 5 figure sums on professional development and support. If you want innovative uses of new media throughout your school (not just in isolated pockets), don’t expect that providing hardware is sufficient. Thirty years of ed-tech research have discredited that idea. We’re going to need to provide extensive support to educators to help them figure out how to develop new lessons, new curricula, and new pedagogy that realize the terrific potential of new media in schools.

If you have questions or responses, please feel free to leave something in the comments. If you have specific criticisms of the methods, advice or findings, you might consider reading the full article before posting.